Learning with options that terminate off-policy

Anna Harutyunyan, Peter Vrancx, Pierre Luc Bacon, Doina Precup, Ann Nowé

Onderzoeksoutput: Conference paperResearch

5 Citaten (Scopus)


A temporally abstract action, or an option, is specified by a policy and a termination condition: the policy guides the option behavior, and the termination condition roughly determines its length. Generally, learning with longer options (like learning with multi-step returns) is known to be more efficient. However, if the option set for the task is not ideal, and cannot express the primitive optimal policy well, shorter options offer more flexibility and can yield a better solution. Thus, the termination condition puts learning efficiency at odds with solution quality. We propose to resolve this dilemma by decoupling the behavior and target terminations, just like it is done with policies in off-policy learning. To this end, we give a new algorithm, Q(β), that learns the solution with respect to any termination condition, regardless of how the options actually terminate. We derive Q(β) by casting learning with options into a common framework with well-studied multi-step off-policy learning. We validate our algorithm empirically, and show that it holds up to its motivating claims.

Originele taal-2English
Titel32nd AAAI Conference on Artificial Intelligence, AAAI 2018
UitgeverijAAAI Press
Aantal pagina's10
ISBN van elektronische versie9781577358008
StatusPublished - 1 jan 2018
Evenement32nd AAAI Conference on Artificial Intelligence - New Orleans, United States
Duur: 2 feb 20187 feb 2018


Conference32nd AAAI Conference on Artificial Intelligence
Verkorte titelAAAI 2018
Land/RegioUnited States
StadNew Orleans


Duik in de onderzoeksthema's van 'Learning with options that terminate off-policy'. Samen vormen ze een unieke vingerafdruk.

Citeer dit