{"data":{"id":"10.48550/arxiv.2107.08966","type":"dois","attributes":{"doi":"10.48550/arxiv.2107.08966","prefix":"10.48550","suffix":"arxiv.2107.08966","identifiers":[{"identifier":"2107.08966","identifierType":"arXiv"}],"alternateIdentifiers":[{"alternateIdentifierType":"arXiv","alternateIdentifier":"2107.08966"}],"creators":[{"name":"Schäfer, Lukas","nameType":"Personal","givenName":"Lukas","familyName":"Schäfer","affiliation":[],"nameIdentifiers":[]},{"name":"Christianos, Filippos","nameType":"Personal","givenName":"Filippos","familyName":"Christianos","affiliation":[],"nameIdentifiers":[]},{"name":"Hanna, Josiah P.","nameType":"Personal","givenName":"Josiah P.","familyName":"Hanna","affiliation":[],"nameIdentifiers":[]},{"name":"Albrecht, Stefano V.","nameType":"Personal","givenName":"Stefano V.","familyName":"Albrecht","affiliation":[],"nameIdentifiers":[]}],"titles":[{"title":"Decoupled Reinforcement Learning to Stabilise Intrinsically-Motivated Exploration"}],"publisher":"arXiv","container":{},"publicationYear":2021,"subjects":[{"lang":"en","subject":"Machine Learning (cs.LG)","subjectScheme":"arXiv"},{"lang":"en","subject":"Artificial Intelligence (cs.AI)","subjectScheme":"arXiv"},{"subject":"FOS: Computer and information sciences","subjectScheme":"Fields of Science and Technology (FOS)"},{"subject":"FOS: Computer and information sciences","schemeUri":"http://www.oecd.org/science/inno/38235147.pdf","subjectScheme":"Fields of Science and Technology (FOS)"}],"contributors":[],"dates":[{"date":"2021-07-19T15:31:02Z","dateType":"Submitted","dateInformation":"v1"},{"date":"2021-07-20T00:37:24Z","dateType":"Updated","dateInformation":"v1"},{"date":"2021-07-22T18:50:01Z","dateType":"Submitted","dateInformation":"v2"},{"date":"2021-07-26T00:01:08Z","dateType":"Updated","dateInformation":"v2"},{"date":"2022-02-09T15:06:21Z","dateType":"Submitted","dateInformation":"v3"},{"date":"2022-02-10T01:22:21Z","dateType":"Updated","dateInformation":"v3"},{"date":"2021-07","dateType":"Available","dateInformation":"v1"},{"date":"2021","dateType":"Issued"}],"language":null,"types":{"ris":"GEN","bibtex":"misc","citeproc":"article","schemaOrg":"CreativeWork","resourceType":"Article","resourceTypeGeneral":"Preprint"},"relatedIdentifiers":[],"relatedItems":[],"sizes":[],"formats":[],"version":"3","rightsList":[{"rights":"Creative Commons Attribution 4.0 International","rightsUri":"https://creativecommons.org/licenses/by/4.0/legalcode","schemeUri":"https://spdx.org/licenses/","rightsIdentifier":"cc-by-4.0","rightsIdentifierScheme":"SPDX"}],"descriptions":[{"description":"Intrinsic rewards can improve exploration in reinforcement learning, but the exploration process may suffer from instability caused by non-stationary reward shaping and strong dependency on hyperparameters. In this work, we introduce Decoupled RL (DeRL) as a general framework which trains separate policies for intrinsically-motivated exploration and exploitation. Such decoupling allows DeRL to leverage the benefits of intrinsic rewards for exploration while demonstrating improved robustness and sample efficiency. We evaluate DeRL algorithms in two sparse-reward environments with multiple types of intrinsic rewards. Our results show that DeRL is more robust to varying scale and rate of decay of intrinsic rewards and converges to the same evaluation returns than intrinsically-motivated baselines in fewer interactions. Lastly, we discuss the challenge of distribution shift and show that divergence constraint regularisers can successfully minimise instability caused by divergence of exploration and exploitation policies.","descriptionType":"Abstract"},{"description":"Published at the International Conference on Autonomous Agents and Multi-Agent Systems (AAMAS) 2022","descriptionType":"Other"}],"geoLocations":[],"fundingReferences":[],"xml":"PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0idXRmLTgiPz4KPHJlc291cmNlIHhtbG5zPSJodHRwOi8vZGF0YWNpdGUub3JnL3NjaGVtYS9rZXJuZWwtNCIgeG1sbnM6eHNpPSJodHRwOi8vd3d3LnczLm9yZy8yMDAxL1hNTFNjaGVtYS1pbnN0YW5jZSIgeHNpOnNjaGVtYUxvY2F0aW9uPSJodHRwOi8vZGF0YWNpdGUub3JnL3NjaGVtYS9rZXJuZWwtNCBodHRwOi8vc2NoZW1hLmRhdGFjaXRlLm9yZy9tZXRhL2tlcm5lbC00LjMvbWV0YWRhdGEueHNkIj4KICA8aWRlbnRpZmllciBpZGVudGlmaWVyVHlwZT0iRE9JIj4xMC40ODU1MC9BUlhJVi4yMTA3LjA4OTY2PC9pZGVudGlmaWVyPgogIDxhbHRlcm5hdGVJZGVudGlmaWVycz4KICAgIDxhbHRlcm5hdGVJZGVudGlmaWVyIGFsdGVybmF0ZUlkZW50aWZpZXJUeXBlPSJhclhpdiI+MjEwNy4wODk2NjwvYWx0ZXJuYXRlSWRlbnRpZmllcj4KICA8L2FsdGVybmF0ZUlkZW50aWZpZXJzPgogIDxjcmVhdG9ycz4KICAgIDxjcmVhdG9yPgogICAgICA8Y3JlYXRvck5hbWUgbmFtZVR5cGU9IlBlcnNvbmFsIj5TY2jDpGZlciwgTHVrYXM8L2NyZWF0b3JOYW1lPgogICAgICA8Z2l2ZW5OYW1lPkx1a2FzPC9naXZlbk5hbWU+CiAgICAgIDxmYW1pbHlOYW1lPlNjaMOkZmVyPC9mYW1pbHlOYW1lPgogICAgPC9jcmVhdG9yPgogICAgPGNyZWF0b3I+CiAgICAgIDxjcmVhdG9yTmFtZSBuYW1lVHlwZT0iUGVyc29uYWwiPkNocmlzdGlhbm9zLCBGaWxpcHBvczwvY3JlYXRvck5hbWU+CiAgICAgIDxnaXZlbk5hbWU+RmlsaXBwb3M8L2dpdmVuTmFtZT4KICAgICAgPGZhbWlseU5hbWU+Q2hyaXN0aWFub3M8L2ZhbWlseU5hbWU+CiAgICA8L2NyZWF0b3I+CiAgICA8Y3JlYXRvcj4KICAgICAgPGNyZWF0b3JOYW1lIG5hbWVUeXBlPSJQZXJzb25hbCI+SGFubmEsIEpvc2lhaCBQLjwvY3JlYXRvck5hbWU+CiAgICAgIDxnaXZlbk5hbWU+Sm9zaWFoIFAuPC9naXZlbk5hbWU+CiAgICAgIDxmYW1pbHlOYW1lPkhhbm5hPC9mYW1pbHlOYW1lPgogICAgPC9jcmVhdG9yPgogICAgPGNyZWF0b3I+CiAgICAgIDxjcmVhdG9yTmFtZSBuYW1lVHlwZT0iUGVyc29uYWwiPkFsYnJlY2h0LCBTdGVmYW5vIFYuPC9jcmVhdG9yTmFtZT4KICAgICAgPGdpdmVuTmFtZT5TdGVmYW5vIFYuPC9naXZlbk5hbWU+CiAgICAgIDxmYW1pbHlOYW1lPkFsYnJlY2h0PC9mYW1pbHlOYW1lPgogICAgPC9jcmVhdG9yPgogIDwvY3JlYXRvcnM+CiAgPHRpdGxlcz4KICAgIDx0aXRsZT5EZWNvdXBsZWQgUmVpbmZvcmNlbWVudCBMZWFybmluZyB0byBTdGFiaWxpc2UgSW50cmluc2ljYWxseS1Nb3RpdmF0ZWQgRXhwbG9yYXRpb248L3RpdGxlPgogIDwvdGl0bGVzPgogIDxwdWJsaXNoZXI+YXJYaXY8L3B1Ymxpc2hlcj4KICA8cHVibGljYXRpb25ZZWFyPjIwMjE8L3B1YmxpY2F0aW9uWWVhcj4KICA8c3ViamVjdHM+CiAgICA8c3ViamVjdCB4bWw6bGFuZz0iZW4iIHN1YmplY3RTY2hlbWU9ImFyWGl2Ij5NYWNoaW5lIExlYXJuaW5nIChjcy5MRyk8L3N1YmplY3Q+CiAgICA8c3ViamVjdCB4bWw6bGFuZz0iZW4iIHN1YmplY3RTY2hlbWU9ImFyWGl2Ij5BcnRpZmljaWFsIEludGVsbGlnZW5jZSAoY3MuQUkpPC9zdWJqZWN0PgogICAgPHN1YmplY3Qgc3ViamVjdFNjaGVtZT0iRmllbGRzIG9mIFNjaWVuY2UgYW5kIFRlY2hub2xvZ3kgKEZPUykiPkZPUzogQ29tcHV0ZXIgYW5kIGluZm9ybWF0aW9uIHNjaWVuY2VzPC9zdWJqZWN0PgogIDwvc3ViamVjdHM+CiAgPGRhdGVzPgogICAgPGRhdGUgZGF0ZVR5cGU9IlN1Ym1pdHRlZCIgZGF0ZUluZm9ybWF0aW9uPSJ2MSI+MjAyMS0wNy0xOVQxNTozMTowMlo8L2RhdGU+CiAgICA8ZGF0ZSBkYXRlVHlwZT0iVXBkYXRlZCIgZGF0ZUluZm9ybWF0aW9uPSJ2MSI+MjAyMS0wNy0yMFQwMDozNzoyNFo8L2RhdGU+CiAgICA8ZGF0ZSBkYXRlVHlwZT0iU3VibWl0dGVkIiBkYXRlSW5mb3JtYXRpb249InYyIj4yMDIxLTA3LTIyVDE4OjUwOjAxWjwvZGF0ZT4KICAgIDxkYXRlIGRhdGVUeXBlPSJVcGRhdGVkIiBkYXRlSW5mb3JtYXRpb249InYyIj4yMDIxLTA3LTI2VDAwOjAxOjA4WjwvZGF0ZT4KICAgIDxkYXRlIGRhdGVUeXBlPSJTdWJtaXR0ZWQiIGRhdGVJbmZvcm1hdGlvbj0idjMiPjIwMjItMDItMDlUMTU6MDY6MjFaPC9kYXRlPgogICAgPGRhdGUgZGF0ZVR5cGU9IlVwZGF0ZWQiIGRhdGVJbmZvcm1hdGlvbj0idjMiPjIwMjItMDItMTBUMDE6MjI6MjFaPC9kYXRlPgogICAgPGRhdGUgZGF0ZVR5cGU9IkF2YWlsYWJsZSIgZGF0ZUluZm9ybWF0aW9uPSJ2MSI+MjAyMS0wNzwvZGF0ZT4KICA8L2RhdGVzPgogIDxyZXNvdXJjZVR5cGUgcmVzb3VyY2VUeXBlR2VuZXJhbD0iUHJlcHJpbnQiPkFydGljbGU8L3Jlc291cmNlVHlwZT4KICA8dmVyc2lvbj4zPC92ZXJzaW9uPgogIDxyaWdodHNMaXN0PgogICAgPHJpZ2h0cyByaWdodHNVUkk9Imh0dHA6Ly9jcmVhdGl2ZWNvbW1vbnMub3JnL2xpY2Vuc2VzL2J5LzQuMC8iIHJpZ2h0c0lkZW50aWZpZXJTY2hlbWU9IlNQRFgiIHJpZ2h0c0lkZW50aWZpZXI9IkNDLUJZLTQuMCI+Q3JlYXRpdmUgQ29tbW9ucyBBdHRyaWJ1dGlvbiA0LjAgSW50ZXJuYXRpb25hbDwvcmlnaHRzPgogIDwvcmlnaHRzTGlzdD4KICA8ZGVzY3JpcHRpb25zPgogICAgPGRlc2NyaXB0aW9uIGRlc2NyaXB0aW9uVHlwZT0iQWJzdHJhY3QiPkludHJpbnNpYyByZXdhcmRzIGNhbiBpbXByb3ZlIGV4cGxvcmF0aW9uIGluIHJlaW5mb3JjZW1lbnQgbGVhcm5pbmcsIGJ1dCB0aGUgZXhwbG9yYXRpb24gcHJvY2VzcyBtYXkgc3VmZmVyIGZyb20gaW5zdGFiaWxpdHkgY2F1c2VkIGJ5IG5vbi1zdGF0aW9uYXJ5IHJld2FyZCBzaGFwaW5nIGFuZCBzdHJvbmcgZGVwZW5kZW5jeSBvbiBoeXBlcnBhcmFtZXRlcnMuIEluIHRoaXMgd29yaywgd2UgaW50cm9kdWNlIERlY291cGxlZCBSTCAoRGVSTCkgYXMgYSBnZW5lcmFsIGZyYW1ld29yayB3aGljaCB0cmFpbnMgc2VwYXJhdGUgcG9saWNpZXMgZm9yIGludHJpbnNpY2FsbHktbW90aXZhdGVkIGV4cGxvcmF0aW9uIGFuZCBleHBsb2l0YXRpb24uIFN1Y2ggZGVjb3VwbGluZyBhbGxvd3MgRGVSTCB0byBsZXZlcmFnZSB0aGUgYmVuZWZpdHMgb2YgaW50cmluc2ljIHJld2FyZHMgZm9yIGV4cGxvcmF0aW9uIHdoaWxlIGRlbW9uc3RyYXRpbmcgaW1wcm92ZWQgcm9idXN0bmVzcyBhbmQgc2FtcGxlIGVmZmljaWVuY3kuIFdlIGV2YWx1YXRlIERlUkwgYWxnb3JpdGhtcyBpbiB0d28gc3BhcnNlLXJld2FyZCBlbnZpcm9ubWVudHMgd2l0aCBtdWx0aXBsZSB0eXBlcyBvZiBpbnRyaW5zaWMgcmV3YXJkcy4gT3VyIHJlc3VsdHMgc2hvdyB0aGF0IERlUkwgaXMgbW9yZSByb2J1c3QgdG8gdmFyeWluZyBzY2FsZSBhbmQgcmF0ZSBvZiBkZWNheSBvZiBpbnRyaW5zaWMgcmV3YXJkcyBhbmQgY29udmVyZ2VzIHRvIHRoZSBzYW1lIGV2YWx1YXRpb24gcmV0dXJucyB0aGFuIGludHJpbnNpY2FsbHktbW90aXZhdGVkIGJhc2VsaW5lcyBpbiBmZXdlciBpbnRlcmFjdGlvbnMuIExhc3RseSwgd2UgZGlzY3VzcyB0aGUgY2hhbGxlbmdlIG9mIGRpc3RyaWJ1dGlvbiBzaGlmdCBhbmQgc2hvdyB0aGF0IGRpdmVyZ2VuY2UgY29uc3RyYWludCByZWd1bGFyaXNlcnMgY2FuIHN1Y2Nlc3NmdWxseSBtaW5pbWlzZSBpbnN0YWJpbGl0eSBjYXVzZWQgYnkgZGl2ZXJnZW5jZSBvZiBleHBsb3JhdGlvbiBhbmQgZXhwbG9pdGF0aW9uIHBvbGljaWVzLjwvZGVzY3JpcHRpb24+CiAgICA8ZGVzY3JpcHRpb24gZGVzY3JpcHRpb25UeXBlPSJPdGhlciI+UHVibGlzaGVkIGF0IHRoZSBJbnRlcm5hdGlvbmFsIENvbmZlcmVuY2Ugb24gQXV0b25vbW91cyBBZ2VudHMgYW5kIE11bHRpLUFnZW50IFN5c3RlbXMgKEFBTUFTKSAyMDIyPC9kZXNjcmlwdGlvbj4KICA8L2Rlc2NyaXB0aW9ucz4KPC9yZXNvdXJjZT4=","url":"https://arxiv.org/abs/2107.08966","contentUrl":null,"metadataVersion":0,"schemaVersion":"http://datacite.org/schema/kernel-4","source":"mds","isActive":true,"state":"findable","reason":null,"viewCount":0,"viewsOverTime":[],"downloadCount":0,"downloadsOverTime":[],"referenceCount":0,"citationCount":0,"citationsOverTime":[],"partCount":0,"partOfCount":0,"versionCount":0,"versionOfCount":0,"created":"2022-02-21T22:31:36.000Z","registered":"2022-02-21T22:31:37.000Z","published":"2021","updated":"2022-02-21T22:31:37.000Z"},"relationships":{"client":{"data":{"id":"arxiv.content","type":"clients"}},"provider":{"data":{"id":"arxiv","type":"providers"}},"media":{"data":{"id":"10.48550/arxiv.2107.08966","type":"media"}},"references":{"data":[]},"citations":{"data":[]},"parts":{"data":[]},"partOf":{"data":[]},"versions":{"data":[]},"versionOf":{"data":[]}}}}