{"data":{"id":"10.48550/arxiv.2106.04117","type":"dois","attributes":{"doi":"10.48550/arxiv.2106.04117","prefix":"10.48550","suffix":"arxiv.2106.04117","identifiers":[{"identifier":"2106.04117","identifierType":"arXiv"}],"alternateIdentifiers":[{"alternateIdentifierType":"arXiv","alternateIdentifier":"2106.04117"}],"creators":[{"name":"Jin, Tiancheng","nameType":"Personal","givenName":"Tiancheng","familyName":"Jin","affiliation":[],"nameIdentifiers":[]},{"name":"Huang, Longbo","nameType":"Personal","givenName":"Longbo","familyName":"Huang","affiliation":[],"nameIdentifiers":[]},{"name":"Luo, Haipeng","nameType":"Personal","givenName":"Haipeng","familyName":"Luo","affiliation":[],"nameIdentifiers":[]}],"titles":[{"title":"The best of both worlds: stochastic and adversarial episodic MDPs with unknown transition"}],"publisher":"arXiv","container":{},"publicationYear":2021,"subjects":[{"lang":"en","subject":"Machine Learning (cs.LG)","subjectScheme":"arXiv"},{"subject":"FOS: Computer and information sciences","subjectScheme":"Fields of Science and Technology (FOS)"},{"subject":"FOS: Computer and information sciences","schemeUri":"http://www.oecd.org/science/inno/38235147.pdf","subjectScheme":"Fields of Science and Technology (FOS)"},{"lang":"en","subject":"I.2.6","subjectScheme":"ACM"},{"lang":"en","subject":"I.2.6","subjectScheme":"MSC"}],"contributors":[],"dates":[{"date":"2021-06-08T05:46:35Z","dateType":"Submitted","dateInformation":"v1"},{"date":"2021-06-09T00:12:13Z","dateType":"Updated","dateInformation":"v1"},{"date":"2021-10-17T07:02:46Z","dateType":"Submitted","dateInformation":"v2"},{"date":"2021-10-19T00:17:36Z","dateType":"Updated","dateInformation":"v2"},{"date":"2021-06","dateType":"Available","dateInformation":"v1"},{"date":"2021","dateType":"Issued"}],"language":null,"types":{"ris":"GEN","bibtex":"misc","citeproc":"article","schemaOrg":"CreativeWork","resourceType":"Article","resourceTypeGeneral":"Preprint"},"relatedIdentifiers":[],"relatedItems":[],"sizes":[],"formats":[],"version":"2","rightsList":[{"rights":"arXiv.org perpetual, non-exclusive license","rightsUri":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/"}],"descriptions":[{"description":"We consider the best-of-both-worlds problem for learning an episodic Markov Decision Process through $T$ episodes, with the goal of achieving $\\widetilde{\\mathcal{O}}(\\sqrt{T})$ regret when the losses are adversarial and simultaneously $\\mathcal{O}(\\text{polylog}(T))$ regret when the losses are (almost) stochastic. Recent work by [Jin and Luo, 2020] achieves this goal when the fixed transition is known, and leaves the case of unknown transition as a major open question. In this work, we resolve this open problem by using the same Follow-the-Regularized-Leader ($\\text{FTRL}$) framework together with a set of new techniques. Specifically, we first propose a loss-shifting trick in the $\\text{FTRL}$ analysis, which greatly simplifies the approach of [Jin and Luo, 2020] and already improves their results for the known transition case. Then, we extend this idea to the unknown transition case and develop a novel analysis which upper bounds the transition estimation error by (a fraction of) the regret itself in the stochastic setting, a key property to ensure $\\mathcal{O}(\\text{polylog}(T))$ regret.","descriptionType":"Abstract"},{"description":"The camera-ready version for NeurIPS 2021","descriptionType":"Other"}],"geoLocations":[],"fundingReferences":[],"xml":"PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0idXRmLTgiPz4KPHJlc291cmNlIHhtbG5zPSJodHRwOi8vZGF0YWNpdGUub3JnL3NjaGVtYS9rZXJuZWwtNCIgeG1sbnM6eHNpPSJodHRwOi8vd3d3LnczLm9yZy8yMDAxL1hNTFNjaGVtYS1pbnN0YW5jZSIgeHNpOnNjaGVtYUxvY2F0aW9uPSJodHRwOi8vZGF0YWNpdGUub3JnL3NjaGVtYS9rZXJuZWwtNCBodHRwOi8vc2NoZW1hLmRhdGFjaXRlLm9yZy9tZXRhL2tlcm5lbC00LjMvbWV0YWRhdGEueHNkIj4KICA8aWRlbnRpZmllciBpZGVudGlmaWVyVHlwZT0iRE9JIj4xMC40ODU1MC9BUlhJVi4yMTA2LjA0MTE3PC9pZGVudGlmaWVyPgogIDxhbHRlcm5hdGVJZGVudGlmaWVycz4KICAgIDxhbHRlcm5hdGVJZGVudGlmaWVyIGFsdGVybmF0ZUlkZW50aWZpZXJUeXBlPSJhclhpdiI+MjEwNi4wNDExNzwvYWx0ZXJuYXRlSWRlbnRpZmllcj4KICA8L2FsdGVybmF0ZUlkZW50aWZpZXJzPgogIDxjcmVhdG9ycz4KICAgIDxjcmVhdG9yPgogICAgICA8Y3JlYXRvck5hbWUgbmFtZVR5cGU9IlBlcnNvbmFsIj5KaW4sIFRpYW5jaGVuZzwvY3JlYXRvck5hbWU+CiAgICAgIDxnaXZlbk5hbWU+VGlhbmNoZW5nPC9naXZlbk5hbWU+CiAgICAgIDxmYW1pbHlOYW1lPkppbjwvZmFtaWx5TmFtZT4KICAgIDwvY3JlYXRvcj4KICAgIDxjcmVhdG9yPgogICAgICA8Y3JlYXRvck5hbWUgbmFtZVR5cGU9IlBlcnNvbmFsIj5IdWFuZywgTG9uZ2JvPC9jcmVhdG9yTmFtZT4KICAgICAgPGdpdmVuTmFtZT5Mb25nYm88L2dpdmVuTmFtZT4KICAgICAgPGZhbWlseU5hbWU+SHVhbmc8L2ZhbWlseU5hbWU+CiAgICA8L2NyZWF0b3I+CiAgICA8Y3JlYXRvcj4KICAgICAgPGNyZWF0b3JOYW1lIG5hbWVUeXBlPSJQZXJzb25hbCI+THVvLCBIYWlwZW5nPC9jcmVhdG9yTmFtZT4KICAgICAgPGdpdmVuTmFtZT5IYWlwZW5nPC9naXZlbk5hbWU+CiAgICAgIDxmYW1pbHlOYW1lPkx1bzwvZmFtaWx5TmFtZT4KICAgIDwvY3JlYXRvcj4KICA8L2NyZWF0b3JzPgogIDx0aXRsZXM+CiAgICA8dGl0bGU+VGhlIGJlc3Qgb2YgYm90aCB3b3JsZHM6IHN0b2NoYXN0aWMgYW5kIGFkdmVyc2FyaWFsIGVwaXNvZGljIE1EUHMgd2l0aCB1bmtub3duIHRyYW5zaXRpb248L3RpdGxlPgogIDwvdGl0bGVzPgogIDxwdWJsaXNoZXI+YXJYaXY8L3B1Ymxpc2hlcj4KICA8cHVibGljYXRpb25ZZWFyPjIwMjE8L3B1YmxpY2F0aW9uWWVhcj4KICA8c3ViamVjdHM+CiAgICA8c3ViamVjdCB4bWw6bGFuZz0iZW4iIHN1YmplY3RTY2hlbWU9ImFyWGl2Ij5NYWNoaW5lIExlYXJuaW5nIChjcy5MRyk8L3N1YmplY3Q+CiAgICA8c3ViamVjdCBzdWJqZWN0U2NoZW1lPSJGaWVsZHMgb2YgU2NpZW5jZSBhbmQgVGVjaG5vbG9neSAoRk9TKSI+Rk9TOiBDb21wdXRlciBhbmQgaW5mb3JtYXRpb24gc2NpZW5jZXM8L3N1YmplY3Q+CiAgICA8c3ViamVjdCB4bWw6bGFuZz0iZW4iIHN1YmplY3RTY2hlbWU9IkFDTSI+SS4yLjY8L3N1YmplY3Q+CiAgICA8c3ViamVjdCB4bWw6bGFuZz0iZW4iIHN1YmplY3RTY2hlbWU9Ik1TQyI+SS4yLjY8L3N1YmplY3Q+CiAgPC9zdWJqZWN0cz4KICA8ZGF0ZXM+CiAgICA8ZGF0ZSBkYXRlVHlwZT0iU3VibWl0dGVkIiBkYXRlSW5mb3JtYXRpb249InYxIj4yMDIxLTA2LTA4VDA1OjQ2OjM1WjwvZGF0ZT4KICAgIDxkYXRlIGRhdGVUeXBlPSJVcGRhdGVkIiBkYXRlSW5mb3JtYXRpb249InYxIj4yMDIxLTA2LTA5VDAwOjEyOjEzWjwvZGF0ZT4KICAgIDxkYXRlIGRhdGVUeXBlPSJTdWJtaXR0ZWQiIGRhdGVJbmZvcm1hdGlvbj0idjIiPjIwMjEtMTAtMTdUMDc6MDI6NDZaPC9kYXRlPgogICAgPGRhdGUgZGF0ZVR5cGU9IlVwZGF0ZWQiIGRhdGVJbmZvcm1hdGlvbj0idjIiPjIwMjEtMTAtMTlUMDA6MTc6MzZaPC9kYXRlPgogICAgPGRhdGUgZGF0ZVR5cGU9IkF2YWlsYWJsZSIgZGF0ZUluZm9ybWF0aW9uPSJ2MSI+MjAyMS0wNjwvZGF0ZT4KICA8L2RhdGVzPgogIDxyZXNvdXJjZVR5cGUgcmVzb3VyY2VUeXBlR2VuZXJhbD0iUHJlcHJpbnQiPkFydGljbGU8L3Jlc291cmNlVHlwZT4KICA8dmVyc2lvbj4yPC92ZXJzaW9uPgogIDxyaWdodHNMaXN0PgogICAgPHJpZ2h0cyByaWdodHNVUkk9Imh0dHA6Ly9hcnhpdi5vcmcvbGljZW5zZXMvbm9uZXhjbHVzaXZlLWRpc3RyaWIvMS4wLyI+YXJYaXYub3JnIHBlcnBldHVhbCwgbm9uLWV4Y2x1c2l2ZSBsaWNlbnNlPC9yaWdodHM+CiAgPC9yaWdodHNMaXN0PgogIDxkZXNjcmlwdGlvbnM+CiAgICA8ZGVzY3JpcHRpb24gZGVzY3JpcHRpb25UeXBlPSJBYnN0cmFjdCI+V2UgY29uc2lkZXIgdGhlIGJlc3Qtb2YtYm90aC13b3JsZHMgcHJvYmxlbSBmb3IgbGVhcm5pbmcgYW4gZXBpc29kaWMgTWFya292IERlY2lzaW9uIFByb2Nlc3MgdGhyb3VnaCAkVCQgZXBpc29kZXMsIHdpdGggdGhlIGdvYWwgb2YgYWNoaWV2aW5nICRcd2lkZXRpbGRle1xtYXRoY2Fse099fShcc3FydHtUfSkkIHJlZ3JldCB3aGVuIHRoZSBsb3NzZXMgYXJlIGFkdmVyc2FyaWFsIGFuZCBzaW11bHRhbmVvdXNseSAkXG1hdGhjYWx7T30oXHRleHR7cG9seWxvZ30oVCkpJCByZWdyZXQgd2hlbiB0aGUgbG9zc2VzIGFyZSAoYWxtb3N0KSBzdG9jaGFzdGljLiBSZWNlbnQgd29yayBieSBbSmluIGFuZCBMdW8sIDIwMjBdIGFjaGlldmVzIHRoaXMgZ29hbCB3aGVuIHRoZSBmaXhlZCB0cmFuc2l0aW9uIGlzIGtub3duLCBhbmQgbGVhdmVzIHRoZSBjYXNlIG9mIHVua25vd24gdHJhbnNpdGlvbiBhcyBhIG1ham9yIG9wZW4gcXVlc3Rpb24uIEluIHRoaXMgd29yaywgd2UgcmVzb2x2ZSB0aGlzIG9wZW4gcHJvYmxlbSBieSB1c2luZyB0aGUgc2FtZSBGb2xsb3ctdGhlLVJlZ3VsYXJpemVkLUxlYWRlciAoJFx0ZXh0e0ZUUkx9JCkgZnJhbWV3b3JrIHRvZ2V0aGVyIHdpdGggYSBzZXQgb2YgbmV3IHRlY2huaXF1ZXMuIFNwZWNpZmljYWxseSwgd2UgZmlyc3QgcHJvcG9zZSBhIGxvc3Mtc2hpZnRpbmcgdHJpY2sgaW4gdGhlICRcdGV4dHtGVFJMfSQgYW5hbHlzaXMsIHdoaWNoIGdyZWF0bHkgc2ltcGxpZmllcyB0aGUgYXBwcm9hY2ggb2YgW0ppbiBhbmQgTHVvLCAyMDIwXSBhbmQgYWxyZWFkeSBpbXByb3ZlcyB0aGVpciByZXN1bHRzIGZvciB0aGUga25vd24gdHJhbnNpdGlvbiBjYXNlLiBUaGVuLCB3ZSBleHRlbmQgdGhpcyBpZGVhIHRvIHRoZSB1bmtub3duIHRyYW5zaXRpb24gY2FzZSBhbmQgZGV2ZWxvcCBhIG5vdmVsIGFuYWx5c2lzIHdoaWNoIHVwcGVyIGJvdW5kcyB0aGUgdHJhbnNpdGlvbiBlc3RpbWF0aW9uIGVycm9yIGJ5IChhIGZyYWN0aW9uIG9mKSB0aGUgcmVncmV0IGl0c2VsZiBpbiB0aGUgc3RvY2hhc3RpYyBzZXR0aW5nLCBhIGtleSBwcm9wZXJ0eSB0byBlbnN1cmUgJFxtYXRoY2Fse099KFx0ZXh0e3BvbHlsb2d9KFQpKSQgcmVncmV0LjwvZGVzY3JpcHRpb24+CiAgICA8ZGVzY3JpcHRpb24gZGVzY3JpcHRpb25UeXBlPSJPdGhlciI+VGhlIGNhbWVyYS1yZWFkeSB2ZXJzaW9uIGZvciBOZXVySVBTIDIwMjE8L2Rlc2NyaXB0aW9uPgogIDwvZGVzY3JpcHRpb25zPgo8L3Jlc291cmNlPg==","url":"https://arxiv.org/abs/2106.04117","contentUrl":null,"metadataVersion":0,"schemaVersion":"http://datacite.org/schema/kernel-4","source":"mds","isActive":true,"state":"findable","reason":null,"viewCount":0,"viewsOverTime":[],"downloadCount":0,"downloadsOverTime":[],"referenceCount":0,"citationCount":0,"citationsOverTime":[],"partCount":0,"partOfCount":0,"versionCount":0,"versionOfCount":0,"created":"2022-02-23T13:47:01.000Z","registered":"2022-02-23T13:47:02.000Z","published":"2021","updated":"2022-02-23T13:47:02.000Z"},"relationships":{"client":{"data":{"id":"arxiv.content","type":"clients"}},"provider":{"data":{"id":"arxiv","type":"providers"}},"media":{"data":{"id":"10.48550/arxiv.2106.04117","type":"media"}},"references":{"data":[]},"citations":{"data":[]},"parts":{"data":[]},"partOf":{"data":[]},"versions":{"data":[]},"versionOf":{"data":[]}}}}