{"data":{"id":"10.48550/arxiv.2110.00115","type":"dois","attributes":{"doi":"10.48550/arxiv.2110.00115","prefix":"10.48550","suffix":"arxiv.2110.00115","identifiers":[{"identifier":"2110.00115","identifierType":"arXiv"}],"alternateIdentifiers":[{"alternateIdentifierType":"arXiv","alternateIdentifier":"2110.00115"}],"creators":[{"name":"Choe, Yo Joong","nameType":"Personal","givenName":"Yo Joong","familyName":"Choe","affiliation":[],"nameIdentifiers":[]},{"name":"Ramdas, Aaditya","nameType":"Personal","givenName":"Aaditya","familyName":"Ramdas","affiliation":[],"nameIdentifiers":[]}],"titles":[{"title":"Comparing Sequential Forecasters"}],"publisher":"arXiv","container":{},"publicationYear":2021,"subjects":[{"lang":"en","subject":"Methodology (stat.ME)","subjectScheme":"arXiv"},{"lang":"en","subject":"Machine Learning (cs.LG)","subjectScheme":"arXiv"},{"lang":"en","subject":"Statistics Theory (math.ST)","subjectScheme":"arXiv"},{"lang":"en","subject":"Applications (stat.AP)","subjectScheme":"arXiv"},{"lang":"en","subject":"Machine Learning (stat.ML)","subjectScheme":"arXiv"},{"subject":"FOS: Computer and information sciences","subjectScheme":"Fields of Science and Technology (FOS)"},{"subject":"FOS: Computer and information sciences","schemeUri":"http://www.oecd.org/science/inno/38235147.pdf","subjectScheme":"Fields of Science and Technology (FOS)"},{"subject":"FOS: Mathematics","subjectScheme":"Fields of Science and Technology (FOS)"},{"subject":"FOS: Mathematics","schemeUri":"http://www.oecd.org/science/inno/38235147.pdf","subjectScheme":"Fields of Science and Technology (FOS)"}],"contributors":[],"dates":[{"date":"2021-09-30T22:54:46Z","dateType":"Submitted","dateInformation":"v1"},{"date":"2021-10-04T00:05:53Z","dateType":"Updated","dateInformation":"v1"},{"date":"2021-11-04T15:13:18Z","dateType":"Submitted","dateInformation":"v2"},{"date":"2021-11-05T00:18:35Z","dateType":"Updated","dateInformation":"v2"},{"date":"2022-01-09T19:05:30Z","dateType":"Submitted","dateInformation":"v3"},{"date":"2022-01-11T01:23:38Z","dateType":"Updated","dateInformation":"v3"},{"date":"2022-12-14T04:34:34Z","dateType":"Submitted","dateInformation":"v4"},{"date":"2022-12-15T01:05:54Z","dateType":"Updated","dateInformation":"v4"},{"date":"2023-09-06T23:01:25Z","dateType":"Submitted","dateInformation":"v5"},{"date":"2023-09-08T00:07:46Z","dateType":"Updated","dateInformation":"v5"},{"date":"2023-11-09T05:09:52Z","dateType":"Submitted","dateInformation":"v6"},{"date":"2023-11-10T01:10:36Z","dateType":"Updated","dateInformation":"v6"},{"date":"2021-10","dateType":"Available","dateInformation":"v1"},{"date":"2021","dateType":"Issued"}],"language":null,"types":{"ris":"RPRT","bibtex":"article","citeproc":"article-journal","schemaOrg":"ScholarlyArticle","resourceType":"Article","resourceTypeGeneral":"Text"},"relatedIdentifiers":[{"relationType":"IsVersionOf","relatedIdentifier":"10.1287/opre.2021.0792","relatedIdentifierType":"DOI"}],"relatedItems":[],"sizes":[],"formats":[],"version":"6","rightsList":[{"rights":"Creative Commons Attribution 4.0 International","rightsUri":"https://creativecommons.org/licenses/by/4.0/legalcode","schemeUri":"https://spdx.org/licenses/","rightsIdentifier":"cc-by-4.0","rightsIdentifierScheme":"SPDX"}],"descriptions":[{"description":"Consider two forecasters, each making a single prediction for a sequence of events over time. We ask a relatively basic question: how might we compare these forecasters, either online or post-hoc, while avoiding unverifiable assumptions on how the forecasts and outcomes were generated? In this paper, we present a rigorous answer to this question by designing novel sequential inference procedures for estimating the time-varying difference in forecast scores. To do this, we employ confidence sequences (CS), which are sequences of confidence intervals that can be continuously monitored and are valid at arbitrary data-dependent stopping times (\"anytime-valid\"). The widths of our CSs are adaptive to the underlying variance of the score differences. Underlying their construction is a game-theoretic statistical framework, in which we further identify e-processes and p-processes for sequentially testing a weak null hypothesis -- whether one forecaster outperforms another on average (rather than always). Our methods do not make distributional assumptions on the forecasts or outcomes; our main theorems apply to any bounded scores, and we later provide alternative methods for unbounded scores. We empirically validate our approaches by comparing real-world baseball and weather forecasters.","descriptionType":"Abstract"},{"description":"Published in Operations Research. Code and data sources available at https://github.com/yjchoe/ComparingForecasters","descriptionType":"Other"}],"geoLocations":[],"fundingReferences":[],"xml":"PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0idXRmLTgiPz4KPHJlc291cmNlIHhtbG5zPSJodHRwOi8vZGF0YWNpdGUub3JnL3NjaGVtYS9rZXJuZWwtNCIgeG1sbnM6eHNpPSJodHRwOi8vd3d3LnczLm9yZy8yMDAxL1hNTFNjaGVtYS1pbnN0YW5jZSIgeHNpOnNjaGVtYUxvY2F0aW9uPSJodHRwOi8vZGF0YWNpdGUub3JnL3NjaGVtYS9rZXJuZWwtNCBodHRwOi8vc2NoZW1hLmRhdGFjaXRlLm9yZy9tZXRhL2tlcm5lbC00LjMvbWV0YWRhdGEueHNkIj4KICA8aWRlbnRpZmllciBpZGVudGlmaWVyVHlwZT0iRE9JIj4xMC40ODU1MC9BUlhJVi4yMTEwLjAwMTE1PC9pZGVudGlmaWVyPgogIDxhbHRlcm5hdGVJZGVudGlmaWVycz4KICAgIDxhbHRlcm5hdGVJZGVudGlmaWVyIGFsdGVybmF0ZUlkZW50aWZpZXJUeXBlPSJhclhpdiI+MjExMC4wMDExNTwvYWx0ZXJuYXRlSWRlbnRpZmllcj4KICA8L2FsdGVybmF0ZUlkZW50aWZpZXJzPgogIDxjcmVhdG9ycz4KICAgIDxjcmVhdG9yPgogICAgICA8Y3JlYXRvck5hbWUgbmFtZVR5cGU9IlBlcnNvbmFsIj5DaG9lLCBZbyBKb29uZzwvY3JlYXRvck5hbWU+CiAgICAgIDxnaXZlbk5hbWU+WW8gSm9vbmc8L2dpdmVuTmFtZT4KICAgICAgPGZhbWlseU5hbWU+Q2hvZTwvZmFtaWx5TmFtZT4KICAgIDwvY3JlYXRvcj4KICAgIDxjcmVhdG9yPgogICAgICA8Y3JlYXRvck5hbWUgbmFtZVR5cGU9IlBlcnNvbmFsIj5SYW1kYXMsIEFhZGl0eWE8L2NyZWF0b3JOYW1lPgogICAgICA8Z2l2ZW5OYW1lPkFhZGl0eWE8L2dpdmVuTmFtZT4KICAgICAgPGZhbWlseU5hbWU+UmFtZGFzPC9mYW1pbHlOYW1lPgogICAgPC9jcmVhdG9yPgogIDwvY3JlYXRvcnM+CiAgPHRpdGxlcz4KICAgIDx0aXRsZT5Db21wYXJpbmcgU2VxdWVudGlhbCBGb3JlY2FzdGVyczwvdGl0bGU+CiAgPC90aXRsZXM+CiAgPHB1Ymxpc2hlcj5hclhpdjwvcHVibGlzaGVyPgogIDxwdWJsaWNhdGlvblllYXI+MjAyMTwvcHVibGljYXRpb25ZZWFyPgogIDxzdWJqZWN0cz4KICAgIDxzdWJqZWN0IHhtbDpsYW5nPSJlbiIgc3ViamVjdFNjaGVtZT0iYXJYaXYiPk1ldGhvZG9sb2d5IChzdGF0Lk1FKTwvc3ViamVjdD4KICAgIDxzdWJqZWN0IHhtbDpsYW5nPSJlbiIgc3ViamVjdFNjaGVtZT0iYXJYaXYiPk1hY2hpbmUgTGVhcm5pbmcgKGNzLkxHKTwvc3ViamVjdD4KICAgIDxzdWJqZWN0IHhtbDpsYW5nPSJlbiIgc3ViamVjdFNjaGVtZT0iYXJYaXYiPlN0YXRpc3RpY3MgVGhlb3J5IChtYXRoLlNUKTwvc3ViamVjdD4KICAgIDxzdWJqZWN0IHhtbDpsYW5nPSJlbiIgc3ViamVjdFNjaGVtZT0iYXJYaXYiPkFwcGxpY2F0aW9ucyAoc3RhdC5BUCk8L3N1YmplY3Q+CiAgICA8c3ViamVjdCB4bWw6bGFuZz0iZW4iIHN1YmplY3RTY2hlbWU9ImFyWGl2Ij5NYWNoaW5lIExlYXJuaW5nIChzdGF0Lk1MKTwvc3ViamVjdD4KICAgIDxzdWJqZWN0IHN1YmplY3RTY2hlbWU9IkZpZWxkcyBvZiBTY2llbmNlIGFuZCBUZWNobm9sb2d5IChGT1MpIj5GT1M6IENvbXB1dGVyIGFuZCBpbmZvcm1hdGlvbiBzY2llbmNlczwvc3ViamVjdD4KICAgIDxzdWJqZWN0IHN1YmplY3RTY2hlbWU9IkZpZWxkcyBvZiBTY2llbmNlIGFuZCBUZWNobm9sb2d5IChGT1MpIj5GT1M6IE1hdGhlbWF0aWNzPC9zdWJqZWN0PgogIDwvc3ViamVjdHM+CiAgPGRhdGVzPgogICAgPGRhdGUgZGF0ZVR5cGU9IlN1Ym1pdHRlZCIgZGF0ZUluZm9ybWF0aW9uPSJ2MSI+MjAyMS0wOS0zMFQyMjo1NDo0Nlo8L2RhdGU+CiAgICA8ZGF0ZSBkYXRlVHlwZT0iVXBkYXRlZCIgZGF0ZUluZm9ybWF0aW9uPSJ2MSI+MjAyMS0xMC0wNFQwMDowNTo1M1o8L2RhdGU+CiAgICA8ZGF0ZSBkYXRlVHlwZT0iU3VibWl0dGVkIiBkYXRlSW5mb3JtYXRpb249InYyIj4yMDIxLTExLTA0VDE1OjEzOjE4WjwvZGF0ZT4KICAgIDxkYXRlIGRhdGVUeXBlPSJVcGRhdGVkIiBkYXRlSW5mb3JtYXRpb249InYyIj4yMDIxLTExLTA1VDAwOjE4OjM1WjwvZGF0ZT4KICAgIDxkYXRlIGRhdGVUeXBlPSJTdWJtaXR0ZWQiIGRhdGVJbmZvcm1hdGlvbj0idjMiPjIwMjItMDEtMDlUMTk6MDU6MzBaPC9kYXRlPgogICAgPGRhdGUgZGF0ZVR5cGU9IlVwZGF0ZWQiIGRhdGVJbmZvcm1hdGlvbj0idjMiPjIwMjItMDEtMTFUMDE6MjM6MzhaPC9kYXRlPgogICAgPGRhdGUgZGF0ZVR5cGU9IlN1Ym1pdHRlZCIgZGF0ZUluZm9ybWF0aW9uPSJ2NCI+MjAyMi0xMi0xNFQwNDozNDozNFo8L2RhdGU+CiAgICA8ZGF0ZSBkYXRlVHlwZT0iVXBkYXRlZCIgZGF0ZUluZm9ybWF0aW9uPSJ2NCI+MjAyMi0xMi0xNVQwMTowNTo1NFo8L2RhdGU+CiAgICA8ZGF0ZSBkYXRlVHlwZT0iU3VibWl0dGVkIiBkYXRlSW5mb3JtYXRpb249InY1Ij4yMDIzLTA5LTA2VDIzOjAxOjI1WjwvZGF0ZT4KICAgIDxkYXRlIGRhdGVUeXBlPSJVcGRhdGVkIiBkYXRlSW5mb3JtYXRpb249InY1Ij4yMDIzLTA5LTA4VDAwOjA3OjQ2WjwvZGF0ZT4KICAgIDxkYXRlIGRhdGVUeXBlPSJTdWJtaXR0ZWQiIGRhdGVJbmZvcm1hdGlvbj0idjYiPjIwMjMtMTEtMDlUMDU6MDk6NTJaPC9kYXRlPgogICAgPGRhdGUgZGF0ZVR5cGU9IlVwZGF0ZWQiIGRhdGVJbmZvcm1hdGlvbj0idjYiPjIwMjMtMTEtMTBUMDE6MTA6MzZaPC9kYXRlPgogICAgPGRhdGUgZGF0ZVR5cGU9IkF2YWlsYWJsZSIgZGF0ZUluZm9ybWF0aW9uPSJ2MSI+MjAyMS0xMDwvZGF0ZT4KICA8L2RhdGVzPgogIDxyZXNvdXJjZVR5cGUgcmVzb3VyY2VUeXBlR2VuZXJhbD0iVGV4dCI+QXJ0aWNsZTwvcmVzb3VyY2VUeXBlPgogIDxyZWxhdGVkSWRlbnRpZmllcnM+CiAgICA8cmVsYXRlZElkZW50aWZpZXIgcmVsYXRlZElkZW50aWZpZXJUeXBlPSJET0kiIHJlbGF0aW9uVHlwZT0iSXNWZXJzaW9uT2YiPjEwLjEyODcvb3ByZS4yMDIxLjA3OTI8L3JlbGF0ZWRJZGVudGlmaWVyPgogIDwvcmVsYXRlZElkZW50aWZpZXJzPgogIDx2ZXJzaW9uPjY8L3ZlcnNpb24+CiAgPHJpZ2h0c0xpc3Q+CiAgICA8cmlnaHRzIHJpZ2h0c1VSST0iaHR0cDovL2NyZWF0aXZlY29tbW9ucy5vcmcvbGljZW5zZXMvYnkvNC4wLyIgcmlnaHRzSWRlbnRpZmllclNjaGVtZT0iU1BEWCIgcmlnaHRzSWRlbnRpZmllcj0iQ0MtQlktNC4wIj5DcmVhdGl2ZSBDb21tb25zIEF0dHJpYnV0aW9uIDQuMCBJbnRlcm5hdGlvbmFsPC9yaWdodHM+CiAgPC9yaWdodHNMaXN0PgogIDxkZXNjcmlwdGlvbnM+CiAgICA8ZGVzY3JpcHRpb24gZGVzY3JpcHRpb25UeXBlPSJBYnN0cmFjdCI+Q29uc2lkZXIgdHdvIGZvcmVjYXN0ZXJzLCBlYWNoIG1ha2luZyBhIHNpbmdsZSBwcmVkaWN0aW9uIGZvciBhIHNlcXVlbmNlIG9mIGV2ZW50cyBvdmVyIHRpbWUuIFdlIGFzayBhIHJlbGF0aXZlbHkgYmFzaWMgcXVlc3Rpb246IGhvdyBtaWdodCB3ZSBjb21wYXJlIHRoZXNlIGZvcmVjYXN0ZXJzLCBlaXRoZXIgb25saW5lIG9yIHBvc3QtaG9jLCB3aGlsZSBhdm9pZGluZyB1bnZlcmlmaWFibGUgYXNzdW1wdGlvbnMgb24gaG93IHRoZSBmb3JlY2FzdHMgYW5kIG91dGNvbWVzIHdlcmUgZ2VuZXJhdGVkPyBJbiB0aGlzIHBhcGVyLCB3ZSBwcmVzZW50IGEgcmlnb3JvdXMgYW5zd2VyIHRvIHRoaXMgcXVlc3Rpb24gYnkgZGVzaWduaW5nIG5vdmVsIHNlcXVlbnRpYWwgaW5mZXJlbmNlIHByb2NlZHVyZXMgZm9yIGVzdGltYXRpbmcgdGhlIHRpbWUtdmFyeWluZyBkaWZmZXJlbmNlIGluIGZvcmVjYXN0IHNjb3Jlcy4gVG8gZG8gdGhpcywgd2UgZW1wbG95IGNvbmZpZGVuY2Ugc2VxdWVuY2VzIChDUyksIHdoaWNoIGFyZSBzZXF1ZW5jZXMgb2YgY29uZmlkZW5jZSBpbnRlcnZhbHMgdGhhdCBjYW4gYmUgY29udGludW91c2x5IG1vbml0b3JlZCBhbmQgYXJlIHZhbGlkIGF0IGFyYml0cmFyeSBkYXRhLWRlcGVuZGVudCBzdG9wcGluZyB0aW1lcyAoImFueXRpbWUtdmFsaWQiKS4gVGhlIHdpZHRocyBvZiBvdXIgQ1NzIGFyZSBhZGFwdGl2ZSB0byB0aGUgdW5kZXJseWluZyB2YXJpYW5jZSBvZiB0aGUgc2NvcmUgZGlmZmVyZW5jZXMuIFVuZGVybHlpbmcgdGhlaXIgY29uc3RydWN0aW9uIGlzIGEgZ2FtZS10aGVvcmV0aWMgc3RhdGlzdGljYWwgZnJhbWV3b3JrLCBpbiB3aGljaCB3ZSBmdXJ0aGVyIGlkZW50aWZ5IGUtcHJvY2Vzc2VzIGFuZCBwLXByb2Nlc3NlcyBmb3Igc2VxdWVudGlhbGx5IHRlc3RpbmcgYSB3ZWFrIG51bGwgaHlwb3RoZXNpcyAtLSB3aGV0aGVyIG9uZSBmb3JlY2FzdGVyIG91dHBlcmZvcm1zIGFub3RoZXIgb24gYXZlcmFnZSAocmF0aGVyIHRoYW4gYWx3YXlzKS4gT3VyIG1ldGhvZHMgZG8gbm90IG1ha2UgZGlzdHJpYnV0aW9uYWwgYXNzdW1wdGlvbnMgb24gdGhlIGZvcmVjYXN0cyBvciBvdXRjb21lczsgb3VyIG1haW4gdGhlb3JlbXMgYXBwbHkgdG8gYW55IGJvdW5kZWQgc2NvcmVzLCBhbmQgd2UgbGF0ZXIgcHJvdmlkZSBhbHRlcm5hdGl2ZSBtZXRob2RzIGZvciB1bmJvdW5kZWQgc2NvcmVzLiBXZSBlbXBpcmljYWxseSB2YWxpZGF0ZSBvdXIgYXBwcm9hY2hlcyBieSBjb21wYXJpbmcgcmVhbC13b3JsZCBiYXNlYmFsbCBhbmQgd2VhdGhlciBmb3JlY2FzdGVycy48L2Rlc2NyaXB0aW9uPgogICAgPGRlc2NyaXB0aW9uIGRlc2NyaXB0aW9uVHlwZT0iT3RoZXIiPlB1Ymxpc2hlZCBpbiBPcGVyYXRpb25zIFJlc2VhcmNoLiBDb2RlIGFuZCBkYXRhIHNvdXJjZXMgYXZhaWxhYmxlIGF0IGh0dHBzOi8vZ2l0aHViLmNvbS95amNob2UvQ29tcGFyaW5nRm9yZWNhc3RlcnM8L2Rlc2NyaXB0aW9uPgogIDwvZGVzY3JpcHRpb25zPgo8L3Jlc291cmNlPg==","url":"https://arxiv.org/abs/2110.00115","contentUrl":null,"metadataVersion":3,"schemaVersion":"http://datacite.org/schema/kernel-4","source":"mds","isActive":true,"state":"findable","reason":null,"viewCount":0,"viewsOverTime":[],"downloadCount":0,"downloadsOverTime":[],"referenceCount":0,"citationCount":0,"citationsOverTime":[],"partCount":0,"partOfCount":0,"versionCount":0,"versionOfCount":1,"created":"2022-02-21T16:30:15.000Z","registered":"2022-02-21T16:30:16.000Z","published":"2021","updated":"2023-11-10T02:31:21.000Z"},"relationships":{"client":{"data":{"id":"arxiv.content","type":"clients"}},"provider":{"data":{"id":"arxiv","type":"providers"}},"media":{"data":{"id":"10.48550/arxiv.2110.00115","type":"media"}},"references":{"data":[]},"citations":{"data":[]},"parts":{"data":[]},"partOf":{"data":[]},"versions":{"data":[]},"versionOf":{"data":[{"id":"10.1287/opre.2021.0792","type":"dois"}]}}}}