{"data":{"id":"10.48550/arxiv.1810.08264","type":"dois","attributes":{"doi":"10.48550/arxiv.1810.08264","prefix":"10.48550","suffix":"arxiv.1810.08264","identifiers":[{"identifier":"1810.08264","identifierType":"arXiv"}],"alternateIdentifiers":[{"alternateIdentifierType":"arXiv","alternateIdentifier":"1810.08264"}],"creators":[{"name":"Chen, Xi","nameType":"Personal","givenName":"Xi","familyName":"Chen","affiliation":[],"nameIdentifiers":[]},{"name":"Liu, Weidong","nameType":"Personal","givenName":"Weidong","familyName":"Liu","affiliation":[],"nameIdentifiers":[]},{"name":"Zhang, Yichen","nameType":"Personal","givenName":"Yichen","familyName":"Zhang","affiliation":[],"nameIdentifiers":[]}],"titles":[{"title":"Quantile Regression Under Memory Constraint"}],"publisher":"arXiv","container":{},"publicationYear":2018,"subjects":[{"lang":"en","subject":"Methodology (stat.ME)","subjectScheme":"arXiv"},{"lang":"en","subject":"Econometrics (econ.EM)","subjectScheme":"arXiv"},{"lang":"en","subject":"Machine Learning (stat.ML)","subjectScheme":"arXiv"},{"subject":"FOS: Computer and information sciences","subjectScheme":"Fields of Science and Technology (FOS)"},{"subject":"FOS: Computer and information sciences","schemeUri":"http://www.oecd.org/science/inno/38235147.pdf","subjectScheme":"Fields of Science and Technology (FOS)"},{"subject":"FOS: Economics and business","subjectScheme":"Fields of Science and Technology (FOS)"},{"subject":"FOS: Economics and business","schemeUri":"http://www.oecd.org/science/inno/38235147.pdf","subjectScheme":"Fields of Science and Technology (FOS)"}],"contributors":[],"dates":[{"date":"2018-10-18T20:03:51Z","dateType":"Submitted","dateInformation":"v1"},{"date":"2021-11-15T01:02:32Z","dateType":"Updated","dateInformation":"v1"},{"date":"2018-10","dateType":"Available","dateInformation":"v1"},{"date":"2018","dateType":"Issued"}],"language":null,"types":{"ris":"RPRT","bibtex":"article","citeproc":"article-journal","schemaOrg":"ScholarlyArticle","resourceType":"Article","resourceTypeGeneral":"Text"},"relatedIdentifiers":[{"relationType":"IsVersionOf","relatedIdentifier":"10.1214/18-aos1777","relatedIdentifierType":"DOI"}],"relatedItems":[],"sizes":[],"formats":[],"version":"1","rightsList":[{"rights":"arXiv.org perpetual, non-exclusive license","rightsUri":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/"}],"descriptions":[{"description":"This paper studies the inference problem in quantile regression (QR) for a large sample size $n$ but under a limited memory constraint, where the memory can only store a small batch of data of size $m$. A natural method is the naïve divide-and-conquer approach, which splits data into batches of size $m$, computes the local QR estimator for each batch, and then aggregates the estimators via averaging. However, this method only works when $n=o(m^2)$ and is computationally expensive. This paper proposes a computationally efficient method, which only requires an initial QR estimator on a small batch of data and then successively refines the estimator via multiple rounds of aggregations. Theoretically, as long as $n$ grows polynomially in $m$, we establish the asymptotic normality for the obtained estimator and show that our estimator with only a few rounds of aggregations achieves the same efficiency as the QR estimator computed on all the data. Moreover, our result allows the case that the dimensionality $p$ goes to infinity. The proposed method can also be applied to address the QR problem under distributed computing environment (e.g., in a large-scale sensor network) or for real-time streaming data.","descriptionType":"Abstract"}],"geoLocations":[],"fundingReferences":[],"xml":"PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0idXRmLTgiPz4KPHJlc291cmNlIHhtbG5zPSJodHRwOi8vZGF0YWNpdGUub3JnL3NjaGVtYS9rZXJuZWwtNCIgeG1sbnM6eHNpPSJodHRwOi8vd3d3LnczLm9yZy8yMDAxL1hNTFNjaGVtYS1pbnN0YW5jZSIgeHNpOnNjaGVtYUxvY2F0aW9uPSJodHRwOi8vZGF0YWNpdGUub3JnL3NjaGVtYS9rZXJuZWwtNCBodHRwOi8vc2NoZW1hLmRhdGFjaXRlLm9yZy9tZXRhL2tlcm5lbC00LjMvbWV0YWRhdGEueHNkIj4KICA8aWRlbnRpZmllciBpZGVudGlmaWVyVHlwZT0iRE9JIj4xMC40ODU1MC9BUlhJVi4xODEwLjA4MjY0PC9pZGVudGlmaWVyPgogIDxhbHRlcm5hdGVJZGVudGlmaWVycz4KICAgIDxhbHRlcm5hdGVJZGVudGlmaWVyIGFsdGVybmF0ZUlkZW50aWZpZXJUeXBlPSJhclhpdiI+MTgxMC4wODI2NDwvYWx0ZXJuYXRlSWRlbnRpZmllcj4KICA8L2FsdGVybmF0ZUlkZW50aWZpZXJzPgogIDxjcmVhdG9ycz4KICAgIDxjcmVhdG9yPgogICAgICA8Y3JlYXRvck5hbWUgbmFtZVR5cGU9IlBlcnNvbmFsIj5DaGVuLCBYaTwvY3JlYXRvck5hbWU+CiAgICAgIDxnaXZlbk5hbWU+WGk8L2dpdmVuTmFtZT4KICAgICAgPGZhbWlseU5hbWU+Q2hlbjwvZmFtaWx5TmFtZT4KICAgIDwvY3JlYXRvcj4KICAgIDxjcmVhdG9yPgogICAgICA8Y3JlYXRvck5hbWUgbmFtZVR5cGU9IlBlcnNvbmFsIj5MaXUsIFdlaWRvbmc8L2NyZWF0b3JOYW1lPgogICAgICA8Z2l2ZW5OYW1lPldlaWRvbmc8L2dpdmVuTmFtZT4KICAgICAgPGZhbWlseU5hbWU+TGl1PC9mYW1pbHlOYW1lPgogICAgPC9jcmVhdG9yPgogICAgPGNyZWF0b3I+CiAgICAgIDxjcmVhdG9yTmFtZSBuYW1lVHlwZT0iUGVyc29uYWwiPlpoYW5nLCBZaWNoZW48L2NyZWF0b3JOYW1lPgogICAgICA8Z2l2ZW5OYW1lPllpY2hlbjwvZ2l2ZW5OYW1lPgogICAgICA8ZmFtaWx5TmFtZT5aaGFuZzwvZmFtaWx5TmFtZT4KICAgIDwvY3JlYXRvcj4KICA8L2NyZWF0b3JzPgogIDx0aXRsZXM+CiAgICA8dGl0bGU+UXVhbnRpbGUgUmVncmVzc2lvbiBVbmRlciBNZW1vcnkgQ29uc3RyYWludDwvdGl0bGU+CiAgPC90aXRsZXM+CiAgPHB1Ymxpc2hlcj5hclhpdjwvcHVibGlzaGVyPgogIDxwdWJsaWNhdGlvblllYXI+MjAxODwvcHVibGljYXRpb25ZZWFyPgogIDxzdWJqZWN0cz4KICAgIDxzdWJqZWN0IHhtbDpsYW5nPSJlbiIgc3ViamVjdFNjaGVtZT0iYXJYaXYiPk1ldGhvZG9sb2d5IChzdGF0Lk1FKTwvc3ViamVjdD4KICAgIDxzdWJqZWN0IHhtbDpsYW5nPSJlbiIgc3ViamVjdFNjaGVtZT0iYXJYaXYiPkVjb25vbWV0cmljcyAoZWNvbi5FTSk8L3N1YmplY3Q+CiAgICA8c3ViamVjdCB4bWw6bGFuZz0iZW4iIHN1YmplY3RTY2hlbWU9ImFyWGl2Ij5NYWNoaW5lIExlYXJuaW5nIChzdGF0Lk1MKTwvc3ViamVjdD4KICAgIDxzdWJqZWN0IHN1YmplY3RTY2hlbWU9IkZpZWxkcyBvZiBTY2llbmNlIGFuZCBUZWNobm9sb2d5IChGT1MpIj5GT1M6IENvbXB1dGVyIGFuZCBpbmZvcm1hdGlvbiBzY2llbmNlczwvc3ViamVjdD4KICAgIDxzdWJqZWN0IHN1YmplY3RTY2hlbWU9IkZpZWxkcyBvZiBTY2llbmNlIGFuZCBUZWNobm9sb2d5IChGT1MpIj5GT1M6IEVjb25vbWljcyBhbmQgYnVzaW5lc3M8L3N1YmplY3Q+CiAgPC9zdWJqZWN0cz4KICA8ZGF0ZXM+CiAgICA8ZGF0ZSBkYXRlVHlwZT0iU3VibWl0dGVkIiBkYXRlSW5mb3JtYXRpb249InYxIj4yMDE4LTEwLTE4VDIwOjAzOjUxWjwvZGF0ZT4KICAgIDxkYXRlIGRhdGVUeXBlPSJVcGRhdGVkIiBkYXRlSW5mb3JtYXRpb249InYxIj4yMDIxLTExLTE1VDAxOjAyOjMyWjwvZGF0ZT4KICAgIDxkYXRlIGRhdGVUeXBlPSJBdmFpbGFibGUiIGRhdGVJbmZvcm1hdGlvbj0idjEiPjIwMTgtMTA8L2RhdGU+CiAgPC9kYXRlcz4KICA8cmVzb3VyY2VUeXBlIHJlc291cmNlVHlwZUdlbmVyYWw9IlRleHQiPkFydGljbGU8L3Jlc291cmNlVHlwZT4KICA8cmVsYXRlZElkZW50aWZpZXJzPgogICAgPHJlbGF0ZWRJZGVudGlmaWVyIHJlbGF0ZWRJZGVudGlmaWVyVHlwZT0iRE9JIiByZWxhdGlvblR5cGU9IklzVmVyc2lvbk9mIj4xMC4xMjE0LzE4LUFPUzE3Nzc8L3JlbGF0ZWRJZGVudGlmaWVyPgogIDwvcmVsYXRlZElkZW50aWZpZXJzPgogIDx2ZXJzaW9uPjE8L3ZlcnNpb24+CiAgPHJpZ2h0c0xpc3Q+CiAgICA8cmlnaHRzIHJpZ2h0c1VSST0iaHR0cDovL2FyeGl2Lm9yZy9saWNlbnNlcy9ub25leGNsdXNpdmUtZGlzdHJpYi8xLjAvIj5hclhpdi5vcmcgcGVycGV0dWFsLCBub24tZXhjbHVzaXZlIGxpY2Vuc2U8L3JpZ2h0cz4KICA8L3JpZ2h0c0xpc3Q+CiAgPGRlc2NyaXB0aW9ucz4KICAgIDxkZXNjcmlwdGlvbiBkZXNjcmlwdGlvblR5cGU9IkFic3RyYWN0Ij5UaGlzIHBhcGVyIHN0dWRpZXMgdGhlIGluZmVyZW5jZSBwcm9ibGVtIGluIHF1YW50aWxlIHJlZ3Jlc3Npb24gKFFSKSBmb3IgYSBsYXJnZSBzYW1wbGUgc2l6ZSAkbiQgYnV0IHVuZGVyIGEgbGltaXRlZCBtZW1vcnkgY29uc3RyYWludCwgd2hlcmUgdGhlIG1lbW9yeSBjYW4gb25seSBzdG9yZSBhIHNtYWxsIGJhdGNoIG9mIGRhdGEgb2Ygc2l6ZSAkbSQuIEEgbmF0dXJhbCBtZXRob2QgaXMgdGhlIG5hw692ZSBkaXZpZGUtYW5kLWNvbnF1ZXIgYXBwcm9hY2gsIHdoaWNoIHNwbGl0cyBkYXRhIGludG8gYmF0Y2hlcyBvZiBzaXplICRtJCwgY29tcHV0ZXMgdGhlIGxvY2FsIFFSIGVzdGltYXRvciBmb3IgZWFjaCBiYXRjaCwgYW5kIHRoZW4gYWdncmVnYXRlcyB0aGUgZXN0aW1hdG9ycyB2aWEgYXZlcmFnaW5nLiBIb3dldmVyLCB0aGlzIG1ldGhvZCBvbmx5IHdvcmtzIHdoZW4gJG49byhtXjIpJCBhbmQgaXMgY29tcHV0YXRpb25hbGx5IGV4cGVuc2l2ZS4gVGhpcyBwYXBlciBwcm9wb3NlcyBhIGNvbXB1dGF0aW9uYWxseSBlZmZpY2llbnQgbWV0aG9kLCB3aGljaCBvbmx5IHJlcXVpcmVzIGFuIGluaXRpYWwgUVIgZXN0aW1hdG9yIG9uIGEgc21hbGwgYmF0Y2ggb2YgZGF0YSBhbmQgdGhlbiBzdWNjZXNzaXZlbHkgcmVmaW5lcyB0aGUgZXN0aW1hdG9yIHZpYSBtdWx0aXBsZSByb3VuZHMgb2YgYWdncmVnYXRpb25zLiBUaGVvcmV0aWNhbGx5LCBhcyBsb25nIGFzICRuJCBncm93cyBwb2x5bm9taWFsbHkgaW4gJG0kLCB3ZSBlc3RhYmxpc2ggdGhlIGFzeW1wdG90aWMgbm9ybWFsaXR5IGZvciB0aGUgb2J0YWluZWQgZXN0aW1hdG9yIGFuZCBzaG93IHRoYXQgb3VyIGVzdGltYXRvciB3aXRoIG9ubHkgYSBmZXcgcm91bmRzIG9mIGFnZ3JlZ2F0aW9ucyBhY2hpZXZlcyB0aGUgc2FtZSBlZmZpY2llbmN5IGFzIHRoZSBRUiBlc3RpbWF0b3IgY29tcHV0ZWQgb24gYWxsIHRoZSBkYXRhLiBNb3Jlb3Zlciwgb3VyIHJlc3VsdCBhbGxvd3MgdGhlIGNhc2UgdGhhdCB0aGUgZGltZW5zaW9uYWxpdHkgJHAkIGdvZXMgdG8gaW5maW5pdHkuIFRoZSBwcm9wb3NlZCBtZXRob2QgY2FuIGFsc28gYmUgYXBwbGllZCB0byBhZGRyZXNzIHRoZSBRUiBwcm9ibGVtIHVuZGVyIGRpc3RyaWJ1dGVkIGNvbXB1dGluZyBlbnZpcm9ubWVudCAoZS5nLiwgaW4gYSBsYXJnZS1zY2FsZSBzZW5zb3IgbmV0d29yaykgb3IgZm9yIHJlYWwtdGltZSBzdHJlYW1pbmcgZGF0YS48L2Rlc2NyaXB0aW9uPgogIDwvZGVzY3JpcHRpb25zPgo8L3Jlc291cmNlPg==","url":"https://arxiv.org/abs/1810.08264","contentUrl":null,"metadataVersion":0,"schemaVersion":"http://datacite.org/schema/kernel-4","source":"mds","isActive":true,"state":"findable","reason":null,"viewCount":0,"viewsOverTime":[],"downloadCount":0,"downloadsOverTime":[],"referenceCount":0,"citationCount":0,"citationsOverTime":[],"partCount":0,"partOfCount":0,"versionCount":0,"versionOfCount":0,"created":"2022-03-01T08:37:00.000Z","registered":"2022-03-01T08:37:01.000Z","published":"2018","updated":"2022-03-01T08:37:01.000Z"},"relationships":{"client":{"data":{"id":"arxiv.content","type":"clients"}},"provider":{"data":{"id":"arxiv","type":"providers"}},"media":{"data":{"id":"10.48550/arxiv.1810.08264","type":"media"}},"references":{"data":[]},"citations":{"data":[]},"parts":{"data":[]},"partOf":{"data":[]},"versions":{"data":[]},"versionOf":{"data":[]}}}}