{"data":{"id":"10.6084/m9.figshare.c.2054504.v1","type":"dois","attributes":{"doi":"10.6084/m9.figshare.c.2054504.v1","prefix":"10.6084","suffix":"m9.figshare.c.2054504.v1","identifiers":[],"alternateIdentifiers":[],"creators":[{"name":"Yu, Bin","givenName":"Bin","familyName":"Yu","affiliation":[],"nameIdentifiers":[]},{"name":"Lim, Chinghway","givenName":"Chinghway","familyName":"Lim","affiliation":[],"nameIdentifiers":[]}],"titles":[{"title":"Estimation Stability with Cross Validation (ESCV)"}],"publisher":"Taylor \u0026 Francis","container":{},"publicationYear":2015,"subjects":[{"subject":"Plant Biology"},{"subject":"FOS: Biological sciences","schemeUri":"http://www.oecd.org/science/inno/38235147.pdf","subjectScheme":"Fields of Science and Technology (FOS)"},{"subject":"Cancer"},{"subject":"Mathematics"},{"subject":"FOS: Mathematics","schemeUri":"http://www.oecd.org/science/inno/38235147.pdf","subjectScheme":"Fields of Science and Technology (FOS)"},{"subject":"Information and Computing Sciences"},{"subject":"Biological Sciences"},{"subject":"Biotechnology"},{"subject":"Genetics"}],"contributors":[],"dates":[{"date":"2015-12-05","dateType":"Created"},{"date":"2019-08-19","dateType":"Updated"},{"date":"2015","dateType":"Issued"}],"language":null,"types":{"ris":"GEN","bibtex":"misc","citeproc":"article","schemaOrg":"Collection","resourceType":"Collection","resourceTypeGeneral":"Collection"},"relatedIdentifiers":[{"relationType":"IsPreviousVersionOf","relatedIdentifier":"10.6084/m9.figshare.c.2054504","relatedIdentifierType":"DOI"}],"relatedItems":[],"sizes":[],"formats":[],"version":null,"rightsList":[{"rights":"Creative Commons Attribution 4.0 International","rightsUri":"https://creativecommons.org/licenses/by/4.0/legalcode","schemeUri":"https://spdx.org/licenses/","rightsIdentifier":"cc-by-4.0","rightsIdentifierScheme":"SPDX"}],"descriptions":[{"description":"Cross-validation (\u003ci\u003eCV\u003c/i\u003e) is often used to select the regularization parameter in high dimensional problems. However, when applied to the sparse modeling method Lasso, \u003ci\u003eCV\u003c/i\u003e leads to models that are unstable in high-dimensions, and consequently not suited for reliable interpretation. In this paper, we propose a model-free criterion \u003ci\u003eESCV\u003c/i\u003e based on a new \u003ci\u003eestimation stability (ES)\u003c/i\u003e metric and \u003ci\u003eCV\u003c/i\u003e. Our proposed \u003ci\u003eESCV\u003c/i\u003e finds a smaller and locally \u003ci\u003eES\u003c/i\u003e-optimal model smaller than the \u003ci\u003eCV\u003c/i\u003e choice so that the it fits the data and also enjoys estimation stability property. We demonstrate that \u003ci\u003eESCV\u003c/i\u003e is an effective alternative to \u003ci\u003eCV\u003c/i\u003e at a similar easily parallelizable computational cost. In particular, we compare the two approaches with respect to several performance measures when applied to the Lasso on both simulated and real data sets. For dependent predictors common in practice, our main finding is that, \u003ci\u003eESCV\u003c/i\u003e cuts down false positive rates often by a large margin, while sacrificing little of true positive rates. \u003ci\u003eESCV\u003c/i\u003e usually outperforms \u003ci\u003eCV\u003c/i\u003e in terms of parameter estimation while giving similar performance as \u003ci\u003eCV\u003c/i\u003e in terms of prediction. For the two real data sets from neuroscience and cell biology, the models found by \u003ci\u003eESCV\u003c/i\u003e are less than half of the model sizes by \u003ci\u003eCV\u003c/i\u003e, but preserves \u003ci\u003eCV\u003c/i\u003e's predictive performance and corroborates with subject knowledge and independent work. We also discuss some regularization parameter alignment issues that come up in both approaches. Supplementary materials are available online.","descriptionType":"Abstract"}],"geoLocations":[],"fundingReferences":[],"xml":"PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiPz4KPHJlc291cmNlIHhtbG5zPSJodHRwOi8vZGF0YWNpdGUub3JnL3NjaGVtYS9rZXJuZWwtNCIgeG1sbnM6eHNpPSJodHRwOi8vd3d3LnczLm9yZy8yMDAxL1hNTFNjaGVtYS1pbnN0YW5jZSIgeHNpOnNjaGVtYUxvY2F0aW9uPSJodHRwOi8vZGF0YWNpdGUub3JnL3NjaGVtYS9rZXJuZWwtNCBodHRwOi8vc2NoZW1hLmRhdGFjaXRlLm9yZy9tZXRhL2tlcm5lbC00LjQvbWV0YWRhdGEueHNkIj4KICA8aWRlbnRpZmllciBpZGVudGlmaWVyVHlwZT0iRE9JIj4xMC42MDg0L005LkZJR1NIQVJFLkMuMjA1NDUwNC5WMTwvaWRlbnRpZmllcj4KICA8Y3JlYXRvcnM+CiAgICA8Y3JlYXRvcj4KICAgICAgPGNyZWF0b3JOYW1lPkJpbiBZdTwvY3JlYXRvck5hbWU+CiAgICAgIDxnaXZlbk5hbWU+QmluPC9naXZlbk5hbWU+CiAgICAgIDxmYW1pbHlOYW1lPll1PC9mYW1pbHlOYW1lPgogICAgPC9jcmVhdG9yPgogICAgPGNyZWF0b3I+CiAgICAgIDxjcmVhdG9yTmFtZT5DaGluZ2h3YXkgTGltPC9jcmVhdG9yTmFtZT4KICAgICAgPGdpdmVuTmFtZT5DaGluZ2h3YXk8L2dpdmVuTmFtZT4KICAgICAgPGZhbWlseU5hbWU+TGltPC9mYW1pbHlOYW1lPgogICAgPC9jcmVhdG9yPgogIDwvY3JlYXRvcnM+CiAgPHRpdGxlcz4KICAgIDx0aXRsZT5Fc3RpbWF0aW9uIFN0YWJpbGl0eSB3aXRoIENyb3NzIFZhbGlkYXRpb24gKEVTQ1YpPC90aXRsZT4KICA8L3RpdGxlcz4KICA8ZGVzY3JpcHRpb25zPgogICAgPGRlc2NyaXB0aW9uIGRlc2NyaXB0aW9uVHlwZT0iQWJzdHJhY3QiPiZsdDtkaXYmZ3Q7Jmx0O3AmZ3Q7Q3Jvc3MtdmFsaWRhdGlvbiAoJmx0O2kmZ3Q7Q1YmbHQ7L2kmZ3Q7KSBpcyBvZnRlbiB1c2VkIHRvIHNlbGVjdCB0aGUgcmVndWxhcml6YXRpb24gcGFyYW1ldGVyIGluIGhpZ2ggZGltZW5zaW9uYWwgcHJvYmxlbXMuIEhvd2V2ZXIsIHdoZW4gYXBwbGllZCB0byB0aGUgc3BhcnNlIG1vZGVsaW5nIG1ldGhvZCBMYXNzbywgJmx0O2kmZ3Q7Q1YmbHQ7L2kmZ3Q7IGxlYWRzIHRvIG1vZGVscyB0aGF0IGFyZSB1bnN0YWJsZSBpbiBoaWdoLWRpbWVuc2lvbnMsIGFuZCBjb25zZXF1ZW50bHkgbm90IHN1aXRlZCBmb3IgcmVsaWFibGUgaW50ZXJwcmV0YXRpb24uIEluIHRoaXMgcGFwZXIsIHdlIHByb3Bvc2UgYSBtb2RlbC1mcmVlIGNyaXRlcmlvbiAmbHQ7aSZndDtFU0NWJmx0Oy9pJmd0OyBiYXNlZCBvbiBhIG5ldyAmbHQ7aSZndDtlc3RpbWF0aW9uIHN0YWJpbGl0eSAoRVMpJmx0Oy9pJmd0OyBtZXRyaWMgYW5kICZsdDtpJmd0O0NWJmx0Oy9pJmd0Oy4gT3VyIHByb3Bvc2VkICZsdDtpJmd0O0VTQ1YmbHQ7L2kmZ3Q7IGZpbmRzIGEgc21hbGxlciBhbmQgbG9jYWxseSAmbHQ7aSZndDtFUyZsdDsvaSZndDstb3B0aW1hbCBtb2RlbCBzbWFsbGVyIHRoYW4gdGhlICZsdDtpJmd0O0NWJmx0Oy9pJmd0OyBjaG9pY2Ugc28gdGhhdCB0aGUgaXQgZml0cyB0aGUgZGF0YSBhbmQgYWxzbyBlbmpveXMgZXN0aW1hdGlvbiBzdGFiaWxpdHkgcHJvcGVydHkuIFdlIGRlbW9uc3RyYXRlIHRoYXQgJmx0O2kmZ3Q7RVNDViZsdDsvaSZndDsgaXMgYW4gZWZmZWN0aXZlIGFsdGVybmF0aXZlIHRvICZsdDtpJmd0O0NWJmx0Oy9pJmd0OyBhdCBhIHNpbWlsYXIgZWFzaWx5IHBhcmFsbGVsaXphYmxlIGNvbXB1dGF0aW9uYWwgY29zdC4gSW4gcGFydGljdWxhciwgd2UgY29tcGFyZSB0aGUgdHdvIGFwcHJvYWNoZXMgd2l0aCByZXNwZWN0IHRvIHNldmVyYWwgcGVyZm9ybWFuY2UgbWVhc3VyZXMgd2hlbiBhcHBsaWVkIHRvIHRoZSBMYXNzbyBvbiBib3RoIHNpbXVsYXRlZCBhbmQgcmVhbCBkYXRhIHNldHMuIEZvciBkZXBlbmRlbnQgcHJlZGljdG9ycyBjb21tb24gaW4gcHJhY3RpY2UsIG91ciBtYWluIGZpbmRpbmcgaXMgdGhhdCwgJmx0O2kmZ3Q7RVNDViZsdDsvaSZndDsgY3V0cyBkb3duIGZhbHNlIHBvc2l0aXZlIHJhdGVzIG9mdGVuIGJ5IGEgbGFyZ2UgbWFyZ2luLCB3aGlsZSBzYWNyaWZpY2luZyBsaXR0bGUgb2YgdHJ1ZSBwb3NpdGl2ZSByYXRlcy4gJmx0O2kmZ3Q7RVNDViZsdDsvaSZndDsgdXN1YWxseSBvdXRwZXJmb3JtcyAmbHQ7aSZndDtDViZsdDsvaSZndDsgaW4gdGVybXMgb2YgcGFyYW1ldGVyIGVzdGltYXRpb24gd2hpbGUgZ2l2aW5nIHNpbWlsYXIgcGVyZm9ybWFuY2UgYXMgJmx0O2kmZ3Q7Q1YmbHQ7L2kmZ3Q7IGluIHRlcm1zIG9mIHByZWRpY3Rpb24uIEZvciB0aGUgdHdvIHJlYWwgZGF0YSBzZXRzIGZyb20gbmV1cm9zY2llbmNlIGFuZCBjZWxsIGJpb2xvZ3ksIHRoZSBtb2RlbHMgZm91bmQgYnkgJmx0O2kmZ3Q7RVNDViZsdDsvaSZndDsgYXJlIGxlc3MgdGhhbiBoYWxmIG9mIHRoZSBtb2RlbCBzaXplcyBieSAmbHQ7aSZndDtDViZsdDsvaSZndDssIGJ1dCBwcmVzZXJ2ZXMgJmx0O2kmZ3Q7Q1YmbHQ7L2kmZ3Q7J3MgcHJlZGljdGl2ZSBwZXJmb3JtYW5jZSBhbmQgY29ycm9ib3JhdGVzIHdpdGggc3ViamVjdCBrbm93bGVkZ2UgYW5kIGluZGVwZW5kZW50IHdvcmsuIFdlIGFsc28gZGlzY3VzcyBzb21lIHJlZ3VsYXJpemF0aW9uIHBhcmFtZXRlciBhbGlnbm1lbnQgaXNzdWVzIHRoYXQgY29tZSB1cCBpbiBib3RoIGFwcHJvYWNoZXMuIFN1cHBsZW1lbnRhcnkgbWF0ZXJpYWxzIGFyZSBhdmFpbGFibGUgb25saW5lLiZsdDsvcCZndDsmbHQ7L2RpdiZndDs8L2Rlc2NyaXB0aW9uPgogIDwvZGVzY3JpcHRpb25zPgogIDxzdWJqZWN0cz4KICAgIDxzdWJqZWN0PlBsYW50IEJpb2xvZ3k8L3N1YmplY3Q+CiAgICA8c3ViamVjdD5DYW5jZXI8L3N1YmplY3Q+CiAgICA8c3ViamVjdD5NYXRoZW1hdGljczwvc3ViamVjdD4KICAgIDxzdWJqZWN0PkluZm9ybWF0aW9uIGFuZCBDb21wdXRpbmcgU2NpZW5jZXM8L3N1YmplY3Q+CiAgICA8c3ViamVjdD5CaW9sb2dpY2FsIFNjaWVuY2VzPC9zdWJqZWN0PgogICAgPHN1YmplY3Q+QmlvdGVjaG5vbG9neTwvc3ViamVjdD4KICAgIDxzdWJqZWN0PkdlbmV0aWNzPC9zdWJqZWN0PgogIDwvc3ViamVjdHM+CiAgPHB1Ymxpc2hlcj5UYXlsb3IgJmFtcDsgRnJhbmNpczwvcHVibGlzaGVyPgogIDxwdWJsaWNhdGlvblllYXI+MjAxNTwvcHVibGljYXRpb25ZZWFyPgogIDxkYXRlcz4KICAgIDxkYXRlIGRhdGVUeXBlPSJDcmVhdGVkIj4yMDE1LTEyLTA1PC9kYXRlPgogICAgPGRhdGUgZGF0ZVR5cGU9IlVwZGF0ZWQiPjIwMTktMDgtMTk8L2RhdGU+CiAgPC9kYXRlcz4KICA8cmVzb3VyY2VUeXBlIHJlc291cmNlVHlwZUdlbmVyYWw9IkNvbGxlY3Rpb24iPkNvbGxlY3Rpb248L3Jlc291cmNlVHlwZT4KICA8cmVsYXRlZElkZW50aWZpZXJzPgogICAgPHJlbGF0ZWRJZGVudGlmaWVyIHJlbGF0ZWRJZGVudGlmaWVyVHlwZT0iRE9JIiByZWxhdGlvblR5cGU9IklzUHJldmlvdXNWZXJzaW9uT2YiPjEwLjYwODQvbTkuZmlnc2hhcmUuYy4yMDU0NTA0PC9yZWxhdGVkSWRlbnRpZmllcj4KICA8L3JlbGF0ZWRJZGVudGlmaWVycz4KICA8cmlnaHRzTGlzdD4KICAgIDxyaWdodHMgcmlnaHRzVVJJPSJodHRwczovL2NyZWF0aXZlY29tbW9ucy5vcmcvbGljZW5zZXMvYnkvNC4wLyI+Q0MgQlkgNC4wPC9yaWdodHM+CiAgPC9yaWdodHNMaXN0Pgo8L3Jlc291cmNlPg==","url":"https://tandf.figshare.com/collections/Estimation_Stability_with_Cross_Validation_ESCV_/2054504/1","contentUrl":null,"metadataVersion":5,"schemaVersion":"http://datacite.org/schema/kernel-4","source":"mds","isActive":true,"state":"findable","reason":null,"viewCount":0,"viewsOverTime":[],"downloadCount":0,"downloadsOverTime":[],"referenceCount":0,"citationCount":0,"citationsOverTime":[],"partCount":0,"partOfCount":0,"versionCount":0,"versionOfCount":0,"created":"2015-12-05T19:55:08.000Z","registered":"2015-12-05T19:55:09.000Z","published":"2015","updated":"2024-03-21T08:41:15.000Z"},"relationships":{"client":{"data":{"id":"figshare.ars","type":"clients"}},"provider":{"data":{"id":"otjm","type":"providers"}},"media":{"data":{"id":"10.6084/m9.figshare.c.2054504.v1","type":"media"}},"references":{"data":[]},"citations":{"data":[]},"parts":{"data":[]},"partOf":{"data":[]},"versions":{"data":[]},"versionOf":{"data":[]}}}}