{"data":{"id":"10.48550/arxiv.1807.07838","type":"dois","attributes":{"doi":"10.48550/arxiv.1807.07838","prefix":"10.48550","suffix":"arxiv.1807.07838","identifiers":[{"identifier":"1807.07838","identifierType":"arXiv"}],"alternateIdentifiers":[{"alternateIdentifierType":"arXiv","alternateIdentifier":"1807.07838"}],"creators":[{"name":"Pendlebury, Feargus","nameType":"Personal","givenName":"Feargus","familyName":"Pendlebury","affiliation":[],"nameIdentifiers":[]},{"name":"Pierazzi, Fabio","nameType":"Personal","givenName":"Fabio","familyName":"Pierazzi","affiliation":[],"nameIdentifiers":[]},{"name":"Jordaney, Roberto","nameType":"Personal","givenName":"Roberto","familyName":"Jordaney","affiliation":[],"nameIdentifiers":[]},{"name":"Kinder, Johannes","nameType":"Personal","givenName":"Johannes","familyName":"Kinder","affiliation":[],"nameIdentifiers":[]},{"name":"Cavallaro, Lorenzo","nameType":"Personal","givenName":"Lorenzo","familyName":"Cavallaro","affiliation":[],"nameIdentifiers":[]}],"titles":[{"title":"TESSERACT: Eliminating Experimental Bias in Malware Classification across Space and Time"}],"publisher":"arXiv","container":{},"publicationYear":2018,"subjects":[{"lang":"en","subject":"Cryptography and Security (cs.CR)","subjectScheme":"arXiv"},{"lang":"en","subject":"Machine Learning (cs.LG)","subjectScheme":"arXiv"},{"subject":"FOS: Computer and information sciences","subjectScheme":"Fields of Science and Technology (FOS)"},{"subject":"FOS: Computer and information sciences","schemeUri":"http://www.oecd.org/science/inno/38235147.pdf","subjectScheme":"Fields of Science and Technology (FOS)"}],"contributors":[],"dates":[{"date":"2018-07-20T13:46:13Z","dateType":"Submitted","dateInformation":"v1"},{"date":"2018-07-23T00:09:59Z","dateType":"Updated","dateInformation":"v1"},{"date":"2019-02-25T17:05:53Z","dateType":"Submitted","dateInformation":"v2"},{"date":"2019-02-26T01:33:26Z","dateType":"Updated","dateInformation":"v2"},{"date":"2019-06-05T14:35:39Z","dateType":"Submitted","dateInformation":"v3"},{"date":"2019-06-06T00:17:30Z","dateType":"Updated","dateInformation":"v3"},{"date":"2019-09-12T14:03:44Z","dateType":"Submitted","dateInformation":"v4"},{"date":"2019-09-13T00:17:11Z","dateType":"Updated","dateInformation":"v4"},{"date":"2018-07","dateType":"Available","dateInformation":"v1"},{"date":"2018","dateType":"Issued"}],"language":null,"types":{"ris":"GEN","bibtex":"misc","citeproc":"article","schemaOrg":"CreativeWork","resourceType":"Article","resourceTypeGeneral":"Preprint"},"relatedIdentifiers":[],"relatedItems":[],"sizes":[],"formats":[],"version":"4","rightsList":[{"rights":"arXiv.org perpetual, non-exclusive license","rightsUri":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/"}],"descriptions":[{"description":"Is Android malware classification a solved problem? Published F1 scores of up to 0.99 appear to leave very little room for improvement. In this paper, we argue that results are commonly inflated due to two pervasive sources of experimental bias: \"spatial bias\" caused by distributions of training and testing data that are not representative of a real-world deployment; and \"temporal bias\" caused by incorrect time splits of training and testing sets, leading to impossible configurations. We propose a set of space and time constraints for experiment design that eliminates both sources of bias. We introduce a new metric that summarizes the expected robustness of a classifier in a real-world setting, and we present an algorithm to tune its performance. Finally, we demonstrate how this allows us to evaluate mitigation strategies for time decay such as active learning. We have implemented our solutions in TESSERACT, an open source evaluation framework for comparing malware classifiers in a realistic setting. We used TESSERACT to evaluate three Android malware classifiers from the literature on a dataset of 129K applications spanning over three years. Our evaluation confirms that earlier published results are biased, while also revealing counter-intuitive performance and showing that appropriate tuning can lead to significant improvements.","descriptionType":"Abstract"},{"description":"This arXiv version (v4) corresponds to the one published at USENIX Security Symposium 2019, with a fixed typo in Equation (4), which reported an extra normalization factor of (1/N). The results in the paper and the released implementation of the TESSERACT framework remain valid and correct as they rely on Python's numpy implementation of area under the curve","descriptionType":"Other"}],"geoLocations":[],"fundingReferences":[],"xml":"PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0idXRmLTgiPz4KPHJlc291cmNlIHhtbG5zPSJodHRwOi8vZGF0YWNpdGUub3JnL3NjaGVtYS9rZXJuZWwtNCIgeG1sbnM6eHNpPSJodHRwOi8vd3d3LnczLm9yZy8yMDAxL1hNTFNjaGVtYS1pbnN0YW5jZSIgeHNpOnNjaGVtYUxvY2F0aW9uPSJodHRwOi8vZGF0YWNpdGUub3JnL3NjaGVtYS9rZXJuZWwtNCBodHRwOi8vc2NoZW1hLmRhdGFjaXRlLm9yZy9tZXRhL2tlcm5lbC00LjMvbWV0YWRhdGEueHNkIj4KICA8aWRlbnRpZmllciBpZGVudGlmaWVyVHlwZT0iRE9JIj4xMC40ODU1MC9BUlhJVi4xODA3LjA3ODM4PC9pZGVudGlmaWVyPgogIDxhbHRlcm5hdGVJZGVudGlmaWVycz4KICAgIDxhbHRlcm5hdGVJZGVudGlmaWVyIGFsdGVybmF0ZUlkZW50aWZpZXJUeXBlPSJhclhpdiI+MTgwNy4wNzgzODwvYWx0ZXJuYXRlSWRlbnRpZmllcj4KICA8L2FsdGVybmF0ZUlkZW50aWZpZXJzPgogIDxjcmVhdG9ycz4KICAgIDxjcmVhdG9yPgogICAgICA8Y3JlYXRvck5hbWUgbmFtZVR5cGU9IlBlcnNvbmFsIj5QZW5kbGVidXJ5LCBGZWFyZ3VzPC9jcmVhdG9yTmFtZT4KICAgICAgPGdpdmVuTmFtZT5GZWFyZ3VzPC9naXZlbk5hbWU+CiAgICAgIDxmYW1pbHlOYW1lPlBlbmRsZWJ1cnk8L2ZhbWlseU5hbWU+CiAgICA8L2NyZWF0b3I+CiAgICA8Y3JlYXRvcj4KICAgICAgPGNyZWF0b3JOYW1lIG5hbWVUeXBlPSJQZXJzb25hbCI+UGllcmF6emksIEZhYmlvPC9jcmVhdG9yTmFtZT4KICAgICAgPGdpdmVuTmFtZT5GYWJpbzwvZ2l2ZW5OYW1lPgogICAgICA8ZmFtaWx5TmFtZT5QaWVyYXp6aTwvZmFtaWx5TmFtZT4KICAgIDwvY3JlYXRvcj4KICAgIDxjcmVhdG9yPgogICAgICA8Y3JlYXRvck5hbWUgbmFtZVR5cGU9IlBlcnNvbmFsIj5Kb3JkYW5leSwgUm9iZXJ0bzwvY3JlYXRvck5hbWU+CiAgICAgIDxnaXZlbk5hbWU+Um9iZXJ0bzwvZ2l2ZW5OYW1lPgogICAgICA8ZmFtaWx5TmFtZT5Kb3JkYW5leTwvZmFtaWx5TmFtZT4KICAgIDwvY3JlYXRvcj4KICAgIDxjcmVhdG9yPgogICAgICA8Y3JlYXRvck5hbWUgbmFtZVR5cGU9IlBlcnNvbmFsIj5LaW5kZXIsIEpvaGFubmVzPC9jcmVhdG9yTmFtZT4KICAgICAgPGdpdmVuTmFtZT5Kb2hhbm5lczwvZ2l2ZW5OYW1lPgogICAgICA8ZmFtaWx5TmFtZT5LaW5kZXI8L2ZhbWlseU5hbWU+CiAgICA8L2NyZWF0b3I+CiAgICA8Y3JlYXRvcj4KICAgICAgPGNyZWF0b3JOYW1lIG5hbWVUeXBlPSJQZXJzb25hbCI+Q2F2YWxsYXJvLCBMb3JlbnpvPC9jcmVhdG9yTmFtZT4KICAgICAgPGdpdmVuTmFtZT5Mb3JlbnpvPC9naXZlbk5hbWU+CiAgICAgIDxmYW1pbHlOYW1lPkNhdmFsbGFybzwvZmFtaWx5TmFtZT4KICAgIDwvY3JlYXRvcj4KICA8L2NyZWF0b3JzPgogIDx0aXRsZXM+CiAgICA8dGl0bGU+VEVTU0VSQUNUOiBFbGltaW5hdGluZyBFeHBlcmltZW50YWwgQmlhcyBpbiBNYWx3YXJlIENsYXNzaWZpY2F0aW9uIGFjcm9zcyBTcGFjZSBhbmQgVGltZTwvdGl0bGU+CiAgPC90aXRsZXM+CiAgPHB1Ymxpc2hlcj5hclhpdjwvcHVibGlzaGVyPgogIDxwdWJsaWNhdGlvblllYXI+MjAxODwvcHVibGljYXRpb25ZZWFyPgogIDxzdWJqZWN0cz4KICAgIDxzdWJqZWN0IHhtbDpsYW5nPSJlbiIgc3ViamVjdFNjaGVtZT0iYXJYaXYiPkNyeXB0b2dyYXBoeSBhbmQgU2VjdXJpdHkgKGNzLkNSKTwvc3ViamVjdD4KICAgIDxzdWJqZWN0IHhtbDpsYW5nPSJlbiIgc3ViamVjdFNjaGVtZT0iYXJYaXYiPk1hY2hpbmUgTGVhcm5pbmcgKGNzLkxHKTwvc3ViamVjdD4KICAgIDxzdWJqZWN0IHN1YmplY3RTY2hlbWU9IkZpZWxkcyBvZiBTY2llbmNlIGFuZCBUZWNobm9sb2d5IChGT1MpIj5GT1M6IENvbXB1dGVyIGFuZCBpbmZvcm1hdGlvbiBzY2llbmNlczwvc3ViamVjdD4KICA8L3N1YmplY3RzPgogIDxkYXRlcz4KICAgIDxkYXRlIGRhdGVUeXBlPSJTdWJtaXR0ZWQiIGRhdGVJbmZvcm1hdGlvbj0idjEiPjIwMTgtMDctMjBUMTM6NDY6MTNaPC9kYXRlPgogICAgPGRhdGUgZGF0ZVR5cGU9IlVwZGF0ZWQiIGRhdGVJbmZvcm1hdGlvbj0idjEiPjIwMTgtMDctMjNUMDA6MDk6NTlaPC9kYXRlPgogICAgPGRhdGUgZGF0ZVR5cGU9IlN1Ym1pdHRlZCIgZGF0ZUluZm9ybWF0aW9uPSJ2MiI+MjAxOS0wMi0yNVQxNzowNTo1M1o8L2RhdGU+CiAgICA8ZGF0ZSBkYXRlVHlwZT0iVXBkYXRlZCIgZGF0ZUluZm9ybWF0aW9uPSJ2MiI+MjAxOS0wMi0yNlQwMTozMzoyNlo8L2RhdGU+CiAgICA8ZGF0ZSBkYXRlVHlwZT0iU3VibWl0dGVkIiBkYXRlSW5mb3JtYXRpb249InYzIj4yMDE5LTA2LTA1VDE0OjM1OjM5WjwvZGF0ZT4KICAgIDxkYXRlIGRhdGVUeXBlPSJVcGRhdGVkIiBkYXRlSW5mb3JtYXRpb249InYzIj4yMDE5LTA2LTA2VDAwOjE3OjMwWjwvZGF0ZT4KICAgIDxkYXRlIGRhdGVUeXBlPSJTdWJtaXR0ZWQiIGRhdGVJbmZvcm1hdGlvbj0idjQiPjIwMTktMDktMTJUMTQ6MDM6NDRaPC9kYXRlPgogICAgPGRhdGUgZGF0ZVR5cGU9IlVwZGF0ZWQiIGRhdGVJbmZvcm1hdGlvbj0idjQiPjIwMTktMDktMTNUMDA6MTc6MTFaPC9kYXRlPgogICAgPGRhdGUgZGF0ZVR5cGU9IkF2YWlsYWJsZSIgZGF0ZUluZm9ybWF0aW9uPSJ2MSI+MjAxOC0wNzwvZGF0ZT4KICA8L2RhdGVzPgogIDxyZXNvdXJjZVR5cGUgcmVzb3VyY2VUeXBlR2VuZXJhbD0iUHJlcHJpbnQiPkFydGljbGU8L3Jlc291cmNlVHlwZT4KICA8dmVyc2lvbj40PC92ZXJzaW9uPgogIDxyaWdodHNMaXN0PgogICAgPHJpZ2h0cyByaWdodHNVUkk9Imh0dHA6Ly9hcnhpdi5vcmcvbGljZW5zZXMvbm9uZXhjbHVzaXZlLWRpc3RyaWIvMS4wLyI+YXJYaXYub3JnIHBlcnBldHVhbCwgbm9uLWV4Y2x1c2l2ZSBsaWNlbnNlPC9yaWdodHM+CiAgPC9yaWdodHNMaXN0PgogIDxkZXNjcmlwdGlvbnM+CiAgICA8ZGVzY3JpcHRpb24gZGVzY3JpcHRpb25UeXBlPSJBYnN0cmFjdCI+SXMgQW5kcm9pZCBtYWx3YXJlIGNsYXNzaWZpY2F0aW9uIGEgc29sdmVkIHByb2JsZW0/IFB1Ymxpc2hlZCBGMSBzY29yZXMgb2YgdXAgdG8gMC45OSBhcHBlYXIgdG8gbGVhdmUgdmVyeSBsaXR0bGUgcm9vbSBmb3IgaW1wcm92ZW1lbnQuIEluIHRoaXMgcGFwZXIsIHdlIGFyZ3VlIHRoYXQgcmVzdWx0cyBhcmUgY29tbW9ubHkgaW5mbGF0ZWQgZHVlIHRvIHR3byBwZXJ2YXNpdmUgc291cmNlcyBvZiBleHBlcmltZW50YWwgYmlhczogInNwYXRpYWwgYmlhcyIgY2F1c2VkIGJ5IGRpc3RyaWJ1dGlvbnMgb2YgdHJhaW5pbmcgYW5kIHRlc3RpbmcgZGF0YSB0aGF0IGFyZSBub3QgcmVwcmVzZW50YXRpdmUgb2YgYSByZWFsLXdvcmxkIGRlcGxveW1lbnQ7IGFuZCAidGVtcG9yYWwgYmlhcyIgY2F1c2VkIGJ5IGluY29ycmVjdCB0aW1lIHNwbGl0cyBvZiB0cmFpbmluZyBhbmQgdGVzdGluZyBzZXRzLCBsZWFkaW5nIHRvIGltcG9zc2libGUgY29uZmlndXJhdGlvbnMuIFdlIHByb3Bvc2UgYSBzZXQgb2Ygc3BhY2UgYW5kIHRpbWUgY29uc3RyYWludHMgZm9yIGV4cGVyaW1lbnQgZGVzaWduIHRoYXQgZWxpbWluYXRlcyBib3RoIHNvdXJjZXMgb2YgYmlhcy4gV2UgaW50cm9kdWNlIGEgbmV3IG1ldHJpYyB0aGF0IHN1bW1hcml6ZXMgdGhlIGV4cGVjdGVkIHJvYnVzdG5lc3Mgb2YgYSBjbGFzc2lmaWVyIGluIGEgcmVhbC13b3JsZCBzZXR0aW5nLCBhbmQgd2UgcHJlc2VudCBhbiBhbGdvcml0aG0gdG8gdHVuZSBpdHMgcGVyZm9ybWFuY2UuIEZpbmFsbHksIHdlIGRlbW9uc3RyYXRlIGhvdyB0aGlzIGFsbG93cyB1cyB0byBldmFsdWF0ZSBtaXRpZ2F0aW9uIHN0cmF0ZWdpZXMgZm9yIHRpbWUgZGVjYXkgc3VjaCBhcyBhY3RpdmUgbGVhcm5pbmcuIFdlIGhhdmUgaW1wbGVtZW50ZWQgb3VyIHNvbHV0aW9ucyBpbiBURVNTRVJBQ1QsIGFuIG9wZW4gc291cmNlIGV2YWx1YXRpb24gZnJhbWV3b3JrIGZvciBjb21wYXJpbmcgbWFsd2FyZSBjbGFzc2lmaWVycyBpbiBhIHJlYWxpc3RpYyBzZXR0aW5nLiBXZSB1c2VkIFRFU1NFUkFDVCB0byBldmFsdWF0ZSB0aHJlZSBBbmRyb2lkIG1hbHdhcmUgY2xhc3NpZmllcnMgZnJvbSB0aGUgbGl0ZXJhdHVyZSBvbiBhIGRhdGFzZXQgb2YgMTI5SyBhcHBsaWNhdGlvbnMgc3Bhbm5pbmcgb3ZlciB0aHJlZSB5ZWFycy4gT3VyIGV2YWx1YXRpb24gY29uZmlybXMgdGhhdCBlYXJsaWVyIHB1Ymxpc2hlZCByZXN1bHRzIGFyZSBiaWFzZWQsIHdoaWxlIGFsc28gcmV2ZWFsaW5nIGNvdW50ZXItaW50dWl0aXZlIHBlcmZvcm1hbmNlIGFuZCBzaG93aW5nIHRoYXQgYXBwcm9wcmlhdGUgdHVuaW5nIGNhbiBsZWFkIHRvIHNpZ25pZmljYW50IGltcHJvdmVtZW50cy48L2Rlc2NyaXB0aW9uPgogICAgPGRlc2NyaXB0aW9uIGRlc2NyaXB0aW9uVHlwZT0iT3RoZXIiPlRoaXMgYXJYaXYgdmVyc2lvbiAodjQpIGNvcnJlc3BvbmRzIHRvIHRoZSBvbmUgcHVibGlzaGVkIGF0IFVTRU5JWCBTZWN1cml0eSBTeW1wb3NpdW0gMjAxOSwgd2l0aCBhIGZpeGVkIHR5cG8gaW4gRXF1YXRpb24gKDQpLCB3aGljaCByZXBvcnRlZCBhbiBleHRyYSBub3JtYWxpemF0aW9uIGZhY3RvciBvZiAoMS9OKS4gVGhlIHJlc3VsdHMgaW4gdGhlIHBhcGVyIGFuZCB0aGUgcmVsZWFzZWQgaW1wbGVtZW50YXRpb24gb2YgdGhlIFRFU1NFUkFDVCBmcmFtZXdvcmsgcmVtYWluIHZhbGlkIGFuZCBjb3JyZWN0IGFzIHRoZXkgcmVseSBvbiBQeXRob24ncyBudW1weSBpbXBsZW1lbnRhdGlvbiBvZiBhcmVhIHVuZGVyIHRoZSBjdXJ2ZTwvZGVzY3JpcHRpb24+CiAgPC9kZXNjcmlwdGlvbnM+CjwvcmVzb3VyY2U+","url":"https://arxiv.org/abs/1807.07838","contentUrl":null,"metadataVersion":0,"schemaVersion":"http://datacite.org/schema/kernel-4","source":"mds","isActive":true,"state":"findable","reason":null,"viewCount":0,"viewsOverTime":[],"downloadCount":0,"downloadsOverTime":[],"referenceCount":0,"citationCount":0,"citationsOverTime":[],"partCount":0,"partOfCount":0,"versionCount":0,"versionOfCount":0,"created":"2022-03-02T07:23:43.000Z","registered":"2022-03-02T07:23:44.000Z","published":"2018","updated":"2022-03-02T07:23:44.000Z"},"relationships":{"client":{"data":{"id":"arxiv.content","type":"clients"}},"provider":{"data":{"id":"arxiv","type":"providers"}},"media":{"data":{"id":"10.48550/arxiv.1807.07838","type":"media"}},"references":{"data":[]},"citations":{"data":[]},"parts":{"data":[]},"partOf":{"data":[]},"versions":{"data":[]},"versionOf":{"data":[]}}}}