{"data":{"id":"10.48550/arxiv.1902.09432","type":"dois","attributes":{"doi":"10.48550/arxiv.1902.09432","prefix":"10.48550","suffix":"arxiv.1902.09432","identifiers":[{"identifier":"1902.09432","identifierType":"arXiv"}],"alternateIdentifiers":[{"alternateIdentifierType":"arXiv","alternateIdentifier":"1902.09432"}],"creators":[{"name":"Yoon, Jaehong","nameType":"Personal","givenName":"Jaehong","familyName":"Yoon","affiliation":[],"nameIdentifiers":[]},{"name":"Kim, Saehoon","nameType":"Personal","givenName":"Saehoon","familyName":"Kim","affiliation":[],"nameIdentifiers":[]},{"name":"Yang, Eunho","nameType":"Personal","givenName":"Eunho","familyName":"Yang","affiliation":[],"nameIdentifiers":[]},{"name":"Hwang, Sung Ju","nameType":"Personal","givenName":"Sung Ju","familyName":"Hwang","affiliation":[],"nameIdentifiers":[]}],"titles":[{"title":"Scalable and Order-robust Continual Learning with Additive Parameter Decomposition"}],"publisher":"arXiv","container":{},"publicationYear":2019,"subjects":[{"lang":"en","subject":"Machine Learning (cs.LG)","subjectScheme":"arXiv"},{"lang":"en","subject":"Machine Learning (stat.ML)","subjectScheme":"arXiv"},{"subject":"FOS: Computer and information sciences","subjectScheme":"Fields of Science and Technology (FOS)"},{"subject":"FOS: Computer and information sciences","schemeUri":"http://www.oecd.org/science/inno/38235147.pdf","subjectScheme":"Fields of Science and Technology (FOS)"},{"lang":"en","subject":"I.2.6; I.2.10","subjectScheme":"ACM"}],"contributors":[],"dates":[{"date":"2019-02-25T16:49:52Z","dateType":"Submitted","dateInformation":"v1"},{"date":"2019-02-26T01:33:06Z","dateType":"Updated","dateInformation":"v1"},{"date":"2019-06-16T19:44:15Z","dateType":"Submitted","dateInformation":"v2"},{"date":"2019-06-18T00:19:12Z","dateType":"Updated","dateInformation":"v2"},{"date":"2020-02-15T14:13:27Z","dateType":"Submitted","dateInformation":"v3"},{"date":"2020-02-18T01:08:08Z","dateType":"Updated","dateInformation":"v3"},{"date":"2019-02","dateType":"Available","dateInformation":"v1"},{"date":"2019","dateType":"Issued"}],"language":null,"types":{"ris":"GEN","bibtex":"misc","citeproc":"article","schemaOrg":"CreativeWork","resourceType":"Article","resourceTypeGeneral":"Preprint"},"relatedIdentifiers":[],"relatedItems":[],"sizes":[],"formats":[],"version":"3","rightsList":[{"rights":"arXiv.org perpetual, non-exclusive license","rightsUri":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/"}],"descriptions":[{"description":"While recent continual learning methods largely alleviate the catastrophic problem on toy-sized datasets, some issues remain to be tackled to apply them to real-world problem domains. First, a continual learning model should effectively handle catastrophic forgetting and be efficient to train even with a large number of tasks. Secondly, it needs to tackle the problem of order-sensitivity, where the performance of the tasks largely varies based on the order of the task arrival sequence, as it may cause serious problems where fairness plays a critical role (e.g. medical diagnosis). To tackle these practical challenges, we propose a novel continual learning method that is scalable as well as order-robust, which instead of learning a completely shared set of weights, represents the parameters for each task as a sum of task-shared and sparse task-adaptive parameters. With our Additive Parameter Decomposition (APD), the task-adaptive parameters for earlier tasks remain mostly unaffected, where we update them only to reflect the changes made to the task-shared parameters. This decomposition of parameters effectively prevents catastrophic forgetting and order-sensitivity, while being computation- and memory-efficient. Further, we can achieve even better scalability with APD using hierarchical knowledge consolidation, which clusters the task-adaptive parameters to obtain hierarchically shared parameters. We validate our network with APD, APD-Net, on multiple benchmark datasets against state-of-the-art continual learning methods, which it largely outperforms in accuracy, scalability, and order-robustness.","descriptionType":"Abstract"},{"description":"Published in \"International Conference on Learning Representation (ICLR)\" 2020","descriptionType":"Other"}],"geoLocations":[],"fundingReferences":[],"xml":"PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0idXRmLTgiPz4KPHJlc291cmNlIHhtbG5zPSJodHRwOi8vZGF0YWNpdGUub3JnL3NjaGVtYS9rZXJuZWwtNCIgeG1sbnM6eHNpPSJodHRwOi8vd3d3LnczLm9yZy8yMDAxL1hNTFNjaGVtYS1pbnN0YW5jZSIgeHNpOnNjaGVtYUxvY2F0aW9uPSJodHRwOi8vZGF0YWNpdGUub3JnL3NjaGVtYS9rZXJuZWwtNCBodHRwOi8vc2NoZW1hLmRhdGFjaXRlLm9yZy9tZXRhL2tlcm5lbC00LjMvbWV0YWRhdGEueHNkIj4KICA8aWRlbnRpZmllciBpZGVudGlmaWVyVHlwZT0iRE9JIj4xMC40ODU1MC9BUlhJVi4xOTAyLjA5NDMyPC9pZGVudGlmaWVyPgogIDxhbHRlcm5hdGVJZGVudGlmaWVycz4KICAgIDxhbHRlcm5hdGVJZGVudGlmaWVyIGFsdGVybmF0ZUlkZW50aWZpZXJUeXBlPSJhclhpdiI+MTkwMi4wOTQzMjwvYWx0ZXJuYXRlSWRlbnRpZmllcj4KICA8L2FsdGVybmF0ZUlkZW50aWZpZXJzPgogIDxjcmVhdG9ycz4KICAgIDxjcmVhdG9yPgogICAgICA8Y3JlYXRvck5hbWUgbmFtZVR5cGU9IlBlcnNvbmFsIj5Zb29uLCBKYWVob25nPC9jcmVhdG9yTmFtZT4KICAgICAgPGdpdmVuTmFtZT5KYWVob25nPC9naXZlbk5hbWU+CiAgICAgIDxmYW1pbHlOYW1lPllvb248L2ZhbWlseU5hbWU+CiAgICA8L2NyZWF0b3I+CiAgICA8Y3JlYXRvcj4KICAgICAgPGNyZWF0b3JOYW1lIG5hbWVUeXBlPSJQZXJzb25hbCI+S2ltLCBTYWVob29uPC9jcmVhdG9yTmFtZT4KICAgICAgPGdpdmVuTmFtZT5TYWVob29uPC9naXZlbk5hbWU+CiAgICAgIDxmYW1pbHlOYW1lPktpbTwvZmFtaWx5TmFtZT4KICAgIDwvY3JlYXRvcj4KICAgIDxjcmVhdG9yPgogICAgICA8Y3JlYXRvck5hbWUgbmFtZVR5cGU9IlBlcnNvbmFsIj5ZYW5nLCBFdW5obzwvY3JlYXRvck5hbWU+CiAgICAgIDxnaXZlbk5hbWU+RXVuaG88L2dpdmVuTmFtZT4KICAgICAgPGZhbWlseU5hbWU+WWFuZzwvZmFtaWx5TmFtZT4KICAgIDwvY3JlYXRvcj4KICAgIDxjcmVhdG9yPgogICAgICA8Y3JlYXRvck5hbWUgbmFtZVR5cGU9IlBlcnNvbmFsIj5Id2FuZywgU3VuZyBKdTwvY3JlYXRvck5hbWU+CiAgICAgIDxnaXZlbk5hbWU+U3VuZyBKdTwvZ2l2ZW5OYW1lPgogICAgICA8ZmFtaWx5TmFtZT5Id2FuZzwvZmFtaWx5TmFtZT4KICAgIDwvY3JlYXRvcj4KICA8L2NyZWF0b3JzPgogIDx0aXRsZXM+CiAgICA8dGl0bGU+U2NhbGFibGUgYW5kIE9yZGVyLXJvYnVzdCBDb250aW51YWwgTGVhcm5pbmcgd2l0aCBBZGRpdGl2ZSBQYXJhbWV0ZXIgRGVjb21wb3NpdGlvbjwvdGl0bGU+CiAgPC90aXRsZXM+CiAgPHB1Ymxpc2hlcj5hclhpdjwvcHVibGlzaGVyPgogIDxwdWJsaWNhdGlvblllYXI+MjAxOTwvcHVibGljYXRpb25ZZWFyPgogIDxzdWJqZWN0cz4KICAgIDxzdWJqZWN0IHhtbDpsYW5nPSJlbiIgc3ViamVjdFNjaGVtZT0iYXJYaXYiPk1hY2hpbmUgTGVhcm5pbmcgKGNzLkxHKTwvc3ViamVjdD4KICAgIDxzdWJqZWN0IHhtbDpsYW5nPSJlbiIgc3ViamVjdFNjaGVtZT0iYXJYaXYiPk1hY2hpbmUgTGVhcm5pbmcgKHN0YXQuTUwpPC9zdWJqZWN0PgogICAgPHN1YmplY3Qgc3ViamVjdFNjaGVtZT0iRmllbGRzIG9mIFNjaWVuY2UgYW5kIFRlY2hub2xvZ3kgKEZPUykiPkZPUzogQ29tcHV0ZXIgYW5kIGluZm9ybWF0aW9uIHNjaWVuY2VzPC9zdWJqZWN0PgogICAgPHN1YmplY3QgeG1sOmxhbmc9ImVuIiBzdWJqZWN0U2NoZW1lPSJBQ00iPkkuMi42OyBJLjIuMTA8L3N1YmplY3Q+CiAgPC9zdWJqZWN0cz4KICA8ZGF0ZXM+CiAgICA8ZGF0ZSBkYXRlVHlwZT0iU3VibWl0dGVkIiBkYXRlSW5mb3JtYXRpb249InYxIj4yMDE5LTAyLTI1VDE2OjQ5OjUyWjwvZGF0ZT4KICAgIDxkYXRlIGRhdGVUeXBlPSJVcGRhdGVkIiBkYXRlSW5mb3JtYXRpb249InYxIj4yMDE5LTAyLTI2VDAxOjMzOjA2WjwvZGF0ZT4KICAgIDxkYXRlIGRhdGVUeXBlPSJTdWJtaXR0ZWQiIGRhdGVJbmZvcm1hdGlvbj0idjIiPjIwMTktMDYtMTZUMTk6NDQ6MTVaPC9kYXRlPgogICAgPGRhdGUgZGF0ZVR5cGU9IlVwZGF0ZWQiIGRhdGVJbmZvcm1hdGlvbj0idjIiPjIwMTktMDYtMThUMDA6MTk6MTJaPC9kYXRlPgogICAgPGRhdGUgZGF0ZVR5cGU9IlN1Ym1pdHRlZCIgZGF0ZUluZm9ybWF0aW9uPSJ2MyI+MjAyMC0wMi0xNVQxNDoxMzoyN1o8L2RhdGU+CiAgICA8ZGF0ZSBkYXRlVHlwZT0iVXBkYXRlZCIgZGF0ZUluZm9ybWF0aW9uPSJ2MyI+MjAyMC0wMi0xOFQwMTowODowOFo8L2RhdGU+CiAgICA8ZGF0ZSBkYXRlVHlwZT0iQXZhaWxhYmxlIiBkYXRlSW5mb3JtYXRpb249InYxIj4yMDE5LTAyPC9kYXRlPgogIDwvZGF0ZXM+CiAgPHJlc291cmNlVHlwZSByZXNvdXJjZVR5cGVHZW5lcmFsPSJQcmVwcmludCI+QXJ0aWNsZTwvcmVzb3VyY2VUeXBlPgogIDx2ZXJzaW9uPjM8L3ZlcnNpb24+CiAgPHJpZ2h0c0xpc3Q+CiAgICA8cmlnaHRzIHJpZ2h0c1VSST0iaHR0cDovL2FyeGl2Lm9yZy9saWNlbnNlcy9ub25leGNsdXNpdmUtZGlzdHJpYi8xLjAvIj5hclhpdi5vcmcgcGVycGV0dWFsLCBub24tZXhjbHVzaXZlIGxpY2Vuc2U8L3JpZ2h0cz4KICA8L3JpZ2h0c0xpc3Q+CiAgPGRlc2NyaXB0aW9ucz4KICAgIDxkZXNjcmlwdGlvbiBkZXNjcmlwdGlvblR5cGU9IkFic3RyYWN0Ij5XaGlsZSByZWNlbnQgY29udGludWFsIGxlYXJuaW5nIG1ldGhvZHMgbGFyZ2VseSBhbGxldmlhdGUgdGhlIGNhdGFzdHJvcGhpYyBwcm9ibGVtIG9uIHRveS1zaXplZCBkYXRhc2V0cywgc29tZSBpc3N1ZXMgcmVtYWluIHRvIGJlIHRhY2tsZWQgdG8gYXBwbHkgdGhlbSB0byByZWFsLXdvcmxkIHByb2JsZW0gZG9tYWlucy4gRmlyc3QsIGEgY29udGludWFsIGxlYXJuaW5nIG1vZGVsIHNob3VsZCBlZmZlY3RpdmVseSBoYW5kbGUgY2F0YXN0cm9waGljIGZvcmdldHRpbmcgYW5kIGJlIGVmZmljaWVudCB0byB0cmFpbiBldmVuIHdpdGggYSBsYXJnZSBudW1iZXIgb2YgdGFza3MuIFNlY29uZGx5LCBpdCBuZWVkcyB0byB0YWNrbGUgdGhlIHByb2JsZW0gb2Ygb3JkZXItc2Vuc2l0aXZpdHksIHdoZXJlIHRoZSBwZXJmb3JtYW5jZSBvZiB0aGUgdGFza3MgbGFyZ2VseSB2YXJpZXMgYmFzZWQgb24gdGhlIG9yZGVyIG9mIHRoZSB0YXNrIGFycml2YWwgc2VxdWVuY2UsIGFzIGl0IG1heSBjYXVzZSBzZXJpb3VzIHByb2JsZW1zIHdoZXJlIGZhaXJuZXNzIHBsYXlzIGEgY3JpdGljYWwgcm9sZSAoZS5nLiBtZWRpY2FsIGRpYWdub3NpcykuIFRvIHRhY2tsZSB0aGVzZSBwcmFjdGljYWwgY2hhbGxlbmdlcywgd2UgcHJvcG9zZSBhIG5vdmVsIGNvbnRpbnVhbCBsZWFybmluZyBtZXRob2QgdGhhdCBpcyBzY2FsYWJsZSBhcyB3ZWxsIGFzIG9yZGVyLXJvYnVzdCwgd2hpY2ggaW5zdGVhZCBvZiBsZWFybmluZyBhIGNvbXBsZXRlbHkgc2hhcmVkIHNldCBvZiB3ZWlnaHRzLCByZXByZXNlbnRzIHRoZSBwYXJhbWV0ZXJzIGZvciBlYWNoIHRhc2sgYXMgYSBzdW0gb2YgdGFzay1zaGFyZWQgYW5kIHNwYXJzZSB0YXNrLWFkYXB0aXZlIHBhcmFtZXRlcnMuIFdpdGggb3VyIEFkZGl0aXZlIFBhcmFtZXRlciBEZWNvbXBvc2l0aW9uIChBUEQpLCB0aGUgdGFzay1hZGFwdGl2ZSBwYXJhbWV0ZXJzIGZvciBlYXJsaWVyIHRhc2tzIHJlbWFpbiBtb3N0bHkgdW5hZmZlY3RlZCwgd2hlcmUgd2UgdXBkYXRlIHRoZW0gb25seSB0byByZWZsZWN0IHRoZSBjaGFuZ2VzIG1hZGUgdG8gdGhlIHRhc2stc2hhcmVkIHBhcmFtZXRlcnMuIFRoaXMgZGVjb21wb3NpdGlvbiBvZiBwYXJhbWV0ZXJzIGVmZmVjdGl2ZWx5IHByZXZlbnRzIGNhdGFzdHJvcGhpYyBmb3JnZXR0aW5nIGFuZCBvcmRlci1zZW5zaXRpdml0eSwgd2hpbGUgYmVpbmcgY29tcHV0YXRpb24tIGFuZCBtZW1vcnktZWZmaWNpZW50LiBGdXJ0aGVyLCB3ZSBjYW4gYWNoaWV2ZSBldmVuIGJldHRlciBzY2FsYWJpbGl0eSB3aXRoIEFQRCB1c2luZyBoaWVyYXJjaGljYWwga25vd2xlZGdlIGNvbnNvbGlkYXRpb24sIHdoaWNoIGNsdXN0ZXJzIHRoZSB0YXNrLWFkYXB0aXZlIHBhcmFtZXRlcnMgdG8gb2J0YWluIGhpZXJhcmNoaWNhbGx5IHNoYXJlZCBwYXJhbWV0ZXJzLiBXZSB2YWxpZGF0ZSBvdXIgbmV0d29yayB3aXRoIEFQRCwgQVBELU5ldCwgb24gbXVsdGlwbGUgYmVuY2htYXJrIGRhdGFzZXRzIGFnYWluc3Qgc3RhdGUtb2YtdGhlLWFydCBjb250aW51YWwgbGVhcm5pbmcgbWV0aG9kcywgd2hpY2ggaXQgbGFyZ2VseSBvdXRwZXJmb3JtcyBpbiBhY2N1cmFjeSwgc2NhbGFiaWxpdHksIGFuZCBvcmRlci1yb2J1c3RuZXNzLjwvZGVzY3JpcHRpb24+CiAgICA8ZGVzY3JpcHRpb24gZGVzY3JpcHRpb25UeXBlPSJPdGhlciI+UHVibGlzaGVkIGluICJJbnRlcm5hdGlvbmFsIENvbmZlcmVuY2Ugb24gTGVhcm5pbmcgUmVwcmVzZW50YXRpb24gKElDTFIpIiAyMDIwPC9kZXNjcmlwdGlvbj4KICA8L2Rlc2NyaXB0aW9ucz4KPC9yZXNvdXJjZT4=","url":"https://arxiv.org/abs/1902.09432","contentUrl":null,"metadataVersion":0,"schemaVersion":"http://datacite.org/schema/kernel-4","source":"mds","isActive":true,"state":"findable","reason":null,"viewCount":0,"viewsOverTime":[],"downloadCount":0,"downloadsOverTime":[],"referenceCount":0,"citationCount":0,"citationsOverTime":[],"partCount":0,"partOfCount":0,"versionCount":0,"versionOfCount":0,"created":"2022-03-01T14:28:16.000Z","registered":"2022-03-01T14:28:18.000Z","published":"2019","updated":"2022-03-01T14:28:18.000Z"},"relationships":{"client":{"data":{"id":"arxiv.content","type":"clients"}},"provider":{"data":{"id":"arxiv","type":"providers"}},"media":{"data":{"id":"10.48550/arxiv.1902.09432","type":"media"}},"references":{"data":[]},"citations":{"data":[]},"parts":{"data":[]},"partOf":{"data":[]},"versions":{"data":[]},"versionOf":{"data":[]}}}}