{"data":{"id":"10.48550/arxiv.2204.12451","type":"dois","attributes":{"doi":"10.48550/arxiv.2204.12451","prefix":"10.48550","suffix":"arxiv.2204.12451","identifiers":[{"identifier":"2204.12451","identifierType":"arXiv"}],"alternateIdentifiers":[{"alternateIdentifierType":"arXiv","alternateIdentifier":"2204.12451"}],"creators":[{"name":"Zhou, Daquan","nameType":"Personal","givenName":"Daquan","familyName":"Zhou","affiliation":[],"nameIdentifiers":[]},{"name":"Yu, Zhiding","nameType":"Personal","givenName":"Zhiding","familyName":"Yu","affiliation":[],"nameIdentifiers":[]},{"name":"Xie, Enze","nameType":"Personal","givenName":"Enze","familyName":"Xie","affiliation":[],"nameIdentifiers":[]},{"name":"Xiao, Chaowei","nameType":"Personal","givenName":"Chaowei","familyName":"Xiao","affiliation":[],"nameIdentifiers":[]},{"name":"Anandkumar, Anima","nameType":"Personal","givenName":"Anima","familyName":"Anandkumar","affiliation":[],"nameIdentifiers":[]},{"name":"Feng, Jiashi","nameType":"Personal","givenName":"Jiashi","familyName":"Feng","affiliation":[],"nameIdentifiers":[]},{"name":"Alvarez, Jose M.","nameType":"Personal","givenName":"Jose M.","familyName":"Alvarez","affiliation":[],"nameIdentifiers":[]}],"titles":[{"title":"Understanding The Robustness in Vision Transformers"}],"publisher":"arXiv","container":{},"publicationYear":2022,"subjects":[{"lang":"en","subject":"Computer Vision and Pattern Recognition (cs.CV)","subjectScheme":"arXiv"},{"subject":"FOS: Computer and information sciences","subjectScheme":"Fields of Science and Technology (FOS)"},{"subject":"FOS: Computer and information sciences","schemeUri":"http://www.oecd.org/science/inno/38235147.pdf","subjectScheme":"Fields of Science and Technology (FOS)"}],"contributors":[],"dates":[{"date":"2022-04-26T17:16:32Z","dateType":"Submitted","dateInformation":"v1"},{"date":"2022-04-27T00:31:40Z","dateType":"Updated","dateInformation":"v1"},{"date":"2022-04-27T13:01:30Z","dateType":"Submitted","dateInformation":"v2"},{"date":"2022-04-28T00:23:06Z","dateType":"Updated","dateInformation":"v2"},{"date":"2022-10-21T15:29:01Z","dateType":"Submitted","dateInformation":"v3"},{"date":"2022-10-24T00:15:21Z","dateType":"Updated","dateInformation":"v3"},{"date":"2022-11-08T15:52:39Z","dateType":"Submitted","dateInformation":"v4"},{"date":"2022-11-09T01:16:02Z","dateType":"Updated","dateInformation":"v4"},{"date":"2022-04","dateType":"Available","dateInformation":"v1"},{"date":"2022","dateType":"Issued"}],"language":null,"types":{"ris":"GEN","bibtex":"misc","citeproc":"article","schemaOrg":"CreativeWork","resourceType":"Article","resourceTypeGeneral":"Preprint"},"relatedIdentifiers":[],"relatedItems":[],"sizes":[],"formats":[],"version":"4","rightsList":[{"rights":"arXiv.org perpetual, non-exclusive license","rightsUri":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/"}],"descriptions":[{"description":"Recent studies show that Vision Transformers(ViTs) exhibit strong robustness against various corruptions. Although this property is partly attributed to the self-attention mechanism, there is still a lack of systematic understanding. In this paper, we examine the role of self-attention in learning robust representations. Our study is motivated by the intriguing properties of the emerging visual grouping in Vision Transformers, which indicates that self-attention may promote robustness through improved mid-level representations. We further propose a family of fully attentional networks (FANs) that strengthen this capability by incorporating an attentional channel processing design. We validate the design comprehensively on various hierarchical backbones. Our model achieves a state-of-the-art 87.1% accuracy and 35.8% mCE on ImageNet-1k and ImageNet-C with 76.8M parameters. We also demonstrate state-of-the-art accuracy and robustness in two downstream tasks: semantic segmentation and object detection. Code is available at: https://github.com/NVlabs/FAN.","descriptionType":"Abstract"}],"geoLocations":[],"fundingReferences":[],"xml":"PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0idXRmLTgiPz4KPHJlc291cmNlIHhtbG5zPSJodHRwOi8vZGF0YWNpdGUub3JnL3NjaGVtYS9rZXJuZWwtNCIgeG1sbnM6eHNpPSJodHRwOi8vd3d3LnczLm9yZy8yMDAxL1hNTFNjaGVtYS1pbnN0YW5jZSIgeHNpOnNjaGVtYUxvY2F0aW9uPSJodHRwOi8vZGF0YWNpdGUub3JnL3NjaGVtYS9rZXJuZWwtNCBodHRwOi8vc2NoZW1hLmRhdGFjaXRlLm9yZy9tZXRhL2tlcm5lbC00LjMvbWV0YWRhdGEueHNkIj4KICA8aWRlbnRpZmllciBpZGVudGlmaWVyVHlwZT0iRE9JIj4xMC40ODU1MC9BUlhJVi4yMjA0LjEyNDUxPC9pZGVudGlmaWVyPgogIDxhbHRlcm5hdGVJZGVudGlmaWVycz4KICAgIDxhbHRlcm5hdGVJZGVudGlmaWVyIGFsdGVybmF0ZUlkZW50aWZpZXJUeXBlPSJhclhpdiI+MjIwNC4xMjQ1MTwvYWx0ZXJuYXRlSWRlbnRpZmllcj4KICA8L2FsdGVybmF0ZUlkZW50aWZpZXJzPgogIDxjcmVhdG9ycz4KICAgIDxjcmVhdG9yPgogICAgICA8Y3JlYXRvck5hbWUgbmFtZVR5cGU9IlBlcnNvbmFsIj5aaG91LCBEYXF1YW48L2NyZWF0b3JOYW1lPgogICAgICA8Z2l2ZW5OYW1lPkRhcXVhbjwvZ2l2ZW5OYW1lPgogICAgICA8ZmFtaWx5TmFtZT5aaG91PC9mYW1pbHlOYW1lPgogICAgPC9jcmVhdG9yPgogICAgPGNyZWF0b3I+CiAgICAgIDxjcmVhdG9yTmFtZSBuYW1lVHlwZT0iUGVyc29uYWwiPll1LCBaaGlkaW5nPC9jcmVhdG9yTmFtZT4KICAgICAgPGdpdmVuTmFtZT5aaGlkaW5nPC9naXZlbk5hbWU+CiAgICAgIDxmYW1pbHlOYW1lPll1PC9mYW1pbHlOYW1lPgogICAgPC9jcmVhdG9yPgogICAgPGNyZWF0b3I+CiAgICAgIDxjcmVhdG9yTmFtZSBuYW1lVHlwZT0iUGVyc29uYWwiPlhpZSwgRW56ZTwvY3JlYXRvck5hbWU+CiAgICAgIDxnaXZlbk5hbWU+RW56ZTwvZ2l2ZW5OYW1lPgogICAgICA8ZmFtaWx5TmFtZT5YaWU8L2ZhbWlseU5hbWU+CiAgICA8L2NyZWF0b3I+CiAgICA8Y3JlYXRvcj4KICAgICAgPGNyZWF0b3JOYW1lIG5hbWVUeXBlPSJQZXJzb25hbCI+WGlhbywgQ2hhb3dlaTwvY3JlYXRvck5hbWU+CiAgICAgIDxnaXZlbk5hbWU+Q2hhb3dlaTwvZ2l2ZW5OYW1lPgogICAgICA8ZmFtaWx5TmFtZT5YaWFvPC9mYW1pbHlOYW1lPgogICAgPC9jcmVhdG9yPgogICAgPGNyZWF0b3I+CiAgICAgIDxjcmVhdG9yTmFtZSBuYW1lVHlwZT0iUGVyc29uYWwiPkFuYW5ka3VtYXIsIEFuaW1hPC9jcmVhdG9yTmFtZT4KICAgICAgPGdpdmVuTmFtZT5BbmltYTwvZ2l2ZW5OYW1lPgogICAgICA8ZmFtaWx5TmFtZT5BbmFuZGt1bWFyPC9mYW1pbHlOYW1lPgogICAgPC9jcmVhdG9yPgogICAgPGNyZWF0b3I+CiAgICAgIDxjcmVhdG9yTmFtZSBuYW1lVHlwZT0iUGVyc29uYWwiPkZlbmcsIEppYXNoaTwvY3JlYXRvck5hbWU+CiAgICAgIDxnaXZlbk5hbWU+Smlhc2hpPC9naXZlbk5hbWU+CiAgICAgIDxmYW1pbHlOYW1lPkZlbmc8L2ZhbWlseU5hbWU+CiAgICA8L2NyZWF0b3I+CiAgICA8Y3JlYXRvcj4KICAgICAgPGNyZWF0b3JOYW1lIG5hbWVUeXBlPSJQZXJzb25hbCI+QWx2YXJleiwgSm9zZSBNLjwvY3JlYXRvck5hbWU+CiAgICAgIDxnaXZlbk5hbWU+Sm9zZSBNLjwvZ2l2ZW5OYW1lPgogICAgICA8ZmFtaWx5TmFtZT5BbHZhcmV6PC9mYW1pbHlOYW1lPgogICAgPC9jcmVhdG9yPgogIDwvY3JlYXRvcnM+CiAgPHRpdGxlcz4KICAgIDx0aXRsZT5VbmRlcnN0YW5kaW5nIFRoZSBSb2J1c3RuZXNzIGluIFZpc2lvbiBUcmFuc2Zvcm1lcnM8L3RpdGxlPgogIDwvdGl0bGVzPgogIDxwdWJsaXNoZXI+YXJYaXY8L3B1Ymxpc2hlcj4KICA8cHVibGljYXRpb25ZZWFyPjIwMjI8L3B1YmxpY2F0aW9uWWVhcj4KICA8c3ViamVjdHM+CiAgICA8c3ViamVjdCB4bWw6bGFuZz0iZW4iIHN1YmplY3RTY2hlbWU9ImFyWGl2Ij5Db21wdXRlciBWaXNpb24gYW5kIFBhdHRlcm4gUmVjb2duaXRpb24gKGNzLkNWKTwvc3ViamVjdD4KICAgIDxzdWJqZWN0IHN1YmplY3RTY2hlbWU9IkZpZWxkcyBvZiBTY2llbmNlIGFuZCBUZWNobm9sb2d5IChGT1MpIj5GT1M6IENvbXB1dGVyIGFuZCBpbmZvcm1hdGlvbiBzY2llbmNlczwvc3ViamVjdD4KICA8L3N1YmplY3RzPgogIDxkYXRlcz4KICAgIDxkYXRlIGRhdGVUeXBlPSJTdWJtaXR0ZWQiIGRhdGVJbmZvcm1hdGlvbj0idjEiPjIwMjItMDQtMjZUMTc6MTY6MzJaPC9kYXRlPgogICAgPGRhdGUgZGF0ZVR5cGU9IlVwZGF0ZWQiIGRhdGVJbmZvcm1hdGlvbj0idjEiPjIwMjItMDQtMjdUMDA6MzE6NDBaPC9kYXRlPgogICAgPGRhdGUgZGF0ZVR5cGU9IlN1Ym1pdHRlZCIgZGF0ZUluZm9ybWF0aW9uPSJ2MiI+MjAyMi0wNC0yN1QxMzowMTozMFo8L2RhdGU+CiAgICA8ZGF0ZSBkYXRlVHlwZT0iVXBkYXRlZCIgZGF0ZUluZm9ybWF0aW9uPSJ2MiI+MjAyMi0wNC0yOFQwMDoyMzowNlo8L2RhdGU+CiAgICA8ZGF0ZSBkYXRlVHlwZT0iU3VibWl0dGVkIiBkYXRlSW5mb3JtYXRpb249InYzIj4yMDIyLTEwLTIxVDE1OjI5OjAxWjwvZGF0ZT4KICAgIDxkYXRlIGRhdGVUeXBlPSJVcGRhdGVkIiBkYXRlSW5mb3JtYXRpb249InYzIj4yMDIyLTEwLTI0VDAwOjE1OjIxWjwvZGF0ZT4KICAgIDxkYXRlIGRhdGVUeXBlPSJTdWJtaXR0ZWQiIGRhdGVJbmZvcm1hdGlvbj0idjQiPjIwMjItMTEtMDhUMTU6NTI6MzlaPC9kYXRlPgogICAgPGRhdGUgZGF0ZVR5cGU9IlVwZGF0ZWQiIGRhdGVJbmZvcm1hdGlvbj0idjQiPjIwMjItMTEtMDlUMDE6MTY6MDJaPC9kYXRlPgogICAgPGRhdGUgZGF0ZVR5cGU9IkF2YWlsYWJsZSIgZGF0ZUluZm9ybWF0aW9uPSJ2MSI+MjAyMi0wNDwvZGF0ZT4KICA8L2RhdGVzPgogIDxyZXNvdXJjZVR5cGUgcmVzb3VyY2VUeXBlR2VuZXJhbD0iUHJlcHJpbnQiPkFydGljbGU8L3Jlc291cmNlVHlwZT4KICA8dmVyc2lvbj40PC92ZXJzaW9uPgogIDxyaWdodHNMaXN0PgogICAgPHJpZ2h0cyByaWdodHNVUkk9Imh0dHA6Ly9hcnhpdi5vcmcvbGljZW5zZXMvbm9uZXhjbHVzaXZlLWRpc3RyaWIvMS4wLyI+YXJYaXYub3JnIHBlcnBldHVhbCwgbm9uLWV4Y2x1c2l2ZSBsaWNlbnNlPC9yaWdodHM+CiAgPC9yaWdodHNMaXN0PgogIDxkZXNjcmlwdGlvbnM+CiAgICA8ZGVzY3JpcHRpb24gZGVzY3JpcHRpb25UeXBlPSJBYnN0cmFjdCI+UmVjZW50IHN0dWRpZXMgc2hvdyB0aGF0IFZpc2lvbiBUcmFuc2Zvcm1lcnMoVmlUcykgZXhoaWJpdCBzdHJvbmcgcm9idXN0bmVzcyBhZ2FpbnN0IHZhcmlvdXMgY29ycnVwdGlvbnMuIEFsdGhvdWdoIHRoaXMgcHJvcGVydHkgaXMgcGFydGx5IGF0dHJpYnV0ZWQgdG8gdGhlIHNlbGYtYXR0ZW50aW9uIG1lY2hhbmlzbSwgdGhlcmUgaXMgc3RpbGwgYSBsYWNrIG9mIHN5c3RlbWF0aWMgdW5kZXJzdGFuZGluZy4gSW4gdGhpcyBwYXBlciwgd2UgZXhhbWluZSB0aGUgcm9sZSBvZiBzZWxmLWF0dGVudGlvbiBpbiBsZWFybmluZyByb2J1c3QgcmVwcmVzZW50YXRpb25zLiBPdXIgc3R1ZHkgaXMgbW90aXZhdGVkIGJ5IHRoZSBpbnRyaWd1aW5nIHByb3BlcnRpZXMgb2YgdGhlIGVtZXJnaW5nIHZpc3VhbCBncm91cGluZyBpbiBWaXNpb24gVHJhbnNmb3JtZXJzLCB3aGljaCBpbmRpY2F0ZXMgdGhhdCBzZWxmLWF0dGVudGlvbiBtYXkgcHJvbW90ZSByb2J1c3RuZXNzIHRocm91Z2ggaW1wcm92ZWQgbWlkLWxldmVsIHJlcHJlc2VudGF0aW9ucy4gV2UgZnVydGhlciBwcm9wb3NlIGEgZmFtaWx5IG9mIGZ1bGx5IGF0dGVudGlvbmFsIG5ldHdvcmtzIChGQU5zKSB0aGF0IHN0cmVuZ3RoZW4gdGhpcyBjYXBhYmlsaXR5IGJ5IGluY29ycG9yYXRpbmcgYW4gYXR0ZW50aW9uYWwgY2hhbm5lbCBwcm9jZXNzaW5nIGRlc2lnbi4gV2UgdmFsaWRhdGUgdGhlIGRlc2lnbiBjb21wcmVoZW5zaXZlbHkgb24gdmFyaW91cyBoaWVyYXJjaGljYWwgYmFja2JvbmVzLiBPdXIgbW9kZWwgYWNoaWV2ZXMgYSBzdGF0ZS1vZi10aGUtYXJ0IDg3LjElIGFjY3VyYWN5IGFuZCAzNS44JSBtQ0Ugb24gSW1hZ2VOZXQtMWsgYW5kIEltYWdlTmV0LUMgd2l0aCA3Ni44TSBwYXJhbWV0ZXJzLiBXZSBhbHNvIGRlbW9uc3RyYXRlIHN0YXRlLW9mLXRoZS1hcnQgYWNjdXJhY3kgYW5kIHJvYnVzdG5lc3MgaW4gdHdvIGRvd25zdHJlYW0gdGFza3M6IHNlbWFudGljIHNlZ21lbnRhdGlvbiBhbmQgb2JqZWN0IGRldGVjdGlvbi4gQ29kZSBpcyBhdmFpbGFibGUgYXQ6IGh0dHBzOi8vZ2l0aHViLmNvbS9OVmxhYnMvRkFOLjwvZGVzY3JpcHRpb24+CiAgPC9kZXNjcmlwdGlvbnM+CjwvcmVzb3VyY2U+","url":"https://arxiv.org/abs/2204.12451","contentUrl":null,"metadataVersion":3,"schemaVersion":"http://datacite.org/schema/kernel-4","source":"mds","isActive":true,"state":"findable","reason":null,"viewCount":0,"viewsOverTime":[],"downloadCount":0,"downloadsOverTime":[],"referenceCount":0,"citationCount":0,"citationsOverTime":[],"partCount":0,"partOfCount":0,"versionCount":0,"versionOfCount":0,"created":"2022-04-27T01:24:51.000Z","registered":"2022-04-27T01:24:52.000Z","published":"2022","updated":"2022-11-09T02:06:00.000Z"},"relationships":{"client":{"data":{"id":"arxiv.content","type":"clients"}},"provider":{"data":{"id":"arxiv","type":"providers"}},"media":{"data":{"id":"10.48550/arxiv.2204.12451","type":"media"}},"references":{"data":[]},"citations":{"data":[]},"parts":{"data":[]},"partOf":{"data":[]},"versions":{"data":[]},"versionOf":{"data":[]}}}}