{"data":{"id":"10.48550/arxiv.2203.00242","type":"dois","attributes":{"doi":"10.48550/arxiv.2203.00242","prefix":"10.48550","suffix":"arxiv.2203.00242","identifiers":[{"identifier":"2203.00242","identifierType":"arXiv"}],"alternateIdentifiers":[{"alternateIdentifierType":"arXiv","alternateIdentifier":"2203.00242"}],"creators":[{"name":"Zhou, Mingyang","nameType":"Personal","givenName":"Mingyang","familyName":"Zhou","affiliation":[],"nameIdentifiers":[]},{"name":"Yu, Licheng","nameType":"Personal","givenName":"Licheng","familyName":"Yu","affiliation":[],"nameIdentifiers":[]},{"name":"Singh, Amanpreet","nameType":"Personal","givenName":"Amanpreet","familyName":"Singh","affiliation":[],"nameIdentifiers":[]},{"name":"Wang, Mengjiao","nameType":"Personal","givenName":"Mengjiao","familyName":"Wang","affiliation":[],"nameIdentifiers":[]},{"name":"Yu, Zhou","nameType":"Personal","givenName":"Zhou","familyName":"Yu","affiliation":[],"nameIdentifiers":[]},{"name":"Zhang, Ning","nameType":"Personal","givenName":"Ning","familyName":"Zhang","affiliation":[],"nameIdentifiers":[]}],"titles":[{"title":"Unsupervised Vision-and-Language Pre-training via Retrieval-based Multi-Granular Alignment"}],"publisher":"arXiv","container":{},"publicationYear":2022,"subjects":[{"lang":"en","subject":"Computer Vision and Pattern Recognition (cs.CV)","subjectScheme":"arXiv"},{"lang":"en","subject":"Artificial Intelligence (cs.AI)","subjectScheme":"arXiv"},{"lang":"en","subject":"Computation and Language (cs.CL)","subjectScheme":"arXiv"},{"lang":"en","subject":"Multimedia (cs.MM)","subjectScheme":"arXiv"},{"subject":"FOS: Computer and information sciences","subjectScheme":"Fields of Science and Technology (FOS)"},{"subject":"FOS: Computer and information sciences","schemeUri":"http://www.oecd.org/science/inno/38235147.pdf","subjectScheme":"Fields of Science and Technology (FOS)"}],"contributors":[],"dates":[{"date":"2022-03-01T05:34:01Z","dateType":"Submitted","dateInformation":"v1"},{"date":"2022-03-02T01:16:13Z","dateType":"Updated","dateInformation":"v1"},{"date":"2022-03","dateType":"Available","dateInformation":"v1"},{"date":"2022","dateType":"Issued"}],"language":null,"types":{"ris":"GEN","bibtex":"misc","citeproc":"article","schemaOrg":"CreativeWork","resourceType":"Article","resourceTypeGeneral":"Preprint"},"relatedIdentifiers":[],"relatedItems":[],"sizes":[],"formats":[],"version":"1","rightsList":[{"rights":"arXiv.org perpetual, non-exclusive license","rightsUri":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/"}],"descriptions":[{"description":"Vision-and-Language (V+L) pre-training models have achieved tremendous success in recent years on various multi-modal benchmarks. However, the majority of existing models require pre-training on a large set of parallel image-text data, which is costly to collect, compared to image-only or text-only data. In this paper, we explore unsupervised Vision-and-Language pre-training (UVLP) to learn the cross-modal representation from non-parallel image and text datasets. We found two key factors that lead to good unsupervised V+L pre-training without parallel data: (i) joint image-and-text input (ii) overall image-text alignment (even for non-parallel data). Accordingly, we propose a novel unsupervised V+L pre-training curriculum for non-parallel texts and images. We first construct a weakly aligned image-text corpus via a retrieval-based approach, then apply a set of multi-granular alignment pre-training tasks, including region-to-tag, region-to-phrase, and image-to-sentence alignment, to bridge the gap between the two modalities. A comprehensive ablation study shows each granularity is helpful to learn a stronger pre-trained model. We adapt our pre-trained model to a set of V+L downstream tasks, including VQA, NLVR2, Visual Entailment, and RefCOCO+. Our model achieves the state-of-art performance in all these tasks under the unsupervised setting.","descriptionType":"Abstract"},{"description":"First two authors contributed equally. 13 pages, 11 figures","descriptionType":"Other"}],"geoLocations":[],"fundingReferences":[],"xml":"PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0idXRmLTgiPz4KPHJlc291cmNlIHhtbG5zPSJodHRwOi8vZGF0YWNpdGUub3JnL3NjaGVtYS9rZXJuZWwtNCIgeG1sbnM6eHNpPSJodHRwOi8vd3d3LnczLm9yZy8yMDAxL1hNTFNjaGVtYS1pbnN0YW5jZSIgeHNpOnNjaGVtYUxvY2F0aW9uPSJodHRwOi8vZGF0YWNpdGUub3JnL3NjaGVtYS9rZXJuZWwtNCBodHRwOi8vc2NoZW1hLmRhdGFjaXRlLm9yZy9tZXRhL2tlcm5lbC00LjMvbWV0YWRhdGEueHNkIj4KICA8aWRlbnRpZmllciBpZGVudGlmaWVyVHlwZT0iRE9JIj4xMC40ODU1MC9BUlhJVi4yMjAzLjAwMjQyPC9pZGVudGlmaWVyPgogIDxhbHRlcm5hdGVJZGVudGlmaWVycz4KICAgIDxhbHRlcm5hdGVJZGVudGlmaWVyIGFsdGVybmF0ZUlkZW50aWZpZXJUeXBlPSJhclhpdiI+MjIwMy4wMDI0MjwvYWx0ZXJuYXRlSWRlbnRpZmllcj4KICA8L2FsdGVybmF0ZUlkZW50aWZpZXJzPgogIDxjcmVhdG9ycz4KICAgIDxjcmVhdG9yPgogICAgICA8Y3JlYXRvck5hbWUgbmFtZVR5cGU9IlBlcnNvbmFsIj5aaG91LCBNaW5neWFuZzwvY3JlYXRvck5hbWU+CiAgICAgIDxnaXZlbk5hbWU+TWluZ3lhbmc8L2dpdmVuTmFtZT4KICAgICAgPGZhbWlseU5hbWU+WmhvdTwvZmFtaWx5TmFtZT4KICAgIDwvY3JlYXRvcj4KICAgIDxjcmVhdG9yPgogICAgICA8Y3JlYXRvck5hbWUgbmFtZVR5cGU9IlBlcnNvbmFsIj5ZdSwgTGljaGVuZzwvY3JlYXRvck5hbWU+CiAgICAgIDxnaXZlbk5hbWU+TGljaGVuZzwvZ2l2ZW5OYW1lPgogICAgICA8ZmFtaWx5TmFtZT5ZdTwvZmFtaWx5TmFtZT4KICAgIDwvY3JlYXRvcj4KICAgIDxjcmVhdG9yPgogICAgICA8Y3JlYXRvck5hbWUgbmFtZVR5cGU9IlBlcnNvbmFsIj5TaW5naCwgQW1hbnByZWV0PC9jcmVhdG9yTmFtZT4KICAgICAgPGdpdmVuTmFtZT5BbWFucHJlZXQ8L2dpdmVuTmFtZT4KICAgICAgPGZhbWlseU5hbWU+U2luZ2g8L2ZhbWlseU5hbWU+CiAgICA8L2NyZWF0b3I+CiAgICA8Y3JlYXRvcj4KICAgICAgPGNyZWF0b3JOYW1lIG5hbWVUeXBlPSJQZXJzb25hbCI+V2FuZywgTWVuZ2ppYW88L2NyZWF0b3JOYW1lPgogICAgICA8Z2l2ZW5OYW1lPk1lbmdqaWFvPC9naXZlbk5hbWU+CiAgICAgIDxmYW1pbHlOYW1lPldhbmc8L2ZhbWlseU5hbWU+CiAgICA8L2NyZWF0b3I+CiAgICA8Y3JlYXRvcj4KICAgICAgPGNyZWF0b3JOYW1lIG5hbWVUeXBlPSJQZXJzb25hbCI+WXUsIFpob3U8L2NyZWF0b3JOYW1lPgogICAgICA8Z2l2ZW5OYW1lPlpob3U8L2dpdmVuTmFtZT4KICAgICAgPGZhbWlseU5hbWU+WXU8L2ZhbWlseU5hbWU+CiAgICA8L2NyZWF0b3I+CiAgICA8Y3JlYXRvcj4KICAgICAgPGNyZWF0b3JOYW1lIG5hbWVUeXBlPSJQZXJzb25hbCI+WmhhbmcsIE5pbmc8L2NyZWF0b3JOYW1lPgogICAgICA8Z2l2ZW5OYW1lPk5pbmc8L2dpdmVuTmFtZT4KICAgICAgPGZhbWlseU5hbWU+Wmhhbmc8L2ZhbWlseU5hbWU+CiAgICA8L2NyZWF0b3I+CiAgPC9jcmVhdG9ycz4KICA8dGl0bGVzPgogICAgPHRpdGxlPlVuc3VwZXJ2aXNlZCBWaXNpb24tYW5kLUxhbmd1YWdlIFByZS10cmFpbmluZyB2aWEgUmV0cmlldmFsLWJhc2VkIE11bHRpLUdyYW51bGFyIEFsaWdubWVudDwvdGl0bGU+CiAgPC90aXRsZXM+CiAgPHB1Ymxpc2hlcj5hclhpdjwvcHVibGlzaGVyPgogIDxwdWJsaWNhdGlvblllYXI+MjAyMjwvcHVibGljYXRpb25ZZWFyPgogIDxzdWJqZWN0cz4KICAgIDxzdWJqZWN0IHhtbDpsYW5nPSJlbiIgc3ViamVjdFNjaGVtZT0iYXJYaXYiPkNvbXB1dGVyIFZpc2lvbiBhbmQgUGF0dGVybiBSZWNvZ25pdGlvbiAoY3MuQ1YpPC9zdWJqZWN0PgogICAgPHN1YmplY3QgeG1sOmxhbmc9ImVuIiBzdWJqZWN0U2NoZW1lPSJhclhpdiI+QXJ0aWZpY2lhbCBJbnRlbGxpZ2VuY2UgKGNzLkFJKTwvc3ViamVjdD4KICAgIDxzdWJqZWN0IHhtbDpsYW5nPSJlbiIgc3ViamVjdFNjaGVtZT0iYXJYaXYiPkNvbXB1dGF0aW9uIGFuZCBMYW5ndWFnZSAoY3MuQ0wpPC9zdWJqZWN0PgogICAgPHN1YmplY3QgeG1sOmxhbmc9ImVuIiBzdWJqZWN0U2NoZW1lPSJhclhpdiI+TXVsdGltZWRpYSAoY3MuTU0pPC9zdWJqZWN0PgogICAgPHN1YmplY3Qgc3ViamVjdFNjaGVtZT0iRmllbGRzIG9mIFNjaWVuY2UgYW5kIFRlY2hub2xvZ3kgKEZPUykiPkZPUzogQ29tcHV0ZXIgYW5kIGluZm9ybWF0aW9uIHNjaWVuY2VzPC9zdWJqZWN0PgogIDwvc3ViamVjdHM+CiAgPGRhdGVzPgogICAgPGRhdGUgZGF0ZVR5cGU9IlN1Ym1pdHRlZCIgZGF0ZUluZm9ybWF0aW9uPSJ2MSI+MjAyMi0wMy0wMVQwNTozNDowMVo8L2RhdGU+CiAgICA8ZGF0ZSBkYXRlVHlwZT0iVXBkYXRlZCIgZGF0ZUluZm9ybWF0aW9uPSJ2MSI+MjAyMi0wMy0wMlQwMToxNjoxM1o8L2RhdGU+CiAgICA8ZGF0ZSBkYXRlVHlwZT0iQXZhaWxhYmxlIiBkYXRlSW5mb3JtYXRpb249InYxIj4yMDIyLTAzPC9kYXRlPgogIDwvZGF0ZXM+CiAgPHJlc291cmNlVHlwZSByZXNvdXJjZVR5cGVHZW5lcmFsPSJQcmVwcmludCI+QXJ0aWNsZTwvcmVzb3VyY2VUeXBlPgogIDx2ZXJzaW9uPjE8L3ZlcnNpb24+CiAgPHJpZ2h0c0xpc3Q+CiAgICA8cmlnaHRzIHJpZ2h0c1VSST0iaHR0cDovL2FyeGl2Lm9yZy9saWNlbnNlcy9ub25leGNsdXNpdmUtZGlzdHJpYi8xLjAvIj5hclhpdi5vcmcgcGVycGV0dWFsLCBub24tZXhjbHVzaXZlIGxpY2Vuc2U8L3JpZ2h0cz4KICA8L3JpZ2h0c0xpc3Q+CiAgPGRlc2NyaXB0aW9ucz4KICAgIDxkZXNjcmlwdGlvbiBkZXNjcmlwdGlvblR5cGU9IkFic3RyYWN0Ij5WaXNpb24tYW5kLUxhbmd1YWdlIChWK0wpIHByZS10cmFpbmluZyBtb2RlbHMgaGF2ZSBhY2hpZXZlZCB0cmVtZW5kb3VzIHN1Y2Nlc3MgaW4gcmVjZW50IHllYXJzIG9uIHZhcmlvdXMgbXVsdGktbW9kYWwgYmVuY2htYXJrcy4gSG93ZXZlciwgdGhlIG1ham9yaXR5IG9mIGV4aXN0aW5nIG1vZGVscyByZXF1aXJlIHByZS10cmFpbmluZyBvbiBhIGxhcmdlIHNldCBvZiBwYXJhbGxlbCBpbWFnZS10ZXh0IGRhdGEsIHdoaWNoIGlzIGNvc3RseSB0byBjb2xsZWN0LCBjb21wYXJlZCB0byBpbWFnZS1vbmx5IG9yIHRleHQtb25seSBkYXRhLiBJbiB0aGlzIHBhcGVyLCB3ZSBleHBsb3JlIHVuc3VwZXJ2aXNlZCBWaXNpb24tYW5kLUxhbmd1YWdlIHByZS10cmFpbmluZyAoVVZMUCkgdG8gbGVhcm4gdGhlIGNyb3NzLW1vZGFsIHJlcHJlc2VudGF0aW9uIGZyb20gbm9uLXBhcmFsbGVsIGltYWdlIGFuZCB0ZXh0IGRhdGFzZXRzLiBXZSBmb3VuZCB0d28ga2V5IGZhY3RvcnMgdGhhdCBsZWFkIHRvIGdvb2QgdW5zdXBlcnZpc2VkIFYrTCBwcmUtdHJhaW5pbmcgd2l0aG91dCBwYXJhbGxlbCBkYXRhOiAoaSkgam9pbnQgaW1hZ2UtYW5kLXRleHQgaW5wdXQgKGlpKSBvdmVyYWxsIGltYWdlLXRleHQgYWxpZ25tZW50IChldmVuIGZvciBub24tcGFyYWxsZWwgZGF0YSkuIEFjY29yZGluZ2x5LCB3ZSBwcm9wb3NlIGEgbm92ZWwgdW5zdXBlcnZpc2VkIFYrTCBwcmUtdHJhaW5pbmcgY3VycmljdWx1bSBmb3Igbm9uLXBhcmFsbGVsIHRleHRzIGFuZCBpbWFnZXMuIFdlIGZpcnN0IGNvbnN0cnVjdCBhIHdlYWtseSBhbGlnbmVkIGltYWdlLXRleHQgY29ycHVzIHZpYSBhIHJldHJpZXZhbC1iYXNlZCBhcHByb2FjaCwgdGhlbiBhcHBseSBhIHNldCBvZiBtdWx0aS1ncmFudWxhciBhbGlnbm1lbnQgcHJlLXRyYWluaW5nIHRhc2tzLCBpbmNsdWRpbmcgcmVnaW9uLXRvLXRhZywgcmVnaW9uLXRvLXBocmFzZSwgYW5kIGltYWdlLXRvLXNlbnRlbmNlIGFsaWdubWVudCwgdG8gYnJpZGdlIHRoZSBnYXAgYmV0d2VlbiB0aGUgdHdvIG1vZGFsaXRpZXMuIEEgY29tcHJlaGVuc2l2ZSBhYmxhdGlvbiBzdHVkeSBzaG93cyBlYWNoIGdyYW51bGFyaXR5IGlzIGhlbHBmdWwgdG8gbGVhcm4gYSBzdHJvbmdlciBwcmUtdHJhaW5lZCBtb2RlbC4gV2UgYWRhcHQgb3VyIHByZS10cmFpbmVkIG1vZGVsIHRvIGEgc2V0IG9mIFYrTCBkb3duc3RyZWFtIHRhc2tzLCBpbmNsdWRpbmcgVlFBLCBOTFZSMiwgVmlzdWFsIEVudGFpbG1lbnQsIGFuZCBSZWZDT0NPKy4gT3VyIG1vZGVsIGFjaGlldmVzIHRoZSBzdGF0ZS1vZi1hcnQgcGVyZm9ybWFuY2UgaW4gYWxsIHRoZXNlIHRhc2tzIHVuZGVyIHRoZSB1bnN1cGVydmlzZWQgc2V0dGluZy48L2Rlc2NyaXB0aW9uPgogICAgPGRlc2NyaXB0aW9uIGRlc2NyaXB0aW9uVHlwZT0iT3RoZXIiPkZpcnN0IHR3byBhdXRob3JzIGNvbnRyaWJ1dGVkIGVxdWFsbHkuIDEzIHBhZ2VzLCAxMSBmaWd1cmVzPC9kZXNjcmlwdGlvbj4KICA8L2Rlc2NyaXB0aW9ucz4KPC9yZXNvdXJjZT4=","url":"https://arxiv.org/abs/2203.00242","contentUrl":null,"metadataVersion":0,"schemaVersion":"http://datacite.org/schema/kernel-4","source":"mds","isActive":true,"state":"findable","reason":null,"viewCount":0,"viewsOverTime":[],"downloadCount":0,"downloadsOverTime":[],"referenceCount":0,"citationCount":0,"citationsOverTime":[],"partCount":0,"partOfCount":0,"versionCount":0,"versionOfCount":0,"created":"2022-03-02T02:31:55.000Z","registered":"2022-03-02T02:31:56.000Z","published":"2022","updated":"2022-03-02T02:31:56.000Z"},"relationships":{"client":{"data":{"id":"arxiv.content","type":"clients"}},"provider":{"data":{"id":"arxiv","type":"providers"}},"media":{"data":{"id":"10.48550/arxiv.2203.00242","type":"media"}},"references":{"data":[]},"citations":{"data":[]},"parts":{"data":[]},"partOf":{"data":[]},"versions":{"data":[]},"versionOf":{"data":[]}}}}