{"data":{"id":"10.48550/arxiv.2109.08029","type":"dois","attributes":{"doi":"10.48550/arxiv.2109.08029","prefix":"10.48550","suffix":"arxiv.2109.08029","identifiers":[{"identifier":"2109.08029","identifierType":"arXiv"}],"alternateIdentifiers":[{"alternateIdentifierType":"arXiv","alternateIdentifier":"2109.08029"}],"creators":[{"name":"Salaberria, Ander","nameType":"Personal","givenName":"Ander","familyName":"Salaberria","affiliation":[],"nameIdentifiers":[]},{"name":"Azkune, Gorka","nameType":"Personal","givenName":"Gorka","familyName":"Azkune","affiliation":[],"nameIdentifiers":[]},{"name":"de Lacalle, Oier Lopez","nameType":"Personal","givenName":"Oier Lopez","familyName":"de Lacalle","affiliation":[],"nameIdentifiers":[]},{"name":"Soroa, Aitor","nameType":"Personal","givenName":"Aitor","familyName":"Soroa","affiliation":[],"nameIdentifiers":[]},{"name":"Agirre, Eneko","nameType":"Personal","givenName":"Eneko","familyName":"Agirre","affiliation":[],"nameIdentifiers":[]}],"titles":[{"title":"Image Captioning for Effective Use of Language Models in Knowledge-Based Visual Question Answering"}],"publisher":"arXiv","container":{},"publicationYear":2021,"subjects":[{"lang":"en","subject":"Computer Vision and Pattern Recognition (cs.CV)","subjectScheme":"arXiv"},{"lang":"en","subject":"Artificial Intelligence (cs.AI)","subjectScheme":"arXiv"},{"subject":"FOS: Computer and information sciences","subjectScheme":"Fields of Science and Technology (FOS)"},{"subject":"FOS: Computer and information sciences","schemeUri":"http://www.oecd.org/science/inno/38235147.pdf","subjectScheme":"Fields of Science and Technology (FOS)"}],"contributors":[],"dates":[{"date":"2021-09-15T14:11:29Z","dateType":"Submitted","dateInformation":"v1"},{"date":"2021-09-17T00:22:48Z","dateType":"Updated","dateInformation":"v1"},{"date":"2021-10-13T10:46:34Z","dateType":"Submitted","dateInformation":"v2"},{"date":"2021-10-14T00:18:00Z","dateType":"Updated","dateInformation":"v2"},{"date":"2022-03-25T08:11:17Z","dateType":"Submitted","dateInformation":"v3"},{"date":"2022-09-14T00:19:40Z","dateType":"Updated","dateInformation":"v3"},{"date":"2021-09","dateType":"Available","dateInformation":"v1"},{"date":"2021","dateType":"Issued"}],"language":null,"types":{"ris":"RPRT","bibtex":"article","citeproc":"article-journal","schemaOrg":"ScholarlyArticle","resourceType":"Article","resourceTypeGeneral":"Text"},"relatedIdentifiers":[{"relationType":"IsVersionOf","relatedIdentifier":"10.1016/j.eswa.2022.118669","relatedIdentifierType":"DOI"}],"relatedItems":[],"sizes":[],"formats":[],"version":"3","rightsList":[{"rights":"Creative Commons Attribution 4.0 International","rightsUri":"https://creativecommons.org/licenses/by/4.0/legalcode","schemeUri":"https://spdx.org/licenses/","rightsIdentifier":"cc-by-4.0","rightsIdentifierScheme":"SPDX"}],"descriptions":[{"description":"Integrating outside knowledge for reasoning in visio-linguistic tasks such as visual question answering (VQA) is an open problem. Given that pretrained language models have been shown to include world knowledge, we propose to use a unimodal (text-only) train and inference procedure based on automatic off-the-shelf captioning of images and pretrained language models. Our results on a visual question answering task which requires external knowledge (OK-VQA) show that our text-only model outperforms pretrained multimodal (image-text) models of comparable number of parameters. In contrast, our model is less effective in a standard VQA task (VQA 2.0) confirming that our text-only method is specially effective for tasks requiring external knowledge. In addition, we show that increasing the language model's size improves notably its performance, yielding results comparable to the state-of-the-art with our largest model, significantly outperforming current multimodal systems, even though augmented with external knowledge. Our qualitative analysis on OK-VQA reveals that automatic captions often fail to capture relevant information in the images, which seems to be balanced by the better inference ability of the text-only language models. Our work opens up possibilities to further improve inference in visio-linguistic tasks","descriptionType":"Abstract"},{"description":"Under review. 25 pages with 4 figures","descriptionType":"Other"}],"geoLocations":[],"fundingReferences":[],"xml":"PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0idXRmLTgiPz4KPHJlc291cmNlIHhtbG5zPSJodHRwOi8vZGF0YWNpdGUub3JnL3NjaGVtYS9rZXJuZWwtNCIgeG1sbnM6eHNpPSJodHRwOi8vd3d3LnczLm9yZy8yMDAxL1hNTFNjaGVtYS1pbnN0YW5jZSIgeHNpOnNjaGVtYUxvY2F0aW9uPSJodHRwOi8vZGF0YWNpdGUub3JnL3NjaGVtYS9rZXJuZWwtNCBodHRwOi8vc2NoZW1hLmRhdGFjaXRlLm9yZy9tZXRhL2tlcm5lbC00LjMvbWV0YWRhdGEueHNkIj4KICA8aWRlbnRpZmllciBpZGVudGlmaWVyVHlwZT0iRE9JIj4xMC40ODU1MC9BUlhJVi4yMTA5LjA4MDI5PC9pZGVudGlmaWVyPgogIDxhbHRlcm5hdGVJZGVudGlmaWVycz4KICAgIDxhbHRlcm5hdGVJZGVudGlmaWVyIGFsdGVybmF0ZUlkZW50aWZpZXJUeXBlPSJhclhpdiI+MjEwOS4wODAyOTwvYWx0ZXJuYXRlSWRlbnRpZmllcj4KICA8L2FsdGVybmF0ZUlkZW50aWZpZXJzPgogIDxjcmVhdG9ycz4KICAgIDxjcmVhdG9yPgogICAgICA8Y3JlYXRvck5hbWUgbmFtZVR5cGU9IlBlcnNvbmFsIj5TYWxhYmVycmlhLCBBbmRlcjwvY3JlYXRvck5hbWU+CiAgICAgIDxnaXZlbk5hbWU+QW5kZXI8L2dpdmVuTmFtZT4KICAgICAgPGZhbWlseU5hbWU+U2FsYWJlcnJpYTwvZmFtaWx5TmFtZT4KICAgIDwvY3JlYXRvcj4KICAgIDxjcmVhdG9yPgogICAgICA8Y3JlYXRvck5hbWUgbmFtZVR5cGU9IlBlcnNvbmFsIj5Bemt1bmUsIEdvcmthPC9jcmVhdG9yTmFtZT4KICAgICAgPGdpdmVuTmFtZT5Hb3JrYTwvZ2l2ZW5OYW1lPgogICAgICA8ZmFtaWx5TmFtZT5Bemt1bmU8L2ZhbWlseU5hbWU+CiAgICA8L2NyZWF0b3I+CiAgICA8Y3JlYXRvcj4KICAgICAgPGNyZWF0b3JOYW1lIG5hbWVUeXBlPSJQZXJzb25hbCI+ZGUgTGFjYWxsZSwgT2llciBMb3BlejwvY3JlYXRvck5hbWU+CiAgICAgIDxnaXZlbk5hbWU+T2llciBMb3BlejwvZ2l2ZW5OYW1lPgogICAgICA8ZmFtaWx5TmFtZT5kZSBMYWNhbGxlPC9mYW1pbHlOYW1lPgogICAgPC9jcmVhdG9yPgogICAgPGNyZWF0b3I+CiAgICAgIDxjcmVhdG9yTmFtZSBuYW1lVHlwZT0iUGVyc29uYWwiPlNvcm9hLCBBaXRvcjwvY3JlYXRvck5hbWU+CiAgICAgIDxnaXZlbk5hbWU+QWl0b3I8L2dpdmVuTmFtZT4KICAgICAgPGZhbWlseU5hbWU+U29yb2E8L2ZhbWlseU5hbWU+CiAgICA8L2NyZWF0b3I+CiAgICA8Y3JlYXRvcj4KICAgICAgPGNyZWF0b3JOYW1lIG5hbWVUeXBlPSJQZXJzb25hbCI+QWdpcnJlLCBFbmVrbzwvY3JlYXRvck5hbWU+CiAgICAgIDxnaXZlbk5hbWU+RW5la288L2dpdmVuTmFtZT4KICAgICAgPGZhbWlseU5hbWU+QWdpcnJlPC9mYW1pbHlOYW1lPgogICAgPC9jcmVhdG9yPgogIDwvY3JlYXRvcnM+CiAgPHRpdGxlcz4KICAgIDx0aXRsZT5JbWFnZSBDYXB0aW9uaW5nIGZvciBFZmZlY3RpdmUgVXNlIG9mIExhbmd1YWdlIE1vZGVscyBpbiBLbm93bGVkZ2UtQmFzZWQgVmlzdWFsIFF1ZXN0aW9uIEFuc3dlcmluZzwvdGl0bGU+CiAgPC90aXRsZXM+CiAgPHB1Ymxpc2hlcj5hclhpdjwvcHVibGlzaGVyPgogIDxwdWJsaWNhdGlvblllYXI+MjAyMTwvcHVibGljYXRpb25ZZWFyPgogIDxzdWJqZWN0cz4KICAgIDxzdWJqZWN0IHhtbDpsYW5nPSJlbiIgc3ViamVjdFNjaGVtZT0iYXJYaXYiPkNvbXB1dGVyIFZpc2lvbiBhbmQgUGF0dGVybiBSZWNvZ25pdGlvbiAoY3MuQ1YpPC9zdWJqZWN0PgogICAgPHN1YmplY3QgeG1sOmxhbmc9ImVuIiBzdWJqZWN0U2NoZW1lPSJhclhpdiI+QXJ0aWZpY2lhbCBJbnRlbGxpZ2VuY2UgKGNzLkFJKTwvc3ViamVjdD4KICAgIDxzdWJqZWN0IHN1YmplY3RTY2hlbWU9IkZpZWxkcyBvZiBTY2llbmNlIGFuZCBUZWNobm9sb2d5IChGT1MpIj5GT1M6IENvbXB1dGVyIGFuZCBpbmZvcm1hdGlvbiBzY2llbmNlczwvc3ViamVjdD4KICA8L3N1YmplY3RzPgogIDxkYXRlcz4KICAgIDxkYXRlIGRhdGVUeXBlPSJTdWJtaXR0ZWQiIGRhdGVJbmZvcm1hdGlvbj0idjEiPjIwMjEtMDktMTVUMTQ6MTE6MjlaPC9kYXRlPgogICAgPGRhdGUgZGF0ZVR5cGU9IlVwZGF0ZWQiIGRhdGVJbmZvcm1hdGlvbj0idjEiPjIwMjEtMDktMTdUMDA6MjI6NDhaPC9kYXRlPgogICAgPGRhdGUgZGF0ZVR5cGU9IlN1Ym1pdHRlZCIgZGF0ZUluZm9ybWF0aW9uPSJ2MiI+MjAyMS0xMC0xM1QxMDo0NjozNFo8L2RhdGU+CiAgICA8ZGF0ZSBkYXRlVHlwZT0iVXBkYXRlZCIgZGF0ZUluZm9ybWF0aW9uPSJ2MiI+MjAyMS0xMC0xNFQwMDoxODowMFo8L2RhdGU+CiAgICA8ZGF0ZSBkYXRlVHlwZT0iU3VibWl0dGVkIiBkYXRlSW5mb3JtYXRpb249InYzIj4yMDIyLTAzLTI1VDA4OjExOjE3WjwvZGF0ZT4KICAgIDxkYXRlIGRhdGVUeXBlPSJVcGRhdGVkIiBkYXRlSW5mb3JtYXRpb249InYzIj4yMDIyLTA5LTE0VDAwOjE5OjQwWjwvZGF0ZT4KICAgIDxkYXRlIGRhdGVUeXBlPSJBdmFpbGFibGUiIGRhdGVJbmZvcm1hdGlvbj0idjEiPjIwMjEtMDk8L2RhdGU+CiAgPC9kYXRlcz4KICA8cmVzb3VyY2VUeXBlIHJlc291cmNlVHlwZUdlbmVyYWw9IlRleHQiPkFydGljbGU8L3Jlc291cmNlVHlwZT4KICA8cmVsYXRlZElkZW50aWZpZXJzPgogICAgPHJlbGF0ZWRJZGVudGlmaWVyIHJlbGF0ZWRJZGVudGlmaWVyVHlwZT0iRE9JIiByZWxhdGlvblR5cGU9IklzVmVyc2lvbk9mIj4xMC4xMDE2L2ouZXN3YS4yMDIyLjExODY2OTwvcmVsYXRlZElkZW50aWZpZXI+CiAgPC9yZWxhdGVkSWRlbnRpZmllcnM+CiAgPHZlcnNpb24+MzwvdmVyc2lvbj4KICA8cmlnaHRzTGlzdD4KICAgIDxyaWdodHMgcmlnaHRzVVJJPSJodHRwOi8vY3JlYXRpdmVjb21tb25zLm9yZy9saWNlbnNlcy9ieS80LjAvIiByaWdodHNJZGVudGlmaWVyU2NoZW1lPSJTUERYIiByaWdodHNJZGVudGlmaWVyPSJDQy1CWS00LjAiPkNyZWF0aXZlIENvbW1vbnMgQXR0cmlidXRpb24gNC4wIEludGVybmF0aW9uYWw8L3JpZ2h0cz4KICA8L3JpZ2h0c0xpc3Q+CiAgPGRlc2NyaXB0aW9ucz4KICAgIDxkZXNjcmlwdGlvbiBkZXNjcmlwdGlvblR5cGU9IkFic3RyYWN0Ij5JbnRlZ3JhdGluZyBvdXRzaWRlIGtub3dsZWRnZSBmb3IgcmVhc29uaW5nIGluIHZpc2lvLWxpbmd1aXN0aWMgdGFza3Mgc3VjaCBhcyB2aXN1YWwgcXVlc3Rpb24gYW5zd2VyaW5nIChWUUEpIGlzIGFuIG9wZW4gcHJvYmxlbS4gR2l2ZW4gdGhhdCBwcmV0cmFpbmVkIGxhbmd1YWdlIG1vZGVscyBoYXZlIGJlZW4gc2hvd24gdG8gaW5jbHVkZSB3b3JsZCBrbm93bGVkZ2UsIHdlIHByb3Bvc2UgdG8gdXNlIGEgdW5pbW9kYWwgKHRleHQtb25seSkgdHJhaW4gYW5kIGluZmVyZW5jZSBwcm9jZWR1cmUgYmFzZWQgb24gYXV0b21hdGljIG9mZi10aGUtc2hlbGYgY2FwdGlvbmluZyBvZiBpbWFnZXMgYW5kIHByZXRyYWluZWQgbGFuZ3VhZ2UgbW9kZWxzLiBPdXIgcmVzdWx0cyBvbiBhIHZpc3VhbCBxdWVzdGlvbiBhbnN3ZXJpbmcgdGFzayB3aGljaCByZXF1aXJlcyBleHRlcm5hbCBrbm93bGVkZ2UgKE9LLVZRQSkgc2hvdyB0aGF0IG91ciB0ZXh0LW9ubHkgbW9kZWwgb3V0cGVyZm9ybXMgcHJldHJhaW5lZCBtdWx0aW1vZGFsIChpbWFnZS10ZXh0KSBtb2RlbHMgb2YgY29tcGFyYWJsZSBudW1iZXIgb2YgcGFyYW1ldGVycy4gSW4gY29udHJhc3QsIG91ciBtb2RlbCBpcyBsZXNzIGVmZmVjdGl2ZSBpbiBhIHN0YW5kYXJkIFZRQSB0YXNrIChWUUEgMi4wKSBjb25maXJtaW5nIHRoYXQgb3VyIHRleHQtb25seSBtZXRob2QgaXMgc3BlY2lhbGx5IGVmZmVjdGl2ZSBmb3IgdGFza3MgcmVxdWlyaW5nIGV4dGVybmFsIGtub3dsZWRnZS4gSW4gYWRkaXRpb24sIHdlIHNob3cgdGhhdCBpbmNyZWFzaW5nIHRoZSBsYW5ndWFnZSBtb2RlbCdzIHNpemUgaW1wcm92ZXMgbm90YWJseSBpdHMgcGVyZm9ybWFuY2UsIHlpZWxkaW5nIHJlc3VsdHMgY29tcGFyYWJsZSB0byB0aGUgc3RhdGUtb2YtdGhlLWFydCB3aXRoIG91ciBsYXJnZXN0IG1vZGVsLCBzaWduaWZpY2FudGx5IG91dHBlcmZvcm1pbmcgY3VycmVudCBtdWx0aW1vZGFsIHN5c3RlbXMsIGV2ZW4gdGhvdWdoIGF1Z21lbnRlZCB3aXRoIGV4dGVybmFsIGtub3dsZWRnZS4gT3VyIHF1YWxpdGF0aXZlIGFuYWx5c2lzIG9uIE9LLVZRQSByZXZlYWxzIHRoYXQgYXV0b21hdGljIGNhcHRpb25zIG9mdGVuIGZhaWwgdG8gY2FwdHVyZSByZWxldmFudCBpbmZvcm1hdGlvbiBpbiB0aGUgaW1hZ2VzLCB3aGljaCBzZWVtcyB0byBiZSBiYWxhbmNlZCBieSB0aGUgYmV0dGVyIGluZmVyZW5jZSBhYmlsaXR5IG9mIHRoZSB0ZXh0LW9ubHkgbGFuZ3VhZ2UgbW9kZWxzLiBPdXIgd29yayBvcGVucyB1cCBwb3NzaWJpbGl0aWVzIHRvIGZ1cnRoZXIgaW1wcm92ZSBpbmZlcmVuY2UgaW4gdmlzaW8tbGluZ3Vpc3RpYyB0YXNrczwvZGVzY3JpcHRpb24+CiAgICA8ZGVzY3JpcHRpb24gZGVzY3JpcHRpb25UeXBlPSJPdGhlciI+VW5kZXIgcmV2aWV3LiAyNSBwYWdlcyB3aXRoIDQgZmlndXJlczwvZGVzY3JpcHRpb24+CiAgPC9kZXNjcmlwdGlvbnM+CjwvcmVzb3VyY2U+","url":"https://arxiv.org/abs/2109.08029","contentUrl":null,"metadataVersion":2,"schemaVersion":"http://datacite.org/schema/kernel-4","source":"mds","isActive":true,"state":"findable","reason":null,"viewCount":0,"viewsOverTime":[],"downloadCount":0,"downloadsOverTime":[],"referenceCount":0,"citationCount":0,"citationsOverTime":[],"partCount":0,"partOfCount":0,"versionCount":0,"versionOfCount":1,"created":"2022-02-21T13:46:23.000Z","registered":"2022-02-21T13:46:24.000Z","published":"2021","updated":"2022-09-14T01:21:27.000Z"},"relationships":{"client":{"data":{"id":"arxiv.content","type":"clients"}},"provider":{"data":{"id":"arxiv","type":"providers"}},"media":{"data":{"id":"10.48550/arxiv.2109.08029","type":"media"}},"references":{"data":[]},"citations":{"data":[]},"parts":{"data":[]},"partOf":{"data":[]},"versions":{"data":[]},"versionOf":{"data":[{"id":"10.1016/j.eswa.2022.118669","type":"dois"}]}}}}