{"data":{"id":"10.48550/arxiv.2302.02083","type":"dois","attributes":{"doi":"10.48550/arxiv.2302.02083","prefix":"10.48550","suffix":"arxiv.2302.02083","identifiers":[{"identifier":"2302.02083","identifierType":"arXiv"}],"alternateIdentifiers":[{"alternateIdentifierType":"arXiv","alternateIdentifier":"2302.02083"}],"creators":[{"name":"Kosinski, Michal","nameType":"Personal","givenName":"Michal","familyName":"Kosinski","affiliation":[],"nameIdentifiers":[]}],"titles":[{"title":"Evaluating Large Language Models in Theory of Mind Tasks"}],"publisher":"arXiv","container":{},"publicationYear":2023,"subjects":[{"lang":"en","subject":"Computation and Language (cs.CL)","subjectScheme":"arXiv"},{"lang":"en","subject":"Computers and Society (cs.CY)","subjectScheme":"arXiv"},{"lang":"en","subject":"Human-Computer Interaction (cs.HC)","subjectScheme":"arXiv"},{"subject":"FOS: Computer and information sciences","subjectScheme":"Fields of Science and Technology (FOS)"},{"subject":"FOS: Computer and information sciences","schemeUri":"http://www.oecd.org/science/inno/38235147.pdf","subjectScheme":"Fields of Science and Technology (FOS)"}],"contributors":[],"dates":[{"date":"2023-02-04T03:50:01Z","dateType":"Submitted","dateInformation":"v1"},{"date":"2023-02-07T01:05:10Z","dateType":"Updated","dateInformation":"v1"},{"date":"2023-02-10T19:01:49Z","dateType":"Submitted","dateInformation":"v2"},{"date":"2023-02-14T01:00:25Z","dateType":"Updated","dateInformation":"v2"},{"date":"2023-03-14T18:49:26Z","dateType":"Submitted","dateInformation":"v3"},{"date":"2023-03-16T00:00:59Z","dateType":"Updated","dateInformation":"v3"},{"date":"2023-08-29T14:55:37Z","dateType":"Submitted","dateInformation":"v4"},{"date":"2023-08-30T00:30:55Z","dateType":"Updated","dateInformation":"v4"},{"date":"2023-11-11T23:05:44Z","dateType":"Submitted","dateInformation":"v5"},{"date":"2023-11-14T01:18:24Z","dateType":"Updated","dateInformation":"v5"},{"date":"2024-02-17T02:05:32Z","dateType":"Submitted","dateInformation":"v6"},{"date":"2024-02-20T01:10:25Z","dateType":"Updated","dateInformation":"v6"},{"date":"2024-11-04T19:51:53Z","dateType":"Submitted","dateInformation":"v7"},{"date":"2024-11-06T01:07:07Z","dateType":"Updated","dateInformation":"v7"},{"date":"2023-02","dateType":"Available","dateInformation":"v1"},{"date":"2023","dateType":"Issued"}],"language":null,"types":{"ris":"RPRT","bibtex":"article","citeproc":"article-journal","schemaOrg":"ScholarlyArticle","resourceType":"Article","resourceTypeGeneral":"Text"},"relatedIdentifiers":[{"relationType":"IsVersionOf","relatedIdentifier":"10.1073/pnas.2405460121","relatedIdentifierType":"DOI"}],"relatedItems":[],"sizes":[],"formats":[],"version":"7","rightsList":[{"rights":"Creative Commons Attribution Non Commercial No Derivatives 4.0 International","rightsUri":"https://creativecommons.org/licenses/by-nc-nd/4.0/legalcode","schemeUri":"https://spdx.org/licenses/","rightsIdentifier":"cc-by-nc-nd-4.0","rightsIdentifierScheme":"SPDX"}],"descriptions":[{"description":"Eleven Large Language Models (LLMs) were assessed using a custom-made battery of false-belief tasks, considered a gold standard in testing Theory of Mind (ToM) in humans. The battery included 640 prompts spread across 40 diverse tasks, each one including a false-belief scenario, three closely matched true-belief control scenarios, and the reversed versions of all four. To solve a single task, a model needed to correctly answer 16 prompts across all eight scenarios. Smaller and older models solved no tasks; GPT-3-davinci-003 (from November 2022) and ChatGPT-3.5-turbo (from March 2023) solved 20% of the tasks; ChatGPT-4 (from June 2023) solved 75% of the tasks, matching the performance of six-year-old children observed in past studies. We explore the potential interpretation of these findings, including the intriguing possibility that ToM, previously considered exclusive to humans, may have spontaneously emerged as a byproduct of LLMs' improving language skills.","descriptionType":"Abstract"},{"description":"TRY RUNNING ToM EXPERIMENTS ON YOUR OWN: The code and tasks used in this study are available at Colab (https://colab.research.google.com/drive/1ZRtmw87CdA4xp24DNS_Ik_uA2ypaRnoU). Don't worry if you are not an expert coder, you should be able to run this code with no-to-minimum Python skills. Or copy-paste the tasks to ChatGPT's web interface. Proceedings of the National Academy of Sciences (PNAS) 2024","descriptionType":"Other"}],"geoLocations":[],"fundingReferences":[],"xml":"PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiPz4KPHJlc291cmNlIHhtbG5zPSJodHRwOi8vZGF0YWNpdGUub3JnL3NjaGVtYS9rZXJuZWwtNCIgeG1sbnM6eHNpPSJodHRwOi8vd3d3LnczLm9yZy8yMDAxL1hNTFNjaGVtYS1pbnN0YW5jZSIgeHNpOnNjaGVtYUxvY2F0aW9uPSJodHRwOi8vZGF0YWNpdGUub3JnL3NjaGVtYS9rZXJuZWwtNCBodHRwOi8vc2NoZW1hLmRhdGFjaXRlLm9yZy9tZXRhL2tlcm5lbC00LjMvbWV0YWRhdGEueHNkIj4KICA8aWRlbnRpZmllciBpZGVudGlmaWVyVHlwZT0iRE9JIj4xMC40ODU1MC9BUlhJVi4yMzAyLjAyMDgzPC9pZGVudGlmaWVyPgogIDxhbHRlcm5hdGVJZGVudGlmaWVycz4KICAgIDxhbHRlcm5hdGVJZGVudGlmaWVyIGFsdGVybmF0ZUlkZW50aWZpZXJUeXBlPSJhclhpdiI+MjMwMi4wMjA4MzwvYWx0ZXJuYXRlSWRlbnRpZmllcj4KICA8L2FsdGVybmF0ZUlkZW50aWZpZXJzPgogIDxjcmVhdG9ycz4KICAgIDxjcmVhdG9yPgogICAgICA8Y3JlYXRvck5hbWUgbmFtZVR5cGU9IlBlcnNvbmFsIj5Lb3NpbnNraSwgTWljaGFsPC9jcmVhdG9yTmFtZT4KICAgICAgPGdpdmVuTmFtZT5NaWNoYWw8L2dpdmVuTmFtZT4KICAgICAgPGZhbWlseU5hbWU+S29zaW5za2k8L2ZhbWlseU5hbWU+CiAgICA8L2NyZWF0b3I+CiAgPC9jcmVhdG9ycz4KICA8dGl0bGVzPgogICAgPHRpdGxlPkV2YWx1YXRpbmcgTGFyZ2UgTGFuZ3VhZ2UgTW9kZWxzIGluIFRoZW9yeSBvZiBNaW5kIFRhc2tzPC90aXRsZT4KICA8L3RpdGxlcz4KICA8cHVibGlzaGVyPmFyWGl2PC9wdWJsaXNoZXI+CiAgPHB1YmxpY2F0aW9uWWVhcj4yMDIzPC9wdWJsaWNhdGlvblllYXI+CiAgPHN1YmplY3RzPgogICAgPHN1YmplY3QgeG1sOmxhbmc9ImVuIiBzdWJqZWN0U2NoZW1lPSJhclhpdiI+Q29tcHV0YXRpb24gYW5kIExhbmd1YWdlIChjcy5DTCk8L3N1YmplY3Q+CiAgICA8c3ViamVjdCB4bWw6bGFuZz0iZW4iIHN1YmplY3RTY2hlbWU9ImFyWGl2Ij5Db21wdXRlcnMgYW5kIFNvY2lldHkgKGNzLkNZKTwvc3ViamVjdD4KICAgIDxzdWJqZWN0IHhtbDpsYW5nPSJlbiIgc3ViamVjdFNjaGVtZT0iYXJYaXYiPkh1bWFuLUNvbXB1dGVyIEludGVyYWN0aW9uIChjcy5IQyk8L3N1YmplY3Q+CiAgICA8c3ViamVjdCBzdWJqZWN0U2NoZW1lPSJGaWVsZHMgb2YgU2NpZW5jZSBhbmQgVGVjaG5vbG9neSAoRk9TKSI+Rk9TOiBDb21wdXRlciBhbmQgaW5mb3JtYXRpb24gc2NpZW5jZXM8L3N1YmplY3Q+CiAgPC9zdWJqZWN0cz4KICA8ZGF0ZXM+CiAgICA8ZGF0ZSBkYXRlVHlwZT0iU3VibWl0dGVkIiBkYXRlSW5mb3JtYXRpb249InYxIj4yMDIzLTAyLTA0VDAzOjUwOjAxWjwvZGF0ZT4KICAgIDxkYXRlIGRhdGVUeXBlPSJVcGRhdGVkIiBkYXRlSW5mb3JtYXRpb249InYxIj4yMDIzLTAyLTA3VDAxOjA1OjEwWjwvZGF0ZT4KICAgIDxkYXRlIGRhdGVUeXBlPSJTdWJtaXR0ZWQiIGRhdGVJbmZvcm1hdGlvbj0idjIiPjIwMjMtMDItMTBUMTk6MDE6NDlaPC9kYXRlPgogICAgPGRhdGUgZGF0ZVR5cGU9IlVwZGF0ZWQiIGRhdGVJbmZvcm1hdGlvbj0idjIiPjIwMjMtMDItMTRUMDE6MDA6MjVaPC9kYXRlPgogICAgPGRhdGUgZGF0ZVR5cGU9IlN1Ym1pdHRlZCIgZGF0ZUluZm9ybWF0aW9uPSJ2MyI+MjAyMy0wMy0xNFQxODo0OToyNlo8L2RhdGU+CiAgICA8ZGF0ZSBkYXRlVHlwZT0iVXBkYXRlZCIgZGF0ZUluZm9ybWF0aW9uPSJ2MyI+MjAyMy0wMy0xNlQwMDowMDo1OVo8L2RhdGU+CiAgICA8ZGF0ZSBkYXRlVHlwZT0iU3VibWl0dGVkIiBkYXRlSW5mb3JtYXRpb249InY0Ij4yMDIzLTA4LTI5VDE0OjU1OjM3WjwvZGF0ZT4KICAgIDxkYXRlIGRhdGVUeXBlPSJVcGRhdGVkIiBkYXRlSW5mb3JtYXRpb249InY0Ij4yMDIzLTA4LTMwVDAwOjMwOjU1WjwvZGF0ZT4KICAgIDxkYXRlIGRhdGVUeXBlPSJTdWJtaXR0ZWQiIGRhdGVJbmZvcm1hdGlvbj0idjUiPjIwMjMtMTEtMTFUMjM6MDU6NDRaPC9kYXRlPgogICAgPGRhdGUgZGF0ZVR5cGU9IlVwZGF0ZWQiIGRhdGVJbmZvcm1hdGlvbj0idjUiPjIwMjMtMTEtMTRUMDE6MTg6MjRaPC9kYXRlPgogICAgPGRhdGUgZGF0ZVR5cGU9IlN1Ym1pdHRlZCIgZGF0ZUluZm9ybWF0aW9uPSJ2NiI+MjAyNC0wMi0xN1QwMjowNTozMlo8L2RhdGU+CiAgICA8ZGF0ZSBkYXRlVHlwZT0iVXBkYXRlZCIgZGF0ZUluZm9ybWF0aW9uPSJ2NiI+MjAyNC0wMi0yMFQwMToxMDoyNVo8L2RhdGU+CiAgICA8ZGF0ZSBkYXRlVHlwZT0iU3VibWl0dGVkIiBkYXRlSW5mb3JtYXRpb249InY3Ij4yMDI0LTExLTA0VDE5OjUxOjUzWjwvZGF0ZT4KICAgIDxkYXRlIGRhdGVUeXBlPSJVcGRhdGVkIiBkYXRlSW5mb3JtYXRpb249InY3Ij4yMDI0LTExLTA2VDAxOjA3OjA3WjwvZGF0ZT4KICAgIDxkYXRlIGRhdGVUeXBlPSJBdmFpbGFibGUiIGRhdGVJbmZvcm1hdGlvbj0idjEiPjIwMjMtMDI8L2RhdGU+CiAgPC9kYXRlcz4KICA8cmVzb3VyY2VUeXBlIHJlc291cmNlVHlwZUdlbmVyYWw9IlRleHQiPkFydGljbGU8L3Jlc291cmNlVHlwZT4KICA8cmVsYXRlZElkZW50aWZpZXJzPgogICAgPHJlbGF0ZWRJZGVudGlmaWVyIHJlbGF0ZWRJZGVudGlmaWVyVHlwZT0iRE9JIiByZWxhdGlvblR5cGU9IklzVmVyc2lvbk9mIj4xMC4xMDczL3BuYXMuMjQwNTQ2MDEyMTwvcmVsYXRlZElkZW50aWZpZXI+CiAgPC9yZWxhdGVkSWRlbnRpZmllcnM+CiAgPHZlcnNpb24+NzwvdmVyc2lvbj4KICA8cmlnaHRzTGlzdD4KICAgIDxyaWdodHMgcmlnaHRzVVJJPSJodHRwOi8vY3JlYXRpdmVjb21tb25zLm9yZy9saWNlbnNlcy9ieS1uYy1uZC80LjAvIiByaWdodHNJZGVudGlmaWVyU2NoZW1lPSJTUERYIiByaWdodHNJZGVudGlmaWVyPSJDQy1CWS1OQy1ORC00LjAiPkNyZWF0aXZlIENvbW1vbnMgQXR0cmlidXRpb24gTm9uIENvbW1lcmNpYWwgTm8gRGVyaXZhdGl2ZXMgNC4wIEludGVybmF0aW9uYWw8L3JpZ2h0cz4KICA8L3JpZ2h0c0xpc3Q+CiAgPGRlc2NyaXB0aW9ucz4KICAgIDxkZXNjcmlwdGlvbiBkZXNjcmlwdGlvblR5cGU9IkFic3RyYWN0Ij5FbGV2ZW4gTGFyZ2UgTGFuZ3VhZ2UgTW9kZWxzIChMTE1zKSB3ZXJlIGFzc2Vzc2VkIHVzaW5nIGEgY3VzdG9tLW1hZGUgYmF0dGVyeSBvZiBmYWxzZS1iZWxpZWYgdGFza3MsIGNvbnNpZGVyZWQgYSBnb2xkIHN0YW5kYXJkIGluIHRlc3RpbmcgVGhlb3J5IG9mIE1pbmQgKFRvTSkgaW4gaHVtYW5zLiBUaGUgYmF0dGVyeSBpbmNsdWRlZCA2NDAgcHJvbXB0cyBzcHJlYWQgYWNyb3NzIDQwIGRpdmVyc2UgdGFza3MsIGVhY2ggb25lIGluY2x1ZGluZyBhIGZhbHNlLWJlbGllZiBzY2VuYXJpbywgdGhyZWUgY2xvc2VseSBtYXRjaGVkIHRydWUtYmVsaWVmIGNvbnRyb2wgc2NlbmFyaW9zLCBhbmQgdGhlIHJldmVyc2VkIHZlcnNpb25zIG9mIGFsbCBmb3VyLiBUbyBzb2x2ZSBhIHNpbmdsZSB0YXNrLCBhIG1vZGVsIG5lZWRlZCB0byBjb3JyZWN0bHkgYW5zd2VyIDE2IHByb21wdHMgYWNyb3NzIGFsbCBlaWdodCBzY2VuYXJpb3MuIFNtYWxsZXIgYW5kIG9sZGVyIG1vZGVscyBzb2x2ZWQgbm8gdGFza3M7IEdQVC0zLWRhdmluY2ktMDAzIChmcm9tIE5vdmVtYmVyIDIwMjIpIGFuZCBDaGF0R1BULTMuNS10dXJibyAoZnJvbSBNYXJjaCAyMDIzKSBzb2x2ZWQgMjAlIG9mIHRoZSB0YXNrczsgQ2hhdEdQVC00IChmcm9tIEp1bmUgMjAyMykgc29sdmVkIDc1JSBvZiB0aGUgdGFza3MsIG1hdGNoaW5nIHRoZSBwZXJmb3JtYW5jZSBvZiBzaXgteWVhci1vbGQgY2hpbGRyZW4gb2JzZXJ2ZWQgaW4gcGFzdCBzdHVkaWVzLiBXZSBleHBsb3JlIHRoZSBwb3RlbnRpYWwgaW50ZXJwcmV0YXRpb24gb2YgdGhlc2UgZmluZGluZ3MsIGluY2x1ZGluZyB0aGUgaW50cmlndWluZyBwb3NzaWJpbGl0eSB0aGF0IFRvTSwgcHJldmlvdXNseSBjb25zaWRlcmVkIGV4Y2x1c2l2ZSB0byBodW1hbnMsIG1heSBoYXZlIHNwb250YW5lb3VzbHkgZW1lcmdlZCBhcyBhIGJ5cHJvZHVjdCBvZiBMTE1zJyBpbXByb3ZpbmcgbGFuZ3VhZ2Ugc2tpbGxzLjwvZGVzY3JpcHRpb24+CiAgICA8ZGVzY3JpcHRpb24gZGVzY3JpcHRpb25UeXBlPSJPdGhlciI+VFJZIFJVTk5JTkcgVG9NIEVYUEVSSU1FTlRTIE9OIFlPVVIgT1dOOiBUaGUgY29kZSBhbmQgdGFza3MgdXNlZCBpbiB0aGlzIHN0dWR5IGFyZSBhdmFpbGFibGUgYXQgQ29sYWIgKGh0dHBzOi8vY29sYWIucmVzZWFyY2guZ29vZ2xlLmNvbS9kcml2ZS8xWlJ0bXc4N0NkQTR4cDI0RE5TX0lrX3VBMnlwYVJub1UpLiBEb24ndCB3b3JyeSBpZiB5b3UgYXJlIG5vdCBhbiBleHBlcnQgY29kZXIsIHlvdSBzaG91bGQgYmUgYWJsZSB0byBydW4gdGhpcyBjb2RlIHdpdGggbm8tdG8tbWluaW11bSBQeXRob24gc2tpbGxzLiBPciBjb3B5LXBhc3RlIHRoZSB0YXNrcyB0byBDaGF0R1BUJ3Mgd2ViIGludGVyZmFjZS4gUHJvY2VlZGluZ3Mgb2YgdGhlIE5hdGlvbmFsIEFjYWRlbXkgb2YgU2NpZW5jZXMgKFBOQVMpIDIwMjQ8L2Rlc2NyaXB0aW9uPgogIDwvZGVzY3JpcHRpb25zPgo8L3Jlc291cmNlPg==","url":"https://arxiv.org/abs/2302.02083","contentUrl":null,"metadataVersion":6,"schemaVersion":"http://datacite.org/schema/kernel-4","source":"mds","isActive":true,"state":"findable","reason":null,"viewCount":0,"viewsOverTime":[],"downloadCount":0,"downloadsOverTime":[],"referenceCount":0,"citationCount":0,"citationsOverTime":[],"partCount":0,"partOfCount":0,"versionCount":0,"versionOfCount":1,"created":"2023-02-07T02:16:06.000Z","registered":"2023-02-07T02:16:07.000Z","published":"2023","updated":"2024-11-06T03:17:06.000Z"},"relationships":{"client":{"data":{"id":"arxiv.content","type":"clients"}},"provider":{"data":{"id":"arxiv","type":"providers"}},"media":{"data":{"id":"10.48550/arxiv.2302.02083","type":"media"}},"references":{"data":[]},"citations":{"data":[]},"parts":{"data":[]},"partOf":{"data":[]},"versions":{"data":[]},"versionOf":{"data":[{"id":"10.1073/pnas.2405460121","type":"dois"}]}}}}