{"data":{"id":"10.48550/arxiv.2310.20494","type":"dois","attributes":{"doi":"10.48550/arxiv.2310.20494","prefix":"10.48550","suffix":"arxiv.2310.20494","identifiers":[{"identifier":"2310.20494","identifierType":"arXiv"}],"alternateIdentifiers":[{"alternateIdentifierType":"arXiv","alternateIdentifier":"2310.20494"}],"creators":[{"name":"Ma, Hui","nameType":"Personal","givenName":"Hui","familyName":"Ma","affiliation":[],"nameIdentifiers":[]},{"name":"Wang, Jian","nameType":"Personal","givenName":"Jian","familyName":"Wang","affiliation":[],"nameIdentifiers":[]},{"name":"Lin, Hongfei","nameType":"Personal","givenName":"Hongfei","familyName":"Lin","affiliation":[],"nameIdentifiers":[]},{"name":"Zhang, Bo","nameType":"Personal","givenName":"Bo","familyName":"Zhang","affiliation":[],"nameIdentifiers":[]},{"name":"Zhang, Yijia","nameType":"Personal","givenName":"Yijia","familyName":"Zhang","affiliation":[],"nameIdentifiers":[]},{"name":"Xu, Bo","nameType":"Personal","givenName":"Bo","familyName":"Xu","affiliation":[],"nameIdentifiers":[]}],"titles":[{"title":"A Transformer-Based Model With Self-Distillation for Multimodal Emotion Recognition in Conversations"}],"publisher":"arXiv","container":{},"publicationYear":2023,"subjects":[{"lang":"en","subject":"Artificial Intelligence (cs.AI)","subjectScheme":"arXiv"},{"lang":"en","subject":"Multimedia (cs.MM)","subjectScheme":"arXiv"},{"subject":"FOS: Computer and information sciences","subjectScheme":"Fields of Science and Technology (FOS)"},{"subject":"FOS: Computer and information sciences","schemeUri":"http://www.oecd.org/science/inno/38235147.pdf","subjectScheme":"Fields of Science and Technology (FOS)"}],"contributors":[],"dates":[{"date":"2023-10-31T14:33:30Z","dateType":"Submitted","dateInformation":"v1"},{"date":"2023-11-01T00:41:51Z","dateType":"Updated","dateInformation":"v1"},{"date":"2023-10","dateType":"Available","dateInformation":"v1"},{"date":"2023","dateType":"Issued"}],"language":null,"types":{"ris":"RPRT","bibtex":"article","citeproc":"article-journal","schemaOrg":"ScholarlyArticle","resourceType":"Article","resourceTypeGeneral":"Text"},"relatedIdentifiers":[{"relationType":"IsVersionOf","relatedIdentifier":"10.1109/tmm.2023.3271019","relatedIdentifierType":"DOI"}],"relatedItems":[],"sizes":[],"formats":[],"version":"1","rightsList":[{"rights":"arXiv.org perpetual, non-exclusive license","rightsUri":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/"}],"descriptions":[{"description":"Emotion recognition in conversations (ERC), the task of recognizing the emotion of each utterance in a conversation, is crucial for building empathetic machines. Existing studies focus mainly on capturing context- and speaker-sensitive dependencies on the textual modality but ignore the significance of multimodal information. Different from emotion recognition in textual conversations, capturing intra- and inter-modal interactions between utterances, learning weights between different modalities, and enhancing modal representations play important roles in multimodal ERC. In this paper, we propose a transformer-based model with self-distillation (SDT) for the task. The transformer-based model captures intra- and inter-modal interactions by utilizing intra- and inter-modal transformers, and learns weights between modalities dynamically by designing a hierarchical gated fusion strategy. Furthermore, to learn more expressive modal representations, we treat soft labels of the proposed model as extra training supervision. Specifically, we introduce self-distillation to transfer knowledge of hard and soft labels from the proposed model to each modality. Experiments on IEMOCAP and MELD datasets demonstrate that SDT outperforms previous state-of-the-art baselines.","descriptionType":"Abstract"},{"description":"13 pages, 10 figures. Accepted by IEEE Transactions on Multimedia (TMM)","descriptionType":"Other"}],"geoLocations":[],"fundingReferences":[],"xml":"PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0idXRmLTgiPz4KPHJlc291cmNlIHhtbG5zPSJodHRwOi8vZGF0YWNpdGUub3JnL3NjaGVtYS9rZXJuZWwtNCIgeG1sbnM6eHNpPSJodHRwOi8vd3d3LnczLm9yZy8yMDAxL1hNTFNjaGVtYS1pbnN0YW5jZSIgeHNpOnNjaGVtYUxvY2F0aW9uPSJodHRwOi8vZGF0YWNpdGUub3JnL3NjaGVtYS9rZXJuZWwtNCBodHRwOi8vc2NoZW1hLmRhdGFjaXRlLm9yZy9tZXRhL2tlcm5lbC00LjMvbWV0YWRhdGEueHNkIj4KICA8aWRlbnRpZmllciBpZGVudGlmaWVyVHlwZT0iRE9JIj4xMC40ODU1MC9BUlhJVi4yMzEwLjIwNDk0PC9pZGVudGlmaWVyPgogIDxhbHRlcm5hdGVJZGVudGlmaWVycz4KICAgIDxhbHRlcm5hdGVJZGVudGlmaWVyIGFsdGVybmF0ZUlkZW50aWZpZXJUeXBlPSJhclhpdiI+MjMxMC4yMDQ5NDwvYWx0ZXJuYXRlSWRlbnRpZmllcj4KICA8L2FsdGVybmF0ZUlkZW50aWZpZXJzPgogIDxjcmVhdG9ycz4KICAgIDxjcmVhdG9yPgogICAgICA8Y3JlYXRvck5hbWUgbmFtZVR5cGU9IlBlcnNvbmFsIj5NYSwgSHVpPC9jcmVhdG9yTmFtZT4KICAgICAgPGdpdmVuTmFtZT5IdWk8L2dpdmVuTmFtZT4KICAgICAgPGZhbWlseU5hbWU+TWE8L2ZhbWlseU5hbWU+CiAgICA8L2NyZWF0b3I+CiAgICA8Y3JlYXRvcj4KICAgICAgPGNyZWF0b3JOYW1lIG5hbWVUeXBlPSJQZXJzb25hbCI+V2FuZywgSmlhbjwvY3JlYXRvck5hbWU+CiAgICAgIDxnaXZlbk5hbWU+SmlhbjwvZ2l2ZW5OYW1lPgogICAgICA8ZmFtaWx5TmFtZT5XYW5nPC9mYW1pbHlOYW1lPgogICAgPC9jcmVhdG9yPgogICAgPGNyZWF0b3I+CiAgICAgIDxjcmVhdG9yTmFtZSBuYW1lVHlwZT0iUGVyc29uYWwiPkxpbiwgSG9uZ2ZlaTwvY3JlYXRvck5hbWU+CiAgICAgIDxnaXZlbk5hbWU+SG9uZ2ZlaTwvZ2l2ZW5OYW1lPgogICAgICA8ZmFtaWx5TmFtZT5MaW48L2ZhbWlseU5hbWU+CiAgICA8L2NyZWF0b3I+CiAgICA8Y3JlYXRvcj4KICAgICAgPGNyZWF0b3JOYW1lIG5hbWVUeXBlPSJQZXJzb25hbCI+WmhhbmcsIEJvPC9jcmVhdG9yTmFtZT4KICAgICAgPGdpdmVuTmFtZT5CbzwvZ2l2ZW5OYW1lPgogICAgICA8ZmFtaWx5TmFtZT5aaGFuZzwvZmFtaWx5TmFtZT4KICAgIDwvY3JlYXRvcj4KICAgIDxjcmVhdG9yPgogICAgICA8Y3JlYXRvck5hbWUgbmFtZVR5cGU9IlBlcnNvbmFsIj5aaGFuZywgWWlqaWE8L2NyZWF0b3JOYW1lPgogICAgICA8Z2l2ZW5OYW1lPllpamlhPC9naXZlbk5hbWU+CiAgICAgIDxmYW1pbHlOYW1lPlpoYW5nPC9mYW1pbHlOYW1lPgogICAgPC9jcmVhdG9yPgogICAgPGNyZWF0b3I+CiAgICAgIDxjcmVhdG9yTmFtZSBuYW1lVHlwZT0iUGVyc29uYWwiPlh1LCBCbzwvY3JlYXRvck5hbWU+CiAgICAgIDxnaXZlbk5hbWU+Qm88L2dpdmVuTmFtZT4KICAgICAgPGZhbWlseU5hbWU+WHU8L2ZhbWlseU5hbWU+CiAgICA8L2NyZWF0b3I+CiAgPC9jcmVhdG9ycz4KICA8dGl0bGVzPgogICAgPHRpdGxlPkEgVHJhbnNmb3JtZXItQmFzZWQgTW9kZWwgV2l0aCBTZWxmLURpc3RpbGxhdGlvbiBmb3IgTXVsdGltb2RhbCBFbW90aW9uIFJlY29nbml0aW9uIGluIENvbnZlcnNhdGlvbnM8L3RpdGxlPgogIDwvdGl0bGVzPgogIDxwdWJsaXNoZXI+YXJYaXY8L3B1Ymxpc2hlcj4KICA8cHVibGljYXRpb25ZZWFyPjIwMjM8L3B1YmxpY2F0aW9uWWVhcj4KICA8c3ViamVjdHM+CiAgICA8c3ViamVjdCB4bWw6bGFuZz0iZW4iIHN1YmplY3RTY2hlbWU9ImFyWGl2Ij5BcnRpZmljaWFsIEludGVsbGlnZW5jZSAoY3MuQUkpPC9zdWJqZWN0PgogICAgPHN1YmplY3QgeG1sOmxhbmc9ImVuIiBzdWJqZWN0U2NoZW1lPSJhclhpdiI+TXVsdGltZWRpYSAoY3MuTU0pPC9zdWJqZWN0PgogICAgPHN1YmplY3Qgc3ViamVjdFNjaGVtZT0iRmllbGRzIG9mIFNjaWVuY2UgYW5kIFRlY2hub2xvZ3kgKEZPUykiPkZPUzogQ29tcHV0ZXIgYW5kIGluZm9ybWF0aW9uIHNjaWVuY2VzPC9zdWJqZWN0PgogIDwvc3ViamVjdHM+CiAgPGRhdGVzPgogICAgPGRhdGUgZGF0ZVR5cGU9IlN1Ym1pdHRlZCIgZGF0ZUluZm9ybWF0aW9uPSJ2MSI+MjAyMy0xMC0zMVQxNDozMzozMFo8L2RhdGU+CiAgICA8ZGF0ZSBkYXRlVHlwZT0iVXBkYXRlZCIgZGF0ZUluZm9ybWF0aW9uPSJ2MSI+MjAyMy0xMS0wMVQwMDo0MTo1MVo8L2RhdGU+CiAgICA8ZGF0ZSBkYXRlVHlwZT0iQXZhaWxhYmxlIiBkYXRlSW5mb3JtYXRpb249InYxIj4yMDIzLTEwPC9kYXRlPgogIDwvZGF0ZXM+CiAgPHJlc291cmNlVHlwZSByZXNvdXJjZVR5cGVHZW5lcmFsPSJUZXh0Ij5BcnRpY2xlPC9yZXNvdXJjZVR5cGU+CiAgPHJlbGF0ZWRJZGVudGlmaWVycz4KICAgIDxyZWxhdGVkSWRlbnRpZmllciByZWxhdGVkSWRlbnRpZmllclR5cGU9IkRPSSIgcmVsYXRpb25UeXBlPSJJc1ZlcnNpb25PZiI+MTAuMTEwOS9UTU0uMjAyMy4zMjcxMDE5PC9yZWxhdGVkSWRlbnRpZmllcj4KICA8L3JlbGF0ZWRJZGVudGlmaWVycz4KICA8dmVyc2lvbj4xPC92ZXJzaW9uPgogIDxyaWdodHNMaXN0PgogICAgPHJpZ2h0cyByaWdodHNVUkk9Imh0dHA6Ly9hcnhpdi5vcmcvbGljZW5zZXMvbm9uZXhjbHVzaXZlLWRpc3RyaWIvMS4wLyI+YXJYaXYub3JnIHBlcnBldHVhbCwgbm9uLWV4Y2x1c2l2ZSBsaWNlbnNlPC9yaWdodHM+CiAgPC9yaWdodHNMaXN0PgogIDxkZXNjcmlwdGlvbnM+CiAgICA8ZGVzY3JpcHRpb24gZGVzY3JpcHRpb25UeXBlPSJBYnN0cmFjdCI+RW1vdGlvbiByZWNvZ25pdGlvbiBpbiBjb252ZXJzYXRpb25zIChFUkMpLCB0aGUgdGFzayBvZiByZWNvZ25pemluZyB0aGUgZW1vdGlvbiBvZiBlYWNoIHV0dGVyYW5jZSBpbiBhIGNvbnZlcnNhdGlvbiwgaXMgY3J1Y2lhbCBmb3IgYnVpbGRpbmcgZW1wYXRoZXRpYyBtYWNoaW5lcy4gRXhpc3Rpbmcgc3R1ZGllcyBmb2N1cyBtYWlubHkgb24gY2FwdHVyaW5nIGNvbnRleHQtIGFuZCBzcGVha2VyLXNlbnNpdGl2ZSBkZXBlbmRlbmNpZXMgb24gdGhlIHRleHR1YWwgbW9kYWxpdHkgYnV0IGlnbm9yZSB0aGUgc2lnbmlmaWNhbmNlIG9mIG11bHRpbW9kYWwgaW5mb3JtYXRpb24uIERpZmZlcmVudCBmcm9tIGVtb3Rpb24gcmVjb2duaXRpb24gaW4gdGV4dHVhbCBjb252ZXJzYXRpb25zLCBjYXB0dXJpbmcgaW50cmEtIGFuZCBpbnRlci1tb2RhbCBpbnRlcmFjdGlvbnMgYmV0d2VlbiB1dHRlcmFuY2VzLCBsZWFybmluZyB3ZWlnaHRzIGJldHdlZW4gZGlmZmVyZW50IG1vZGFsaXRpZXMsIGFuZCBlbmhhbmNpbmcgbW9kYWwgcmVwcmVzZW50YXRpb25zIHBsYXkgaW1wb3J0YW50IHJvbGVzIGluIG11bHRpbW9kYWwgRVJDLiBJbiB0aGlzIHBhcGVyLCB3ZSBwcm9wb3NlIGEgdHJhbnNmb3JtZXItYmFzZWQgbW9kZWwgd2l0aCBzZWxmLWRpc3RpbGxhdGlvbiAoU0RUKSBmb3IgdGhlIHRhc2suIFRoZSB0cmFuc2Zvcm1lci1iYXNlZCBtb2RlbCBjYXB0dXJlcyBpbnRyYS0gYW5kIGludGVyLW1vZGFsIGludGVyYWN0aW9ucyBieSB1dGlsaXppbmcgaW50cmEtIGFuZCBpbnRlci1tb2RhbCB0cmFuc2Zvcm1lcnMsIGFuZCBsZWFybnMgd2VpZ2h0cyBiZXR3ZWVuIG1vZGFsaXRpZXMgZHluYW1pY2FsbHkgYnkgZGVzaWduaW5nIGEgaGllcmFyY2hpY2FsIGdhdGVkIGZ1c2lvbiBzdHJhdGVneS4gRnVydGhlcm1vcmUsIHRvIGxlYXJuIG1vcmUgZXhwcmVzc2l2ZSBtb2RhbCByZXByZXNlbnRhdGlvbnMsIHdlIHRyZWF0IHNvZnQgbGFiZWxzIG9mIHRoZSBwcm9wb3NlZCBtb2RlbCBhcyBleHRyYSB0cmFpbmluZyBzdXBlcnZpc2lvbi4gU3BlY2lmaWNhbGx5LCB3ZSBpbnRyb2R1Y2Ugc2VsZi1kaXN0aWxsYXRpb24gdG8gdHJhbnNmZXIga25vd2xlZGdlIG9mIGhhcmQgYW5kIHNvZnQgbGFiZWxzIGZyb20gdGhlIHByb3Bvc2VkIG1vZGVsIHRvIGVhY2ggbW9kYWxpdHkuIEV4cGVyaW1lbnRzIG9uIElFTU9DQVAgYW5kIE1FTEQgZGF0YXNldHMgZGVtb25zdHJhdGUgdGhhdCBTRFQgb3V0cGVyZm9ybXMgcHJldmlvdXMgc3RhdGUtb2YtdGhlLWFydCBiYXNlbGluZXMuPC9kZXNjcmlwdGlvbj4KICAgIDxkZXNjcmlwdGlvbiBkZXNjcmlwdGlvblR5cGU9Ik90aGVyIj4xMyBwYWdlcywgMTAgZmlndXJlcy4gQWNjZXB0ZWQgYnkgSUVFRSBUcmFuc2FjdGlvbnMgb24gTXVsdGltZWRpYSAoVE1NKTwvZGVzY3JpcHRpb24+CiAgPC9kZXNjcmlwdGlvbnM+CjwvcmVzb3VyY2U+","url":"https://arxiv.org/abs/2310.20494","contentUrl":null,"metadataVersion":0,"schemaVersion":"http://datacite.org/schema/kernel-4","source":"mds","isActive":true,"state":"findable","reason":null,"viewCount":0,"viewsOverTime":[],"downloadCount":0,"downloadsOverTime":[],"referenceCount":0,"citationCount":0,"citationsOverTime":[],"partCount":0,"partOfCount":0,"versionCount":0,"versionOfCount":0,"created":"2023-11-01T02:05:02.000Z","registered":"2023-11-01T02:05:02.000Z","published":"2023","updated":"2023-11-01T02:05:02.000Z"},"relationships":{"client":{"data":{"id":"arxiv.content","type":"clients"}},"provider":{"data":{"id":"arxiv","type":"providers"}},"media":{"data":{"id":"10.48550/arxiv.2310.20494","type":"media"}},"references":{"data":[]},"citations":{"data":[]},"parts":{"data":[]},"partOf":{"data":[]},"versions":{"data":[]},"versionOf":{"data":[]}}}}