{"data":{"id":"10.48550/arxiv.2110.14583","type":"dois","attributes":{"doi":"10.48550/arxiv.2110.14583","prefix":"10.48550","suffix":"arxiv.2110.14583","identifiers":[{"identifier":"2110.14583","identifierType":"arXiv"}],"alternateIdentifiers":[{"alternateIdentifierType":"arXiv","alternateIdentifier":"2110.14583"}],"creators":[{"name":"Lucibello, Carlo","nameType":"Personal","givenName":"Carlo","familyName":"Lucibello","affiliation":[],"nameIdentifiers":[]},{"name":"Pittorino, Fabrizio","nameType":"Personal","givenName":"Fabrizio","familyName":"Pittorino","affiliation":[],"nameIdentifiers":[]},{"name":"Perugini, Gabriele","nameType":"Personal","givenName":"Gabriele","familyName":"Perugini","affiliation":[],"nameIdentifiers":[]},{"name":"Zecchina, Riccardo","nameType":"Personal","givenName":"Riccardo","familyName":"Zecchina","affiliation":[],"nameIdentifiers":[]}],"titles":[{"title":"Deep learning via message passing algorithms based on belief propagation"}],"publisher":"arXiv","container":{},"publicationYear":2021,"subjects":[{"lang":"en","subject":"Machine Learning (cs.LG)","subjectScheme":"arXiv"},{"lang":"en","subject":"Disordered Systems and Neural Networks (cond-mat.dis-nn)","subjectScheme":"arXiv"},{"lang":"en","subject":"Machine Learning (stat.ML)","subjectScheme":"arXiv"},{"subject":"FOS: Computer and information sciences","subjectScheme":"Fields of Science and Technology (FOS)"},{"subject":"FOS: Computer and information sciences","schemeUri":"http://www.oecd.org/science/inno/38235147.pdf","subjectScheme":"Fields of Science and Technology (FOS)"},{"subject":"FOS: Physical sciences","subjectScheme":"Fields of Science and Technology (FOS)"},{"subject":"FOS: Physical sciences","schemeUri":"http://www.oecd.org/science/inno/38235147.pdf","subjectScheme":"Fields of Science and Technology (FOS)"}],"contributors":[],"dates":[{"date":"2021-10-27T16:52:26Z","dateType":"Submitted","dateInformation":"v1"},{"date":"2021-10-28T00:29:38Z","dateType":"Updated","dateInformation":"v1"},{"date":"2022-01-29T15:35:35Z","dateType":"Submitted","dateInformation":"v2"},{"date":"2022-02-01T01:15:59Z","dateType":"Updated","dateInformation":"v2"},{"date":"2022-03-15T10:26:16Z","dateType":"Submitted","dateInformation":"v3"},{"date":"2022-07-20T00:19:18Z","dateType":"Updated","dateInformation":"v3"},{"date":"2021-10","dateType":"Available","dateInformation":"v1"},{"date":"2021","dateType":"Issued"}],"language":null,"types":{"ris":"RPRT","bibtex":"article","citeproc":"article-journal","schemaOrg":"ScholarlyArticle","resourceType":"Article","resourceTypeGeneral":"Text"},"relatedIdentifiers":[{"relationType":"IsVersionOf","relatedIdentifier":"10.1088/2632-2153/ac7d3b","relatedIdentifierType":"DOI"}],"relatedItems":[],"sizes":[],"formats":[],"version":"3","rightsList":[{"rights":"arXiv.org perpetual, non-exclusive license","rightsUri":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/"}],"descriptions":[{"description":"Message-passing algorithms based on the Belief Propagation (BP) equations constitute a well-known distributed computational scheme. It is exact on tree-like graphical models and has also proven to be effective in many problems defined on graphs with loops (from inference to optimization, from signal processing to clustering). The BP-based scheme is fundamentally different from stochastic gradient descent (SGD), on which the current success of deep networks is based. In this paper, we present and adapt to mini-batch training on GPUs a family of BP-based message-passing algorithms with a reinforcement field that biases distributions towards locally entropic solutions. These algorithms are capable of training multi-layer neural networks with discrete weights and activations with performance comparable to SGD-inspired heuristics (BinaryNet) and are naturally well-adapted to continual learning. Furthermore, using these algorithms to estimate the marginals of the weights allows us to make approximate Bayesian predictions that have higher accuracy than point-wise solutions.","descriptionType":"Abstract"}],"geoLocations":[],"fundingReferences":[],"xml":"PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0idXRmLTgiPz4KPHJlc291cmNlIHhtbG5zPSJodHRwOi8vZGF0YWNpdGUub3JnL3NjaGVtYS9rZXJuZWwtNCIgeG1sbnM6eHNpPSJodHRwOi8vd3d3LnczLm9yZy8yMDAxL1hNTFNjaGVtYS1pbnN0YW5jZSIgeHNpOnNjaGVtYUxvY2F0aW9uPSJodHRwOi8vZGF0YWNpdGUub3JnL3NjaGVtYS9rZXJuZWwtNCBodHRwOi8vc2NoZW1hLmRhdGFjaXRlLm9yZy9tZXRhL2tlcm5lbC00LjMvbWV0YWRhdGEueHNkIj4KICA8aWRlbnRpZmllciBpZGVudGlmaWVyVHlwZT0iRE9JIj4xMC40ODU1MC9BUlhJVi4yMTEwLjE0NTgzPC9pZGVudGlmaWVyPgogIDxhbHRlcm5hdGVJZGVudGlmaWVycz4KICAgIDxhbHRlcm5hdGVJZGVudGlmaWVyIGFsdGVybmF0ZUlkZW50aWZpZXJUeXBlPSJhclhpdiI+MjExMC4xNDU4MzwvYWx0ZXJuYXRlSWRlbnRpZmllcj4KICA8L2FsdGVybmF0ZUlkZW50aWZpZXJzPgogIDxjcmVhdG9ycz4KICAgIDxjcmVhdG9yPgogICAgICA8Y3JlYXRvck5hbWUgbmFtZVR5cGU9IlBlcnNvbmFsIj5MdWNpYmVsbG8sIENhcmxvPC9jcmVhdG9yTmFtZT4KICAgICAgPGdpdmVuTmFtZT5DYXJsbzwvZ2l2ZW5OYW1lPgogICAgICA8ZmFtaWx5TmFtZT5MdWNpYmVsbG88L2ZhbWlseU5hbWU+CiAgICA8L2NyZWF0b3I+CiAgICA8Y3JlYXRvcj4KICAgICAgPGNyZWF0b3JOYW1lIG5hbWVUeXBlPSJQZXJzb25hbCI+UGl0dG9yaW5vLCBGYWJyaXppbzwvY3JlYXRvck5hbWU+CiAgICAgIDxnaXZlbk5hbWU+RmFicml6aW88L2dpdmVuTmFtZT4KICAgICAgPGZhbWlseU5hbWU+UGl0dG9yaW5vPC9mYW1pbHlOYW1lPgogICAgPC9jcmVhdG9yPgogICAgPGNyZWF0b3I+CiAgICAgIDxjcmVhdG9yTmFtZSBuYW1lVHlwZT0iUGVyc29uYWwiPlBlcnVnaW5pLCBHYWJyaWVsZTwvY3JlYXRvck5hbWU+CiAgICAgIDxnaXZlbk5hbWU+R2FicmllbGU8L2dpdmVuTmFtZT4KICAgICAgPGZhbWlseU5hbWU+UGVydWdpbmk8L2ZhbWlseU5hbWU+CiAgICA8L2NyZWF0b3I+CiAgICA8Y3JlYXRvcj4KICAgICAgPGNyZWF0b3JOYW1lIG5hbWVUeXBlPSJQZXJzb25hbCI+WmVjY2hpbmEsIFJpY2NhcmRvPC9jcmVhdG9yTmFtZT4KICAgICAgPGdpdmVuTmFtZT5SaWNjYXJkbzwvZ2l2ZW5OYW1lPgogICAgICA8ZmFtaWx5TmFtZT5aZWNjaGluYTwvZmFtaWx5TmFtZT4KICAgIDwvY3JlYXRvcj4KICA8L2NyZWF0b3JzPgogIDx0aXRsZXM+CiAgICA8dGl0bGU+RGVlcCBsZWFybmluZyB2aWEgbWVzc2FnZSBwYXNzaW5nIGFsZ29yaXRobXMgYmFzZWQgb24gYmVsaWVmIHByb3BhZ2F0aW9uPC90aXRsZT4KICA8L3RpdGxlcz4KICA8cHVibGlzaGVyPmFyWGl2PC9wdWJsaXNoZXI+CiAgPHB1YmxpY2F0aW9uWWVhcj4yMDIxPC9wdWJsaWNhdGlvblllYXI+CiAgPHN1YmplY3RzPgogICAgPHN1YmplY3QgeG1sOmxhbmc9ImVuIiBzdWJqZWN0U2NoZW1lPSJhclhpdiI+TWFjaGluZSBMZWFybmluZyAoY3MuTEcpPC9zdWJqZWN0PgogICAgPHN1YmplY3QgeG1sOmxhbmc9ImVuIiBzdWJqZWN0U2NoZW1lPSJhclhpdiI+RGlzb3JkZXJlZCBTeXN0ZW1zIGFuZCBOZXVyYWwgTmV0d29ya3MgKGNvbmQtbWF0LmRpcy1ubik8L3N1YmplY3Q+CiAgICA8c3ViamVjdCB4bWw6bGFuZz0iZW4iIHN1YmplY3RTY2hlbWU9ImFyWGl2Ij5NYWNoaW5lIExlYXJuaW5nIChzdGF0Lk1MKTwvc3ViamVjdD4KICAgIDxzdWJqZWN0IHN1YmplY3RTY2hlbWU9IkZpZWxkcyBvZiBTY2llbmNlIGFuZCBUZWNobm9sb2d5IChGT1MpIj5GT1M6IENvbXB1dGVyIGFuZCBpbmZvcm1hdGlvbiBzY2llbmNlczwvc3ViamVjdD4KICAgIDxzdWJqZWN0IHN1YmplY3RTY2hlbWU9IkZpZWxkcyBvZiBTY2llbmNlIGFuZCBUZWNobm9sb2d5IChGT1MpIj5GT1M6IFBoeXNpY2FsIHNjaWVuY2VzPC9zdWJqZWN0PgogIDwvc3ViamVjdHM+CiAgPGRhdGVzPgogICAgPGRhdGUgZGF0ZVR5cGU9IlN1Ym1pdHRlZCIgZGF0ZUluZm9ybWF0aW9uPSJ2MSI+MjAyMS0xMC0yN1QxNjo1MjoyNlo8L2RhdGU+CiAgICA8ZGF0ZSBkYXRlVHlwZT0iVXBkYXRlZCIgZGF0ZUluZm9ybWF0aW9uPSJ2MSI+MjAyMS0xMC0yOFQwMDoyOTozOFo8L2RhdGU+CiAgICA8ZGF0ZSBkYXRlVHlwZT0iU3VibWl0dGVkIiBkYXRlSW5mb3JtYXRpb249InYyIj4yMDIyLTAxLTI5VDE1OjM1OjM1WjwvZGF0ZT4KICAgIDxkYXRlIGRhdGVUeXBlPSJVcGRhdGVkIiBkYXRlSW5mb3JtYXRpb249InYyIj4yMDIyLTAyLTAxVDAxOjE1OjU5WjwvZGF0ZT4KICAgIDxkYXRlIGRhdGVUeXBlPSJTdWJtaXR0ZWQiIGRhdGVJbmZvcm1hdGlvbj0idjMiPjIwMjItMDMtMTVUMTA6MjY6MTZaPC9kYXRlPgogICAgPGRhdGUgZGF0ZVR5cGU9IlVwZGF0ZWQiIGRhdGVJbmZvcm1hdGlvbj0idjMiPjIwMjItMDctMjBUMDA6MTk6MThaPC9kYXRlPgogICAgPGRhdGUgZGF0ZVR5cGU9IkF2YWlsYWJsZSIgZGF0ZUluZm9ybWF0aW9uPSJ2MSI+MjAyMS0xMDwvZGF0ZT4KICA8L2RhdGVzPgogIDxyZXNvdXJjZVR5cGUgcmVzb3VyY2VUeXBlR2VuZXJhbD0iVGV4dCI+QXJ0aWNsZTwvcmVzb3VyY2VUeXBlPgogIDxyZWxhdGVkSWRlbnRpZmllcnM+CiAgICA8cmVsYXRlZElkZW50aWZpZXIgcmVsYXRlZElkZW50aWZpZXJUeXBlPSJET0kiIHJlbGF0aW9uVHlwZT0iSXNWZXJzaW9uT2YiPjEwLjEwODgvMjYzMi0yMTUzL2FjN2QzYjwvcmVsYXRlZElkZW50aWZpZXI+CiAgPC9yZWxhdGVkSWRlbnRpZmllcnM+CiAgPHZlcnNpb24+MzwvdmVyc2lvbj4KICA8cmlnaHRzTGlzdD4KICAgIDxyaWdodHMgcmlnaHRzVVJJPSJodHRwOi8vYXJ4aXYub3JnL2xpY2Vuc2VzL25vbmV4Y2x1c2l2ZS1kaXN0cmliLzEuMC8iPmFyWGl2Lm9yZyBwZXJwZXR1YWwsIG5vbi1leGNsdXNpdmUgbGljZW5zZTwvcmlnaHRzPgogIDwvcmlnaHRzTGlzdD4KICA8ZGVzY3JpcHRpb25zPgogICAgPGRlc2NyaXB0aW9uIGRlc2NyaXB0aW9uVHlwZT0iQWJzdHJhY3QiPk1lc3NhZ2UtcGFzc2luZyBhbGdvcml0aG1zIGJhc2VkIG9uIHRoZSBCZWxpZWYgUHJvcGFnYXRpb24gKEJQKSBlcXVhdGlvbnMgY29uc3RpdHV0ZSBhIHdlbGwta25vd24gZGlzdHJpYnV0ZWQgY29tcHV0YXRpb25hbCBzY2hlbWUuIEl0IGlzIGV4YWN0IG9uIHRyZWUtbGlrZSBncmFwaGljYWwgbW9kZWxzIGFuZCBoYXMgYWxzbyBwcm92ZW4gdG8gYmUgZWZmZWN0aXZlIGluIG1hbnkgcHJvYmxlbXMgZGVmaW5lZCBvbiBncmFwaHMgd2l0aCBsb29wcyAoZnJvbSBpbmZlcmVuY2UgdG8gb3B0aW1pemF0aW9uLCBmcm9tIHNpZ25hbCBwcm9jZXNzaW5nIHRvIGNsdXN0ZXJpbmcpLiBUaGUgQlAtYmFzZWQgc2NoZW1lIGlzIGZ1bmRhbWVudGFsbHkgZGlmZmVyZW50IGZyb20gc3RvY2hhc3RpYyBncmFkaWVudCBkZXNjZW50IChTR0QpLCBvbiB3aGljaCB0aGUgY3VycmVudCBzdWNjZXNzIG9mIGRlZXAgbmV0d29ya3MgaXMgYmFzZWQuIEluIHRoaXMgcGFwZXIsIHdlIHByZXNlbnQgYW5kIGFkYXB0IHRvIG1pbmktYmF0Y2ggdHJhaW5pbmcgb24gR1BVcyBhIGZhbWlseSBvZiBCUC1iYXNlZCBtZXNzYWdlLXBhc3NpbmcgYWxnb3JpdGhtcyB3aXRoIGEgcmVpbmZvcmNlbWVudCBmaWVsZCB0aGF0IGJpYXNlcyBkaXN0cmlidXRpb25zIHRvd2FyZHMgbG9jYWxseSBlbnRyb3BpYyBzb2x1dGlvbnMuIFRoZXNlIGFsZ29yaXRobXMgYXJlIGNhcGFibGUgb2YgdHJhaW5pbmcgbXVsdGktbGF5ZXIgbmV1cmFsIG5ldHdvcmtzIHdpdGggZGlzY3JldGUgd2VpZ2h0cyBhbmQgYWN0aXZhdGlvbnMgd2l0aCBwZXJmb3JtYW5jZSBjb21wYXJhYmxlIHRvIFNHRC1pbnNwaXJlZCBoZXVyaXN0aWNzIChCaW5hcnlOZXQpIGFuZCBhcmUgbmF0dXJhbGx5IHdlbGwtYWRhcHRlZCB0byBjb250aW51YWwgbGVhcm5pbmcuIEZ1cnRoZXJtb3JlLCB1c2luZyB0aGVzZSBhbGdvcml0aG1zIHRvIGVzdGltYXRlIHRoZSBtYXJnaW5hbHMgb2YgdGhlIHdlaWdodHMgYWxsb3dzIHVzIHRvIG1ha2UgYXBwcm94aW1hdGUgQmF5ZXNpYW4gcHJlZGljdGlvbnMgdGhhdCBoYXZlIGhpZ2hlciBhY2N1cmFjeSB0aGFuIHBvaW50LXdpc2Ugc29sdXRpb25zLjwvZGVzY3JpcHRpb24+CiAgPC9kZXNjcmlwdGlvbnM+CjwvcmVzb3VyY2U+","url":"https://arxiv.org/abs/2110.14583","contentUrl":null,"metadataVersion":2,"schemaVersion":"http://datacite.org/schema/kernel-4","source":"mds","isActive":true,"state":"findable","reason":null,"viewCount":0,"viewsOverTime":[],"downloadCount":0,"downloadsOverTime":[],"referenceCount":0,"citationCount":0,"citationsOverTime":[],"partCount":0,"partOfCount":0,"versionCount":0,"versionOfCount":1,"created":"2022-02-21T17:04:02.000Z","registered":"2022-02-21T17:04:03.000Z","published":"2021","updated":"2022-07-20T01:18:08.000Z"},"relationships":{"client":{"data":{"id":"arxiv.content","type":"clients"}},"provider":{"data":{"id":"arxiv","type":"providers"}},"media":{"data":{"id":"10.48550/arxiv.2110.14583","type":"media"}},"references":{"data":[]},"citations":{"data":[]},"parts":{"data":[]},"partOf":{"data":[]},"versions":{"data":[]},"versionOf":{"data":[{"id":"10.1088/2632-2153/ac7d3b","type":"dois"}]}}}}