{"data":{"id":"10.48550/arxiv.1908.11502","type":"dois","attributes":{"doi":"10.48550/arxiv.1908.11502","prefix":"10.48550","suffix":"arxiv.1908.11502","identifiers":[{"identifier":"1908.11502","identifierType":"arXiv"}],"alternateIdentifiers":[{"alternateIdentifierType":"arXiv","alternateIdentifier":"1908.11502"}],"creators":[{"name":"Monakhova, Kristina","nameType":"Personal","givenName":"Kristina","familyName":"Monakhova","affiliation":[],"nameIdentifiers":[]},{"name":"Yurtsever, Joshua","nameType":"Personal","givenName":"Joshua","familyName":"Yurtsever","affiliation":[],"nameIdentifiers":[]},{"name":"Kuo, Grace","nameType":"Personal","givenName":"Grace","familyName":"Kuo","affiliation":[],"nameIdentifiers":[]},{"name":"Antipa, Nick","nameType":"Personal","givenName":"Nick","familyName":"Antipa","affiliation":[],"nameIdentifiers":[]},{"name":"Yanny, Kyrollos","nameType":"Personal","givenName":"Kyrollos","familyName":"Yanny","affiliation":[],"nameIdentifiers":[]},{"name":"Waller, Laura","nameType":"Personal","givenName":"Laura","familyName":"Waller","affiliation":[],"nameIdentifiers":[]}],"titles":[{"title":"Learned reconstructions for practical mask-based lensless imaging"}],"publisher":"arXiv","container":{},"publicationYear":2019,"subjects":[{"lang":"en","subject":"Image and Video Processing (eess.IV)","subjectScheme":"arXiv"},{"lang":"en","subject":"Computer Vision and Pattern Recognition (cs.CV)","subjectScheme":"arXiv"},{"subject":"FOS: Electrical engineering, electronic engineering, information engineering","subjectScheme":"Fields of Science and Technology (FOS)"},{"subject":"FOS: Electrical engineering, electronic engineering, information engineering","schemeUri":"http://www.oecd.org/science/inno/38235147.pdf","subjectScheme":"Fields of Science and Technology (FOS)"},{"subject":"FOS: Computer and information sciences","subjectScheme":"Fields of Science and Technology (FOS)"},{"subject":"FOS: Computer and information sciences","schemeUri":"http://www.oecd.org/science/inno/38235147.pdf","subjectScheme":"Fields of Science and Technology (FOS)"}],"contributors":[],"dates":[{"date":"2019-08-30T01:45:05Z","dateType":"Submitted","dateInformation":"v1"},{"date":"2019-09-27T00:05:09Z","dateType":"Updated","dateInformation":"v1"},{"date":"2019-08","dateType":"Available","dateInformation":"v1"},{"date":"2019","dateType":"Issued"}],"language":null,"types":{"ris":"RPRT","bibtex":"article","citeproc":"article-journal","schemaOrg":"ScholarlyArticle","resourceType":"Article","resourceTypeGeneral":"Text"},"relatedIdentifiers":[{"relationType":"IsVersionOf","relatedIdentifier":"10.1364/oe.27.028075","relatedIdentifierType":"DOI"}],"relatedItems":[],"sizes":[],"formats":[],"version":"1","rightsList":[{"rights":"arXiv.org perpetual, non-exclusive license","rightsUri":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/"}],"descriptions":[{"description":"Mask-based lensless imagers are smaller and lighter than traditional lensed cameras. In these imagers, the sensor does not directly record an image of the scene; rather, a computational algorithm reconstructs it. Typically, mask-based lensless imagers use a model-based reconstruction approach that suffers from long compute times and a heavy reliance on both system calibration and heuristically chosen denoisers. In this work, we address these limitations using a bounded-compute, trainable neural network to reconstruct the image. We leverage our knowledge of the physical system by unrolling a traditional model-based optimization algorithm, whose parameters we optimize using experimentally gathered ground-truth data. Optionally, images produced by the unrolled network are then fed into a jointly-trained denoiser. As compared to traditional methods, our architecture achieves better perceptual image quality and runs 20x faster, enabling interactive previewing of the scene. We explore a spectrum between model-based and deep learning methods, showing the benefits of using an intermediate approach. Finally, we test our network on images taken in the wild with a prototype mask-based camera, demonstrating that our network generalizes to natural images.","descriptionType":"Abstract"}],"geoLocations":[],"fundingReferences":[],"xml":"PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0idXRmLTgiPz4KPHJlc291cmNlIHhtbG5zPSJodHRwOi8vZGF0YWNpdGUub3JnL3NjaGVtYS9rZXJuZWwtNCIgeG1sbnM6eHNpPSJodHRwOi8vd3d3LnczLm9yZy8yMDAxL1hNTFNjaGVtYS1pbnN0YW5jZSIgeHNpOnNjaGVtYUxvY2F0aW9uPSJodHRwOi8vZGF0YWNpdGUub3JnL3NjaGVtYS9rZXJuZWwtNCBodHRwOi8vc2NoZW1hLmRhdGFjaXRlLm9yZy9tZXRhL2tlcm5lbC00LjMvbWV0YWRhdGEueHNkIj4KICA8aWRlbnRpZmllciBpZGVudGlmaWVyVHlwZT0iRE9JIj4xMC40ODU1MC9BUlhJVi4xOTA4LjExNTAyPC9pZGVudGlmaWVyPgogIDxhbHRlcm5hdGVJZGVudGlmaWVycz4KICAgIDxhbHRlcm5hdGVJZGVudGlmaWVyIGFsdGVybmF0ZUlkZW50aWZpZXJUeXBlPSJhclhpdiI+MTkwOC4xMTUwMjwvYWx0ZXJuYXRlSWRlbnRpZmllcj4KICA8L2FsdGVybmF0ZUlkZW50aWZpZXJzPgogIDxjcmVhdG9ycz4KICAgIDxjcmVhdG9yPgogICAgICA8Y3JlYXRvck5hbWUgbmFtZVR5cGU9IlBlcnNvbmFsIj5Nb25ha2hvdmEsIEtyaXN0aW5hPC9jcmVhdG9yTmFtZT4KICAgICAgPGdpdmVuTmFtZT5LcmlzdGluYTwvZ2l2ZW5OYW1lPgogICAgICA8ZmFtaWx5TmFtZT5Nb25ha2hvdmE8L2ZhbWlseU5hbWU+CiAgICA8L2NyZWF0b3I+CiAgICA8Y3JlYXRvcj4KICAgICAgPGNyZWF0b3JOYW1lIG5hbWVUeXBlPSJQZXJzb25hbCI+WXVydHNldmVyLCBKb3NodWE8L2NyZWF0b3JOYW1lPgogICAgICA8Z2l2ZW5OYW1lPkpvc2h1YTwvZ2l2ZW5OYW1lPgogICAgICA8ZmFtaWx5TmFtZT5ZdXJ0c2V2ZXI8L2ZhbWlseU5hbWU+CiAgICA8L2NyZWF0b3I+CiAgICA8Y3JlYXRvcj4KICAgICAgPGNyZWF0b3JOYW1lIG5hbWVUeXBlPSJQZXJzb25hbCI+S3VvLCBHcmFjZTwvY3JlYXRvck5hbWU+CiAgICAgIDxnaXZlbk5hbWU+R3JhY2U8L2dpdmVuTmFtZT4KICAgICAgPGZhbWlseU5hbWU+S3VvPC9mYW1pbHlOYW1lPgogICAgPC9jcmVhdG9yPgogICAgPGNyZWF0b3I+CiAgICAgIDxjcmVhdG9yTmFtZSBuYW1lVHlwZT0iUGVyc29uYWwiPkFudGlwYSwgTmljazwvY3JlYXRvck5hbWU+CiAgICAgIDxnaXZlbk5hbWU+TmljazwvZ2l2ZW5OYW1lPgogICAgICA8ZmFtaWx5TmFtZT5BbnRpcGE8L2ZhbWlseU5hbWU+CiAgICA8L2NyZWF0b3I+CiAgICA8Y3JlYXRvcj4KICAgICAgPGNyZWF0b3JOYW1lIG5hbWVUeXBlPSJQZXJzb25hbCI+WWFubnksIEt5cm9sbG9zPC9jcmVhdG9yTmFtZT4KICAgICAgPGdpdmVuTmFtZT5LeXJvbGxvczwvZ2l2ZW5OYW1lPgogICAgICA8ZmFtaWx5TmFtZT5ZYW5ueTwvZmFtaWx5TmFtZT4KICAgIDwvY3JlYXRvcj4KICAgIDxjcmVhdG9yPgogICAgICA8Y3JlYXRvck5hbWUgbmFtZVR5cGU9IlBlcnNvbmFsIj5XYWxsZXIsIExhdXJhPC9jcmVhdG9yTmFtZT4KICAgICAgPGdpdmVuTmFtZT5MYXVyYTwvZ2l2ZW5OYW1lPgogICAgICA8ZmFtaWx5TmFtZT5XYWxsZXI8L2ZhbWlseU5hbWU+CiAgICA8L2NyZWF0b3I+CiAgPC9jcmVhdG9ycz4KICA8dGl0bGVzPgogICAgPHRpdGxlPkxlYXJuZWQgcmVjb25zdHJ1Y3Rpb25zIGZvciBwcmFjdGljYWwgbWFzay1iYXNlZCBsZW5zbGVzcyBpbWFnaW5nPC90aXRsZT4KICA8L3RpdGxlcz4KICA8cHVibGlzaGVyPmFyWGl2PC9wdWJsaXNoZXI+CiAgPHB1YmxpY2F0aW9uWWVhcj4yMDE5PC9wdWJsaWNhdGlvblllYXI+CiAgPHN1YmplY3RzPgogICAgPHN1YmplY3QgeG1sOmxhbmc9ImVuIiBzdWJqZWN0U2NoZW1lPSJhclhpdiI+SW1hZ2UgYW5kIFZpZGVvIFByb2Nlc3NpbmcgKGVlc3MuSVYpPC9zdWJqZWN0PgogICAgPHN1YmplY3QgeG1sOmxhbmc9ImVuIiBzdWJqZWN0U2NoZW1lPSJhclhpdiI+Q29tcHV0ZXIgVmlzaW9uIGFuZCBQYXR0ZXJuIFJlY29nbml0aW9uIChjcy5DVik8L3N1YmplY3Q+CiAgICA8c3ViamVjdCBzdWJqZWN0U2NoZW1lPSJGaWVsZHMgb2YgU2NpZW5jZSBhbmQgVGVjaG5vbG9neSAoRk9TKSI+Rk9TOiBFbGVjdHJpY2FsIGVuZ2luZWVyaW5nLCBlbGVjdHJvbmljIGVuZ2luZWVyaW5nLCBpbmZvcm1hdGlvbiBlbmdpbmVlcmluZzwvc3ViamVjdD4KICAgIDxzdWJqZWN0IHN1YmplY3RTY2hlbWU9IkZpZWxkcyBvZiBTY2llbmNlIGFuZCBUZWNobm9sb2d5IChGT1MpIj5GT1M6IENvbXB1dGVyIGFuZCBpbmZvcm1hdGlvbiBzY2llbmNlczwvc3ViamVjdD4KICA8L3N1YmplY3RzPgogIDxkYXRlcz4KICAgIDxkYXRlIGRhdGVUeXBlPSJTdWJtaXR0ZWQiIGRhdGVJbmZvcm1hdGlvbj0idjEiPjIwMTktMDgtMzBUMDE6NDU6MDVaPC9kYXRlPgogICAgPGRhdGUgZGF0ZVR5cGU9IlVwZGF0ZWQiIGRhdGVJbmZvcm1hdGlvbj0idjEiPjIwMTktMDktMjdUMDA6MDU6MDlaPC9kYXRlPgogICAgPGRhdGUgZGF0ZVR5cGU9IkF2YWlsYWJsZSIgZGF0ZUluZm9ybWF0aW9uPSJ2MSI+MjAxOS0wODwvZGF0ZT4KICA8L2RhdGVzPgogIDxyZXNvdXJjZVR5cGUgcmVzb3VyY2VUeXBlR2VuZXJhbD0iVGV4dCI+QXJ0aWNsZTwvcmVzb3VyY2VUeXBlPgogIDxyZWxhdGVkSWRlbnRpZmllcnM+CiAgICA8cmVsYXRlZElkZW50aWZpZXIgcmVsYXRlZElkZW50aWZpZXJUeXBlPSJET0kiIHJlbGF0aW9uVHlwZT0iSXNWZXJzaW9uT2YiPjEwLjEzNjQvT0UuMjcuMDI4MDc1PC9yZWxhdGVkSWRlbnRpZmllcj4KICA8L3JlbGF0ZWRJZGVudGlmaWVycz4KICA8dmVyc2lvbj4xPC92ZXJzaW9uPgogIDxyaWdodHNMaXN0PgogICAgPHJpZ2h0cyByaWdodHNVUkk9Imh0dHA6Ly9hcnhpdi5vcmcvbGljZW5zZXMvbm9uZXhjbHVzaXZlLWRpc3RyaWIvMS4wLyI+YXJYaXYub3JnIHBlcnBldHVhbCwgbm9uLWV4Y2x1c2l2ZSBsaWNlbnNlPC9yaWdodHM+CiAgPC9yaWdodHNMaXN0PgogIDxkZXNjcmlwdGlvbnM+CiAgICA8ZGVzY3JpcHRpb24gZGVzY3JpcHRpb25UeXBlPSJBYnN0cmFjdCI+TWFzay1iYXNlZCBsZW5zbGVzcyBpbWFnZXJzIGFyZSBzbWFsbGVyIGFuZCBsaWdodGVyIHRoYW4gdHJhZGl0aW9uYWwgbGVuc2VkIGNhbWVyYXMuIEluIHRoZXNlIGltYWdlcnMsIHRoZSBzZW5zb3IgZG9lcyBub3QgZGlyZWN0bHkgcmVjb3JkIGFuIGltYWdlIG9mIHRoZSBzY2VuZTsgcmF0aGVyLCBhIGNvbXB1dGF0aW9uYWwgYWxnb3JpdGhtIHJlY29uc3RydWN0cyBpdC4gVHlwaWNhbGx5LCBtYXNrLWJhc2VkIGxlbnNsZXNzIGltYWdlcnMgdXNlIGEgbW9kZWwtYmFzZWQgcmVjb25zdHJ1Y3Rpb24gYXBwcm9hY2ggdGhhdCBzdWZmZXJzIGZyb20gbG9uZyBjb21wdXRlIHRpbWVzIGFuZCBhIGhlYXZ5IHJlbGlhbmNlIG9uIGJvdGggc3lzdGVtIGNhbGlicmF0aW9uIGFuZCBoZXVyaXN0aWNhbGx5IGNob3NlbiBkZW5vaXNlcnMuIEluIHRoaXMgd29yaywgd2UgYWRkcmVzcyB0aGVzZSBsaW1pdGF0aW9ucyB1c2luZyBhIGJvdW5kZWQtY29tcHV0ZSwgdHJhaW5hYmxlIG5ldXJhbCBuZXR3b3JrIHRvIHJlY29uc3RydWN0IHRoZSBpbWFnZS4gV2UgbGV2ZXJhZ2Ugb3VyIGtub3dsZWRnZSBvZiB0aGUgcGh5c2ljYWwgc3lzdGVtIGJ5IHVucm9sbGluZyBhIHRyYWRpdGlvbmFsIG1vZGVsLWJhc2VkIG9wdGltaXphdGlvbiBhbGdvcml0aG0sIHdob3NlIHBhcmFtZXRlcnMgd2Ugb3B0aW1pemUgdXNpbmcgZXhwZXJpbWVudGFsbHkgZ2F0aGVyZWQgZ3JvdW5kLXRydXRoIGRhdGEuIE9wdGlvbmFsbHksIGltYWdlcyBwcm9kdWNlZCBieSB0aGUgdW5yb2xsZWQgbmV0d29yayBhcmUgdGhlbiBmZWQgaW50byBhIGpvaW50bHktdHJhaW5lZCBkZW5vaXNlci4gQXMgY29tcGFyZWQgdG8gdHJhZGl0aW9uYWwgbWV0aG9kcywgb3VyIGFyY2hpdGVjdHVyZSBhY2hpZXZlcyBiZXR0ZXIgcGVyY2VwdHVhbCBpbWFnZSBxdWFsaXR5IGFuZCBydW5zIDIweCBmYXN0ZXIsIGVuYWJsaW5nIGludGVyYWN0aXZlIHByZXZpZXdpbmcgb2YgdGhlIHNjZW5lLiBXZSBleHBsb3JlIGEgc3BlY3RydW0gYmV0d2VlbiBtb2RlbC1iYXNlZCBhbmQgZGVlcCBsZWFybmluZyBtZXRob2RzLCBzaG93aW5nIHRoZSBiZW5lZml0cyBvZiB1c2luZyBhbiBpbnRlcm1lZGlhdGUgYXBwcm9hY2guIEZpbmFsbHksIHdlIHRlc3Qgb3VyIG5ldHdvcmsgb24gaW1hZ2VzIHRha2VuIGluIHRoZSB3aWxkIHdpdGggYSBwcm90b3R5cGUgbWFzay1iYXNlZCBjYW1lcmEsIGRlbW9uc3RyYXRpbmcgdGhhdCBvdXIgbmV0d29yayBnZW5lcmFsaXplcyB0byBuYXR1cmFsIGltYWdlcy48L2Rlc2NyaXB0aW9uPgogIDwvZGVzY3JpcHRpb25zPgo8L3Jlc291cmNlPg==","url":"https://arxiv.org/abs/1908.11502","contentUrl":null,"metadataVersion":0,"schemaVersion":"http://datacite.org/schema/kernel-4","source":"mds","isActive":true,"state":"findable","reason":null,"viewCount":0,"viewsOverTime":[],"downloadCount":0,"downloadsOverTime":[],"referenceCount":0,"citationCount":0,"citationsOverTime":[],"partCount":0,"partOfCount":0,"versionCount":0,"versionOfCount":0,"created":"2022-02-27T07:22:45.000Z","registered":"2022-02-27T07:22:46.000Z","published":"2019","updated":"2022-02-27T07:22:46.000Z"},"relationships":{"client":{"data":{"id":"arxiv.content","type":"clients"}},"provider":{"data":{"id":"arxiv","type":"providers"}},"media":{"data":{"id":"10.48550/arxiv.1908.11502","type":"media"}},"references":{"data":[]},"citations":{"data":[]},"parts":{"data":[]},"partOf":{"data":[]},"versions":{"data":[]},"versionOf":{"data":[]}}}}