Combining Learning And Computational Imaging For 3d Inference

Acquiring 3D geometry of the scene is a key task in computer vision. Applications are numerous, from classical object reconstruction and scene understanding to the more recent visual SLAM and autonomous driving. Recent advances in computational imaging have enabled many new solutions to tackle the problem of 3D reconstruction. By modifying the camera's components, computational imaging optically encodes the scene, then decodes it with tailored algorithms. ☐ This dissertation focuses on exploring new computational imaging techniques, combined with recent advances in deep learning, to infer 3D geometry of the scene. In general, our approaches can be categorized into active and passive 3D sensing. ☐ For active illumination methods, we propose two solutions: first, we present a multi-flash (MF) system implemented on the mobile platform. Using the sequence of images captured by the MF system, we can extract the depth edges of the scene, and further estimate a depth map on a mobile device. Next, we show a portable immersive system that is capable of acquiring and displaying high fidelity 3D reconstructions using a set of RGB-D sensors. The system is based on structured light technique and is able to recover 3D geometry of the scene in real time. We have also developed a visualization system that allows users to dynamically visualize the event from new perspectives at arbitrary time instances in real time. ☐ For passive sensing methods, we focus on light field based depth estimation. For depth inference from a single light field, we present an algorithm that is tailored for barcode images. Our algorithm analyzes the statistics of raw light field images and conducts depth estimation with real time speed for fast refocusing and decoding. To mimic the human vision system, we investigate the dual light field input and propose a unified deep learning based framework to extract depth from both disparity cue and focus cue. To facilitate training, we have created a large dual focal stack database with ground truth disparity. While above solution focuses on fusing depth from focus and stereo, we also exploit combing depth from defocus and stereo, with an all-focus stereo pair and a defocused image of one of the stereo views as input. We have adopted the hourglass network architecture to extract depth from the image triplets. We have then studied and explored multiple neural network architectures to improve depth inference. We demonstrate that our deep learning based approaches preserve the strength of focus/defocus cue and disparity cue while effectively suppressing their weaknesses.

This book presents the thoroughly revised versions of lectures given by leading researchers during the Workshop on Advanced 3D Imaging for Safety and Security in conjunction with the International Conference on Computer Vision and Pattern Recognition CVPR 2005, held in San Diego, CA, USA in June 2005. It covers the current state of the art in 3D imaging for safety and security.

One of the grand challenges of artificial intelligence is to enable computers to interpret 3D scenes and objects from imagery. This book organizes and introduces major concepts in 3D scene and object representation and inference from still images, with a focus on recent efforts to fuse models of geometry and perspective with statistical machine learning. The book is organized into three sections: (1) Interpretation of Physical Space; (2) Recognition of 3D Objects; and (3) Integrated 3D Scene Interpretation. The first discusses representations of spatial layout and techniques to interpret physical scenes from images. The second section introduces representations for 3D object categories that account for the intrinsically 3D nature of objects and provide robustness to change in viewpoints. The third section discusses strategies to unite inference of scene geometry and object pose and identity into a coherent scene interpretation. Each section broadly surveys important ideas from cognitive science and artificial intelligence research, organizes and discusses key concepts and techniques from recent work in computer vision, and describes a few sample approaches in detail. Newcomers to computer vision will benefit from introductions to basic concepts, such as single-view geometry and image classification, while experts and novices alike may find inspiration from the book's organization and discussion of the most recent ideas in 3D scene understanding and 3D object recognition. Specific topics include: mathematics of perspective geometry; visual elements of the physical scene, structural 3D scene representations; techniques and features for image and region categorization; historical perspective, computational models, and datasets and machine learning techniques for 3D object recognition; inferences of geometrical attributes of objects, such as size and pose; and probabilistic and feature-passing approaches for contextual reasoning about 3D objects and scenes. Table of Contents: Background on 3D Scene Models / Single-view Geometry / Modeling the Physical Scene / Categorizing Images and Regions / Examples of 3D Scene Interpretation / Background on 3D Recognition / Modeling 3D Objects / Recognizing and Understanding 3D Objects / Examples of 2D 1/2 Layout Models / Reasoning about Objects and Scenes / Cascades of Classifiers / Conclusion and Future Directions

This textbook is designed for postgraduate studies in the field of 3D Computer Vision. It also provides a useful reference for industrial practitioners; for example, in the areas of 3D data capture, computer-aided geometric modelling and industrial quality assurance. This second edition is a significant upgrade of existing topics with novel findings. Additionally, it has new material covering consumer-grade RGB-D cameras, 3D morphable models, deep learning on 3D datasets, as well as new applications in the 3D digitization of cultural heritage and the 3D phenotyping of crops. Overall, the book covers three main areas: ● 3D imaging, including passive 3D imaging, active triangulation 3D imaging, active time-of-flight 3D imaging, consumer RGB-D cameras, and 3D data representation and visualisation; ● 3D shape analysis, including local descriptors, registration, matching, 3D morphable models, and deep learning on 3D datasets; and ● 3D applications, including 3D face recognition, cultural heritage and 3D phenotyping of plants. 3D computer vision is a rapidly advancing area in computer science. There are many real-world applications that demand high-performance 3D imaging and analysis and, as a result, many new techniques and commercial products have been developed. However, many challenges remain on how to analyse the captured data in a way that is sufficiently fast, robust and accurate for the application. Such challenges include metrology, semantic segmentation, classification and recognition. Thus, 3D imaging, analysis and their applications remain a highly-active research field that will continue to attract intensive attention from the research community with the ultimate goal of fully automating the 3D data capture, analysis and inference pipeline.

"A collection of innovative research on the methods and common techniques of image reconstruction as well as the accuracy of these methods. Featuring coverage on a wide range of topics such as ray casting, holographic techniques, and machine learning"--Page 4 of cover

This book presents high-quality research in the field of 3D imaging technology. The second edition of International Conference on 3D Imaging Technology (3DDIT-MSP&DL) continues the good traditions already established by the first 3DIT conference (IC3DIT2019) to provide a wide scientific forum for researchers, academia and practitioners to exchange newest ideas and recent achievements in all aspects of image processing and analysis, together with their contemporary applications. The conference proceedings are published in 2 volumes. The main topics of the papers comprise famous trends as: 3D image representation, 3D image technology, 3D images and graphics, and computing and 3D information technology. In these proceedings, special attention is paid at the 3D tensor image representation, the 3D content generation technologies, big data analysis, and also deep learning, artificial intelligence, the 3D image analysis and video understanding, the 3D virtual and augmented reality, and many related areas. The first volume contains papers in 3D image processing, transforms and technologies. The second volume is about computing and information technologies, computer images and graphics and related applications. The two volumes of the book cover a wide area of the aspects of the contemporary multidimensional imaging and the related future trends from data acquisition to real-world applications based on various techniques and theoretical approaches.

Computational imaging (CI) is a class of imaging systems that uses inverse algorithms to recover an unknown object from the physical measurement. Traditional inverse algorithms in CI obtain an estimate of the object by minimizing the Tikhonov functional, which requires explicit formulations of the forward operator of the physical system, as well as the prior knowledge about the class of objects being imaged. In recent years, machine learning architectures, and deep learning (DL) in particular, have attracted increasing attentions from CI researchers. Unlike traditional inverse algorithms in CI, DL approach learns both the forward operator and the objects’ prior implicitly from training examples. Therefore, it is especially attractive when the forward imaging model is uncertain (e.g. imaging through random scattering media), or the prior about the class of objects is difficult to be expressed analytically (e.g. natural images). In this thesis, the application of DL approaches in two different CI scenarios are investigated: imaging through a glass diffuser and quantitative phase retrieval (QPR), where an Imaging through Diffuser Network (IDiffNet) and a Phase Extraction Neural Network (PhENN) are experimentally demonstrated, respectively. This thesis also studies the influences of the two main factors that determine the performance of a trained neural network: network architecture (connectivity, network depth, etc) and training example quality (spatial frequency content in particular). Motivated by the analysis of the latter factor, two novel approaches, spectral pre-modulation approach and Learning Synthesis by DNN (LS-DNN) method, are successively proposed to improve the visual qualities of the network outputs. Finally, the LS-DNN enhanced PhENN is applied to a phase microscope to recover the phase of a red blood cell (RBC) sample. Furthermore, through simulation of the learned weak object transfer function (WOTF) and experiment on a star-like phase target, we demonstrate that our network has indeed learned the correct physical model rather than doing something trivial as pattern matching.

The eight-volume set LNCS 13431, 13432, 13433, 13434, 13435, 13436, 13437, and 13438 constitutes the refereed proceedings of the 25th International Conference on Medical Image Computing and Computer-Assisted Intervention, MICCAI 2022, which was held in Singapore in September 2022. The 574 revised full papers presented were carefully reviewed and selected from 1831 submissions in a double-blind review process. The papers are organized in the following topical sections: Part I: Brain development and atlases; DWI and tractography; functional brain networks; neuroimaging; heart and lung imaging; dermatology; Part II: Computational (integrative) pathology; computational anatomy and physiology; ophthalmology; fetal imaging; Part III: Breast imaging; colonoscopy; computer aided diagnosis; Part IV: Microscopic image analysis; positron emission tomography; ultrasound imaging; video data analysis; image segmentation I; Part V: Image segmentation II; integration of imaging with non-imaging biomarkers; Part VI: Image registration; image reconstruction; Part VII: Image-Guided interventions and surgery; outcome and disease prediction; surgical data science; surgical planning and simulation; machine learning – domain adaptation and generalization; Part VIII: Machine learning – weakly-supervised learning; machine learning – model interpretation; machine learning – uncertainty; machine learning theory and methodologies.

Exponential advancements in computational resources and algorithms have given birth to the new paradigm in imaging that rely on computation to digitally reconstruct and enhance images. These computational imaging modalities have enabled higher resolution, larger throughput and/or automatic detection capabilities for optical microscopy. An example is lens-less digital holographic microscope, which enables snapshot imaging of volumetric samples over wide field-of-view without using imaging lenses. Recent developments in the field of deep learning have further opened up exciting avenues for computational imaging, which offer unprecedented performance thanks to their capability to robustly learn content-specific complex image priors. This dissertation introduces a novel and universal modeling framework of deep learning -based image reconstruction technique to tackle various challenges in optical microscopic imaging, including digital holography reconstructions and 3D fluorescence microscopy. Firstly, auto-focusing and phase recovery in holography reconstruction are conventionally challenging and time-consuming to digitally perform. A convolutional neural network (CNN) based approach was developed that solves both problems rapidly in parallel, enabling extended depth-of-field holographic reconstruction with significantly improved time complexity from O(mn) to O(1). Secondly, to fuse advantages of snapshot volumetric capability in digital holography and speckle- and artifact-free image contrast in bright-field microscopy, a CNN was used to transform across microscopy modalities from holographic image reconstructions to their equivalent high contrast bright-field microscopic images. Thirdly, 3D fluorescence microscopy generally requires axial scanning. A CNN was trained to learn defocuses of fluorescence and digitally refocusing a single 2D fluorescence image onto user-defined 3D surfaces within the sample volume, which extends depth-of-field of fluorescence microscopy by 20-fold without any axial scanning, additional hardware, or a trade-off of imaging resolution or speed. This enables high-speed volumetric imaging and digital aberration correction for live samples. Based on deep learning powered computational microscopy, a hand-held device was also developed to measure the particulate matters and bio-aerosols in the air using the lens-less digital holographic microscopic imaging geometry. This device, named c-Air, demonstrates accurate, high-throughput and automatic detection, sizing and classification of the particles in the air, which opens new opportunities in deep learning based environmental sensing and personalized and/or distributed air quality monitoring.

This book constitutes the proceedings of the 11th International Workshop on Machine Learning in Medical Imaging, MLMI 2020, held in conjunction with MICCAI 2020, in Lima, Peru, in October 2020. The conference was held virtually due to the COVID-19 pandemic. The 68 papers presented in this volume were carefully reviewed and selected from 101 submissions. They focus on major trends and challenges in the above-mentioned area, aiming to identify new-cutting-edge techniques and their uses in medical imaging. Topics dealt with are: deep learning, generative adversarial learning, ensemble learning, sparse learning, multi-task learning, multi-view learning, manifold learning, and reinforcement learning, with their applications to medical image analysis, computer-aided detection and diagnosis, multi-modality fusion, image reconstruction, image retrieval, cellular image analysis, molecular imaging, digital pathology, etc.