Multi Modal 3d Mapping

With the democratization of collaborative navigation applications and autonomous robots, mobile mapping has received an increasing attention in recent years, both in academic and industrial circles. The digitization of the environment not only provides detailed and extensive knowledge enabling end-users to anticipate and plan their journeys, but also guarantees the availability of reliable up-to-date information in critical scenarios (e.g., when sensors of an autonomous car fail to perceive the surroundings). Mobile mapping raises, however, many challenges in terms of robustness, accuracy and scalability. Processing mapping data requires methods that are able to handle massive data with centimetric accuracy while coping with the acquisition specificities i.e., variability in levels of detail, occlusions, strong variations in lighting conditions, etc.In the context of the french ANR project pLaTINUM, this thesis focuses on the development of a global geolocalised map of an urban environment made of 3D representations based on geometric, photometric and semantic information. Firstly, a comparative investigation of the suitable geometric representation options yields to the reconstruction of a large scale, high definition map through a textured 3D mesh. This representation is the result of a multi-modal fusion of oriented images and geo-referenced LiDAR scans acquired by a terrestrial mobile mapping platform. Subsequently, we propose to infer high level semantics to the reconstructed frame by exploiting the complementarity between the two acquisition modalities (photometry and geometry). Throughout the rich literature regarding this subject, we have identified a need of an annotated multi-modal urban dataset comprising a large scale textured mesh. This has led us to produce our own dataset composed of 3D point clouds, 2D geolocalized panoramic and perspective images, depth and reflectance maps, and a 3D textured mesh with the corresponding ground truth annotations for each modality.Secondly, we assume that the global map is represented by means of 3D point cloud structured by an adjacency graph. We introduce a novel supervised over-segmentation approach. This method operates in two steps: (i) local descriptors of 3D points are computed via deep metric learning, (ii) the point cloud is partitioned into uniform clusters called superpoints. The descriptors are learned such that they present high contrast at the interface between objects, thereby encouraging the partition to follow their natural contours. Our experiments on indoor and outdoor scenes show the clear superiority of our approach over state-of-the-art point cloud partitioning methods. We further illustrate how our method can be combined with a superpoint-based classification algorithm to enhance the performance of semantic segmentation of 3D point clouds, also improving the state-of-the-art in this field.Finally, we extend this approach to textured meshes. Triangles, structured this time by the dual graph of the mesh, are partitioned into homogeneous groups called superfacets. Much like point clouds, local descriptors of the textured mesh are learned so that the boundaries of the objects exhibit high contrast. These descriptors are the result of merging descriptors learned from the convolution of the mesh edges on the one hand, and the texture descriptors extracted from the 2D image domain on the other. The experiments conducted on our own multi-modal dataset, show the superiority of our approach compared to state-of-the-art methods for the task of 3D mesh over-segmentation.

Multimodal Scene Understanding: Algorithms, Applications and Deep Learning presents recent advances in multi-modal computing, with a focus on computer vision and photogrammetry. It provides the latest algorithms and applications that involve combining multiple sources of information and describes the role and approaches of multi-sensory data and multi-modal deep learning. The book is ideal for researchers from the fields of computer vision, remote sensing, robotics, and photogrammetry, thus helping foster interdisciplinary interaction and collaboration between these realms. Researchers collecting and analyzing multi-sensory data collections – for example, KITTI benchmark (stereo+laser) - from different platforms, such as autonomous vehicles, surveillance cameras, UAVs, planes and satellites will find this book to be very useful. Contains state-of-the-art developments on multi-modal computing Shines a focus on algorithms and applications Presents novel deep learning topics on multi-sensor fusion and multi-modal deep learning

This book presents the breakthrough and cutting-edge progress for collaborative perception and mapping by proposing a novel framework of multimodal perception-relative localization–collaborative mapping for collaborative robot systems. The organization of the book allows the readers to analyze, model and design collaborative perception technology for autonomous robots. It presents the basic foundation in the field of collaborative robot systems and the fundamental theory and technical guidelines for collaborative perception and mapping. The book significantly promotes the development of autonomous systems from individual intelligence to collaborative intelligence by providing extensive simulations and real experiments results in the different chapters. This book caters to engineers, graduate students and researchers in the fields of autonomous systems, robotics, computer vision and collaborative perception.

The Handbook of Multimodal-Multisensor Interfaces provides the first authoritative resource on what has become the dominant paradigm for new computer interfaces— user input involving new media (speech, multi-touch, gestures, writing) embedded in multimodal-multisensor interfaces. These interfaces support smart phones, wearables, in-vehicle and robotic applications, and many other areas that are now highly competitive commercially. This edited collection is written by international experts and pioneers in the field. It provides a textbook, reference, and technology roadmap for professionals working in this and related areas. This first volume of the handbook presents relevant theory and neuroscience foundations for guiding the development of high-performance systems. Additional chapters discuss approaches to user modeling and interface designs that support user choice, that synergistically combine modalities with sensors, and that blend multimodal input and output. This volume also highlights an in-depth look at the most common multimodal-multisensor combinations—for example, touch and pen input, haptic and non-speech audio output, and speech-centric systems that co-process either gestures, pen input, gaze, or visible lip movements. A common theme throughout these chapters is supporting mobility and individual differences among users. These handbook chapters provide walk-through examples of system design and processing, information on tools and practical resources for developing and evaluating new systems, and terminology and tutorial support for mastering this emerging field. In the final section of this volume, experts exchange views on a timely and controversial challenge topic, and how they believe multimodal-multisensor interfaces should be designed in the future to most effectively advance human performance.

This book constitutes the refereed proceedings of the First International Workshop on Multimodal Brain Image Analysis, held in conjunction with MICCAI 2011, in Toronto, Canada, in September 2011. The 15 revised full papers presented together with 4 poster papers were carefully reviewed and selected from 24 submissions. The objective of this workshop is to facilitate advancements in the multimodal brain image analysis field, in terms of analysis methodologies, algorithms, software systems, validation approaches, benchmark datasets, neuroscience, and clinical applications.

This book constitutes the refereed proceedings of the 5th International Workshop on Machine Learning for Multimodal Interaction, MLMI 2008, held in Utrecht, The Netherlands, in September 2008. The 12 revised full papers and 15 revised poster papers presented together with 5 papers of a special session on user requirements and evaluation of multimodal meeting browsers/assistants were carefully reviewed and selected from 47 submissions. The papers cover a wide range of topics related to human-human communication modeling and processing, as well as to human-computer interaction, using several communication modalities. Special focus is given to the analysis of non-verbal communication cues and social signal processing, the analysis of communicative content, audio-visual scene analysis, speech processing, interactive systems and applications.

A comprehensive review of the most recent applications of intelligent multi-modal data processing Intelligent Multi-Modal Data Processing contains a review of the most recent applications of data processing. The Editors and contributors – noted experts on the topic – offer a review of the new and challenging areas of multimedia data processing as well as state-of-the-art algorithms to solve the problems in an intelligent manner. The text provides a clear understanding of the real-life implementation of different statistical theories and explains how to implement various statistical theories. Intelligent Multi-Modal Data Processing is an authoritative guide for developing innovative research ideas for interdisciplinary research practices. Designed as a practical resource, the book contains tables to compare statistical analysis results of a novel technique to that of the state-of-the-art techniques and illustrations in the form of algorithms to establish a pre-processing and/or post-processing technique for model building. The book also contains images that show the efficiency of the algorithm on standard data set. This important book: Includes an in-depth analysis of the state-of-the-art applications of signal and data processing Contains contributions from noted experts in the field Offers information on hybrid differential evolution for optimal multilevel image thresholding Presents a fuzzy decision based multi-objective evolutionary method for video summarisation Written for students of technology and management, computer scientists and professionals in information technology, Intelligent Multi-Modal Data Processing brings together in one volume the range of multi-modal data processing.

This book constitutes the refereed joint proceedings of the 4th International Workshop on Multimodal Brain Image Analysis, MBAI 2019, and the 7th International Workshop on Mathematical Foundations of Computational Anatomy, MFCA 2019, held in conjunction with the 22nd International Conference on Medical Imaging and Computer-Assisted Intervention, MICCAI 2019, in Shenzhen, China, in October 2019. The 16 full papers presented at MBAI 2019 and the 7 full papers presented at MFCA 2019 were carefully reviewed and selected. The MBAI papers intend to move forward the state of the art in multimodal brain image analysis, in terms of analysis methodologies, algorithms, software systems, validation approaches, benchmark datasets, neuroscience, and clinical applications. The MFCA papers are devoted to statistical and geometrical methods for modeling the variability of biological shapes. The goal is to foster the interactions between the mathematical community around shapes and the MICCAI community around computational anatomy applications.