Uncertainties In Neural Networks

In science, technology, and engineering, creating models of the environment to predict future events has always been a key component. The models could be everything from how the friction of a tire depends on the wheels slip to how a pathogen is spread throughout society. As more data becomes available, the use of data-driven black-box models becomes more attractive. In many areas they have shown promising results, but for them to be used widespread in safety-critical applications such as autonomous driving some notion of uncertainty in the prediction is required. An example of such a black-box model is neural networks (NNs). This thesis aims to increase the usefulness of NNs by presenting an method where uncertainty in the prediction is obtained by linearization of the model. In system identification and sensor fusion, under the condition that the model structure is identifiable, this is a commonly used approach to get uncertainty in the prediction from a nonlinear model. If the model structure is not identifiable, such as for NNs, the ambiguities that cause this have to be taken care of in order to make the approach applicable. This is handled in the first part of the thesis where NNs are analyzed from a system identification perspective, and sources of uncertainty are discussed. Another problem with data-driven black-box models is that it is difficult to know how flexible the model needs to be in order to correctly model the true system. One solution to this problem is to use a model that is more flexible than necessary to make sure that the model is flexible enough. But how would that extra flexibility affect the uncertainty in the prediction? This is handled in the later part of the thesis where it is shown that the uncertainty in the prediction is bounded from below by the uncertainty in the prediction of the model with lowest flexibility required for representing true system accurately. In the literature, many other approaches to handle the uncertainty in predictions by NNs have been suggested, of which some are summarized in this work. Furthermore, a simulation and an experimental studies inspired by autonomous driving are conducted. In the simulation study, different sources of uncertainty are investigated, as well as how large the uncertainty in the predictions by NNs are in areas without training data. In the experimental study, the uncertainty in predictions done by different models are investigated. The results show that, compared to existing methods, the linearization method produces similar results for the uncertainty in predictions by NNs. An introduction video is available at https://youtu.be/O4ZcUTGXFN0 Inom forskning och utveckling har det har alltid varit centralt att skapa modeller av verkligheten. Dessa modeller har bland annat använts till att förutspå framtida händelser eller för att styra ett system till att bete sig som man önskar. Modellerna kan beskriva allt från hur friktionen hos ett bildäck påverkas av hur mycket hjulen glider till hur ett virus kan sprida sig i ett samhälle. I takt med att mer och mer data blir tillgänglig ökar potentialen för datadrivna black-box modeller. Dessa modeller är universella approximationer vilka ska kunna representera vilken godtycklig funktion som helst. Användningen av dessa modeller har haft stor framgång inom många områden men för att verkligen kunna etablera sig inom säkerhetskritiska områden såsom självkörande farkoster behövs en förståelse för osäkerhet i prediktionen från modellen. Neuronnät är ett exempel på en sådan black-box modell. I denna avhandling kommer olika sätt att tillförskaffa sig kunskap om osäkerhet i prediktionen av neuronnät undersökas. En metod som bygger på linjärisering av modellen för att tillförskaffa sig osäkerhet i prediktionen av neuronnätet kommer att presenteras. Denna metod är välbeprövad inom systemidentifiering och sensorfusion under antagandet att modellen är identifierbar. För modeller såsom neuronnät, vilka inte är identifierbara behövs det att det tas hänsyn till tvetydigheterna i modellen. En annan utmaning med datadrivna black-box modeller, är att veta om den valda modellmängden är tillräckligt generell för att kunna modellera det sanna systemet. En lösning på detta problem är att använda modeller som har mer flexibilitet än vad som behövs, det vill säga en överparameteriserad modell. Men hur påverkas osäkerheten i prediktionen av detta? Detta är något som undersöks i denna avhandling, vilken visar att osäkerheten i den överparameteriserad modellen kommer att vara begränsad underifrån av modellen med minst flexibilitet som ändå är tillräckligt generell för att modellera det sanna systemet. Som avslutning kommer dessa resultat att demonstreras i både en simuleringsstudie och en experimentstudie inspirerad av självkörande farkoster. Fokuset i simuleringsstudien är hur osäkerheten hos modellen är i områden med och utan tillgång till träningsdata medan experimentstudien fokuserar på jämförelsen mellan osäkerheten i olika typer av modeller.Resultaten från dessa studier visar att metoden som bygger på linjärisering ger liknande resultat för skattningen av osäkerheten i prediktionen av neuronnät, jämfört med existerande metoder.

This book constitutes the proceedings of the 12th IFIP TC 8 International Conference, CISIM 2013, held in Cracow, Poland, in September 2013. The 44 papers presented in this volume were carefully reviewed and selected from over 60 submissions. They are organized in topical sections on biometric and biomedical applications; pattern recognition and image processing; various aspects of computer security, networking, algorithms, and industrial applications. The book also contains full papers of a keynote speech and the invited talk.

Bayesian methods for machine learning have been widely investigated, yielding principled methods for incorporating prior information into inference algorithms. This monograph provides the reader with an in-depth review of the role of Bayesian methods for the reinforcement learning (RL) paradigm. The major incentives for incorporating Bayesian reasoning in RL are that it provides an elegant approach to action-selection (exploration/exploitation) as a function of the uncertainty in learning, and it provides a machinery to incorporate prior knowledge into the algorithms. Bayesian Reinforcement Learning: A Survey first discusses models and methods for Bayesian inference in the simple single-step Bandit model. It then reviews the extensive recent literature on Bayesian methods for model-based RL, where prior information can be expressed on the parameters of the Markov model. It also presents Bayesian methods for model-free RL, where priors are expressed over the value function or policy class. Bayesian Reinforcement Learning: A Survey is a comprehensive reference for students and researchers with an interest in Bayesian RL algorithms and their theoretical and empirical properties.

Artificial "neural networks" are widely used as flexible models for classification and regression applications, but questions remain about how the power of these models can be safely exploited when training data is limited. This book demonstrates how Bayesian methods allow complex neural network models to be used without fear of the "overfitting" that can occur with traditional training methods. Insight into the nature of these complex Bayesian models is provided by a theoretical investigation of the priors over functions that underlie them. A practical implementation of Bayesian neural network learning using Markov chain Monte Carlo methods is also described, and software for it is freely available over the Internet. Presupposing only basic knowledge of probability and statistics, this book should be of interest to researchers in statistics, engineering, and artificial intelligence.

Uncertainty has been of concern to engineers, managers and . scientists for many centuries. In management sciences there have existed definitions of uncertainty in a rather narrow sense since the beginning of this century. In engineering and uncertainty has for a long time been considered as in sciences, however, synonymous with random, stochastic, statistic, or probabilistic. Only since the early sixties views on uncertainty have ~ecome more heterogeneous and more tools to model uncertainty than statistics have been proposed by several scientists. The problem of modeling uncertainty adequately has become more important the more complex systems have become, the faster the scientific and engineering world develops, and the more important, but also more difficult, forecasting of future states of systems have become. The first question one should probably ask is whether uncertainty is a phenomenon, a feature of real world systems, a state of mind or a label for a situation in which a human being wants to make statements about phenomena, i. e. , reality, models, and theories, respectively. One cart also ask whether uncertainty is an objective fact or just a subjective impression which is closely related to individual persons. Whether uncertainty is an objective feature of physical real systems seems to be a philosophical question. This shall not be answered in this volume.

This book constitutes the refereed proceedings of the First International Workshop on Uncertainty for Safe Utilization of Machine Learning in Medical Imaging, UNSURE 2019, and the 8th International Workshop on Clinical Image-Based Procedures, CLIP 2019, held in conjunction with MICCAI 2019, in Shenzhen, China, in October 2019. For UNSURE 2019, 8 papers from 15 submissions were accepted for publication. They focus on developing awareness and encouraging research in the field of uncertainty modelling to enable safe implementation of machine learning tools in the clinical world. CLIP 2019 accepted 11 papers from the 15 submissions received. The workshops provides a forum for work centred on specific clinical applications, including techniques and procedures based on comprehensive clinical image and other data.

Probability is the bedrock of machine learning. You cannot develop a deep understanding and application of machine learning without it. Cut through the equations, Greek letters, and confusion, and discover the topics in probability that you need to know. Using clear explanations, standard Python libraries, and step-by-step tutorial lessons, you will discover the importance of probability to machine learning, Bayesian probability, entropy, density estimation, maximum likelihood, and much more.

This book constitutes the proceedings of the 19th International Symposium on Intelligent Data Analysis, IDA 2021, which was planned to take place in Porto, Portugal. Due to the COVID-19 pandemic the conference was held online during April 26-28, 2021. The 35 papers included in this book were carefully reviewed and selected from 113 submissions. The papers were organized in topical sections named: modeling with neural networks; modeling with statistical learning; modeling language and graphs; and modeling special data formats.

This volume constitutes the refereed proceedings of the 19th International Conference on Business Process Management, BPM 2021, held in Rome, Italy, in September 2021. The 23 full papers, one keynote paper, and 4 tutorial papers presented in this volume were carefully reviewed and selected from 92 submissions. The papers are organized in topical sections named: foundations, engineering, and management.

This book covers theoretical aspects as well as recent innovative applications of Artificial Neural networks (ANNs) in natural, environmental, biological, social, industrial and automated systems. It presents recent results of ANNs in modelling small, large and complex systems under three categories, namely, 1) Networks, Structure Optimisation, Robustness and Stochasticity 2) Advances in Modelling Biological and Environmental Systems and 3) Advances in Modelling Social and Economic Systems. The book aims at serving undergraduates, postgraduates and researchers in ANN computational modelling.