Understanding And Accelerating The Optimization Of Modern Machine Learning

Over the last decade, we have seen impressive progress of deep learning on a variety of intelligence tasks. The success of deep learning is due, to a great extent, to the remarkable effectiveness of gradient-based optimization methods applied to large neural networks, which is often over-parameterized, i.e., the number of parameters greatly exceeds the number of training samples. However, theoretically, it is still far from clear that why gradient descent algorithms can efficiently optimize the seemingly highly non-convex loss functions (a.k.a. objective functions). In this dissertation, we target to close this gap between the theory and the practice. We first show that certain sufficiently wide neural networks, as typical examples of large non-linear models, exhibit an amazing, while somehow counter-intuitive, phenomenon--transition to linearity. Specifically, these networks can be well approximated by linear models; moreover, they become linear models in the infinite network width limit. Based on this phenomenon, we further provide an optimization theory that describes the loss landscape of over-parameterized machine learning models and further explains the convergence of gradient descent methods on these models. Note that this theory covers both the models that have the "transition to linearity'', and those may not have this property, e.g., wide networks with non-linear output layer. Finally, we prove that, in the stochastic setting, the popularly used Nesterov's momentum does not accelerate the stochastic gradient descent, even for quadratic optimization problems. Furthermore, we propose a new method, MaSS, that provably accelerate SGD in the over-parameterized setting.

This book on optimization includes forewords by Michael I. Jordan, Zongben Xu and Zhi-Quan Luo. Machine learning relies heavily on optimization to solve problems with its learning models, and first-order optimization algorithms are the mainstream approaches. The acceleration of first-order optimization algorithms is crucial for the efficiency of machine learning. Written by leading experts in the field, this book provides a comprehensive introduction to, and state-of-the-art review of accelerated first-order optimization algorithms for machine learning. It discusses a variety of methods, including deterministic and stochastic algorithms, where the algorithms can be synchronous or asynchronous, for unconstrained and constrained problems, which can be convex or non-convex. Offering a rich blend of ideas, theories and proofs, the book is up-to-date and self-contained. It is an excellent reference resource for users who are seeking faster optimization algorithms, as well as for graduate students and researchers wanting to grasp the frontiers of optimization in machine learning in a short time.

Given their tremendous success in commercial applications, machine learning (ML) models are increasingly being considered as alternatives to science-based models in many disciplines. Yet, these "black-box" ML models have found limited success due to their inability to work well in the presence of limited training data and generalize to unseen scenarios. As a result, there is a growing interest in the scientific community on creating a new generation of methods that integrate scientific knowledge in ML frameworks. This emerging field, called scientific knowledge-guided ML (KGML), seeks a distinct departure from existing "data-only" or "scientific knowledge-only" methods to use knowledge and data at an equal footing. Indeed, KGML involves diverse scientific and ML communities, where researchers and practitioners from various backgrounds and application domains are continually adding richness to the problem formulations and research methods in this emerging field. Knowledge Guided Machine Learning: Accelerating Discovery using Scientific Knowledge and Data provides an introduction to this rapidly growing field by discussing some of the common themes of research in KGML using illustrative examples, case studies, and reviews from diverse application domains and research communities as book chapters by leading researchers. KEY FEATURES First-of-its-kind book in an emerging area of research that is gaining widespread attention in the scientific and data science fields Accessible to a broad audience in data science and scientific and engineering fields Provides a coherent organizational structure to the problem formulations and research methods in the emerging field of KGML using illustrative examples from diverse application domains Contains chapters by leading researchers, which illustrate the cutting-edge research trends, opportunities, and challenges in KGML research from multiple perspectives Enables cross-pollination of KGML problem formulations and research methods across disciplines Highlights critical gaps that require further investigation by the broader community of researchers and practitioners to realize the full potential of KGML

An up-to-date account of the interplay between optimization and machine learning, accessible to students and researchers in both communities. The interplay between optimization and machine learning is one of the most important developments in modern computational science. Optimization formulations and methods are proving to be vital in designing algorithms to extract essential knowledge from huge volumes of data. Machine learning, however, is not simply a consumer of optimization technology but a rapidly evolving field that is itself generating new optimization ideas. This book captures the state of the art of the interaction between optimization and machine learning in a way that is accessible to researchers in both fields. Optimization approaches have enjoyed prominence in machine learning because of their wide applicability and attractive theoretical properties. The increasing complexity, size, and variety of today's machine learning models call for the reassessment of existing assumptions. This book starts the process of reassessment. It describes the resurgence in novel contexts of established frameworks such as first-order methods, stochastic approximations, convex relaxations, interior-point methods, and proximal methods. It also devotes attention to newer themes such as regularized optimization, robust optimization, gradient and subgradient methods, splitting techniques, and second-order methods. Many of these techniques draw inspiration from other fields, including operations research, theoretical computer science, and subfields of optimization. The book will enrich the ongoing cross-fertilization between the machine learning community and these other fields, and within the broader optimization community.

Machine learning and optimization techniques are revolutionizing our world. Other types of information technology have not progressed as rapidly in recent years, in terms of real impact. The aim of this book is to present some of the innovative techniques in the field of optimization and machine learning, and to demonstrate how to apply them in the fields of engineering. Optimization and Machine Learning presents modern advances in the selection, configuration and engineering of algorithms that rely on machine learning and optimization. The first part of the book is dedicated to applications where optimization plays a major role, and the second part describes and implements several applications that are mainly based on machine learning techniques. The methods addressed in these chapters are compared against their competitors, and their effectiveness in their chosen field of application is illustrated.

Optimization happens everywhere. Machine learning is one example of such and gradient descent is probably the most famous algorithm for performing optimization. Optimization means to find the best value of some function or model. That can be the maximum or the minimum according to some metric. Using clear explanations, standard Python libraries, and step-by-step tutorial lessons, you will learn how to find the optimum point to numerical functions confidently using modern optimization algorithms.

This book discusses one of the major applications of artificial intelligence: the use of machine learning to extract useful information from multimodal data. It discusses the optimization methods that help minimize the error in developing patterns and classifications, which further helps improve prediction and decision-making. The book also presents formulations of real-world machine learning problems, and discusses AI solution methodologies as standalone or hybrid approaches. Lastly, it proposes novel metaheuristic methods to solve complex machine learning problems. Featuring valuable insights, the book helps readers explore new avenues leading toward multidisciplinary research discussions.

Companies are spending billions on machine learning projects, but it’s money wasted if the models can’t be deployed effectively. In this practical guide, Hannes Hapke and Catherine Nelson walk you through the steps of automating a machine learning pipeline using the TensorFlow ecosystem. You’ll learn the techniques and tools that will cut deployment time from days to minutes, so that you can focus on developing new models rather than maintaining legacy systems. Data scientists, machine learning engineers, and DevOps engineers will discover how to go beyond model development to successfully productize their data science projects, while managers will better understand the role they play in helping to accelerate these projects. Understand the steps to build a machine learning pipeline Build your pipeline using components from TensorFlow Extended Orchestrate your machine learning pipeline with Apache Beam, Apache Airflow, and Kubeflow Pipelines Work with data using TensorFlow Data Validation and TensorFlow Transform Analyze a model in detail using TensorFlow Model Analysis Examine fairness and bias in your model performance Deploy models with TensorFlow Serving or TensorFlow Lite for mobile devices Learn privacy-preserving machine learning techniques