A Reinforcement Learning Approach To Spacecraft Trajectory Optimization

This dissertation explores a novel method of solving low-thrust spacecraft targeting problems using reinforcement learning. A reinforcement learning algorithm based on Deep Deterministic Policy Gradients was developed to solve low-thrust trajectory optimization problems. The algorithm consists of two neural networks, an actor network and a critic network. The actor approximates a thrust magnitude given the current spacecraft state expressed as a set of orbital elements. The critic network evaluates the action taken by the actor based on the state and action taken. Three different types of trajectory problems were solved, a generalized orbit change maneuver, a semimajor axis change maneuver, and an inclination change maneuver. When training the algorithm in a simulated space environment, it was able to solve both the generalized orbit change and semimajor axis change maneuvers with no prior knowledge of the environment’s dynamics. The robustness of the algorithm was tested on an inclination change maneuver with a randomized set of initial states. After training, the algorithm was able to successfully generalize and solve new inclination changes that it has not seen before. This method has potential future applications in developing more complex low-thrust maneuvers or real-time autonomous spaceflight control.

This is a long-overdue volume dedicated to space trajectory optimization. Interest in the subject has grown, as space missions of increasing levels of sophistication, complexity, and scientific return - hardly imaginable in the 1960s - have been designed and flown. Although the basic tools of optimization theory remain an accepted canon, there has been a revolution in the manner in which they are applied and in the development of numerical optimization. This volume purposely includes a variety of both analytical and numerical approaches to trajectory optimization. The choice of authors has been guided by the editor's intention to assemble the most expert and active researchers in the various specialities presented. The authors were given considerable freedom to choose their subjects, and although this may yield a somewhat eclectic volume, it also yields chapters written with palpable enthusiasm and relevance to contemporary problems.

This book explores the design of optimal trajectories for space maneuver vehicles (SMVs) using optimal control-based techniques. It begins with a comprehensive introduction to and overview of three main approaches to trajectory optimization, and subsequently focuses on the design of a novel hybrid optimization strategy that combines an initial guess generator with an improved gradient-based inner optimizer. Further, it highlights the development of multi-objective spacecraft trajectory optimization problems, with a particular focus on multi-objective transcription methods and multi-objective evolutionary algorithms. In its final sections, the book studies spacecraft flight scenarios with noise-perturbed dynamics and probabilistic constraints, and designs and validates new chance-constrained optimal control frameworks. The comprehensive and systematic treatment of practical issues in spacecraft trajectory optimization is one of the book’s major features, making it particularly suited for readers who are seeking practical solutions in spacecraft trajectory optimization. It offers a valuable asset for researchers, engineers, and graduate students in GNC systems, engineering optimization, applied optimal control theory, etc.

The use of electric propulsion (EP) in satellites for transfer to geosynchronous equatorial orbit (GEO) is increasingly gaining importance among the space industry all around the world, and is proven a key for new space missions. In a conventional launch, the satellite is placed into a geostationary transfer orbit (GTO) by the launch vehicle and uses chemical propellants to reach GEO. This orbital transfer maneuver typically takes a few days. However, even though EP is far more e cient than the conventional chemical propulsion, its low thrust generation adds the complexity of longer transfer time from an equatorial orbit to GEO. This longer transit time leads to exposure of spacecraft to hazardous radiation of Van Allen belts. Therefore, there is a need to develop a method to determine the minimum transfer time trajectory for all-electric low thrust orbit raising problem. This thesis proposes a new formulation that facilitates the application of reinforcement learning to the problem of orbit raising. This work is based on the approach that the electric orbit-raising problem is posed as a sequence of multiple trajectory optimization sub-problems. Each sub-problem aims to move the spacecraft closest to GEO by minimizing a convex combination of suitably selected objectives. A mathematical formulation for the orbit-raising problem is proposed in the framework of reinforcement learning to enable adaptive modi cation of the objective function weights during a transfer. Due to high dimensionality of the planning states of the orbit-raising problem, arti cial neural networks are then constructed and trained on orbit-raising scenarios in order to compute the reward functions associated with reinforcement learning. The reward function for a planning state is de ned as the time required to reach GEO from that planning state. With the help of numerical simulations for planar and non-planar transfer scenarios, it is demonstrated that there is a reduction in transfer time for low-thrust orbit raising problem with the proposed methodology.

November 28-December 1, 1994, Denver, Colorado NIPS is the longest running annual meeting devoted to Neural Information Processing Systems. Drawing on such disparate domains as neuroscience, cognitive science, computer science, statistics, mathematics, engineering, and theoretical physics, the papers collected in the proceedings of NIPS7 reflect the enduring scientific and practical merit of a broad-based, inclusive approach to neural information processing. The primary focus remains the study of a wide variety of learning algorithms and architectures, for both supervised and unsupervised learning. The 139 contributions are divided into eight parts: Cognitive Science, Neuroscience, Learning Theory, Algorithms and Architectures, Implementations, Speech and Signal Processing, Visual Processing, and Applications. Topics of special interest include the analysis of recurrent nets, connections to HMMs and the EM procedure, and reinforcement- learning algorithms and the relation to dynamic programming. On the theoretical front, progress is reported in the theory of generalization, regularization, combining multiple models, and active learning. Neuroscientific studies range from the large-scale systems such as visual cortex to single-cell electrotonic structure, and work in cognitive scientific is closely tied to underlying neural constraints. There are also many novel applications such as tokamak plasma control, Glove-Talk, and hand tracking, and a variety of hardware implementations, with particular focus on analog VLSI.

This book focuses on the design and application of advanced trajectory optimization and guidance and control (G&C) techniques for aerospace vehicles. Part I of the book focuses on the introduction of constrained aerospace vehicle trajectory optimization problems, with particular emphasis on the design of high-fidelity trajectory optimization methods, heuristic optimization-based strategies, and fast convexification-based algorithms. In Part II, various optimization theory/artificial intelligence (AI)-based methods are constructed and presented, including dynamic programming-based methods, model predictive control-based methods, and deep neural network-based algorithms. Key aspects of the application of these approaches, such as their main advantages and inherent challenges, are detailed and discussed. Some practical implementation considerations are then summarized, together with a number of future research topics. The comprehensive and systematic treatment of practical issues in aerospace trajectory optimization and guidance and control problems is one of the main features of the book, which is particularly suitable for readers interested in learning practical solutions in aerospace trajectory optimization and guidance and control. The book is useful to researchers, engineers, and graduate students in the fields of G&C systems, engineering optimization, applied optimal control theory, etc.

Artificial intelligence is a rapidly developing field that promises to revolutionize spaceflight with greater robotic autonomy and innovative decision making. However, it remains to be determined which applications are best addressed using this new technology. In the coming decades, future spacecraft will be required to possess autonomous guidance and control in the complex, nonlinear dynamical regimes of cis-lunar space. In the realm of trajectory design, current methods struggle with local minima, and searching large solutions spaces. This thesis investigates the use of the Reinforcement Learning (RL) algorithm Proximal Policy Optimization (PPO) for solving low-thrust spacecraft guidance and control problems. First, an agent is trained to complete a 302 day mass-optimal low-thrust transfer between the Earth and Mars. This is accomplished while only providing the agent with information regarding its own state and that of Mars. By comparing these results to those generated by the Evolutionary Mission Trajectory Generator (EMTG), the optimality of the trajectory designed using PPO is assessed. Next, an agent is trained as an onboard regulator capable of correcting state errors and following pre-calculated transfers between libration point orbits. The feasibility of this method is examined by evaluating the agent’s ability to correct varying levels of initial state error via Monte Carlo testing. The generalizability of the agent’s control solution is appraised on three similar transfers of increasing difficulty not seen during the training process. The results show both the promise of the proposed PPO methodology and its limitations, which are discussed.

Key Features: Provides an interdisciplinary approach, with chapter contributions from expert teams working in the governmental or private space sectors, with valuable contributions from computer scientists and legal experts; Presents insights into AI implementation and how to unlock AI technologies in the field; Up to date with the latest developments and cutting-edge applications

A textbook on the theory and applications of optimal spacecraft trajectories

The optimization of spacecraft trajectories has been, and continues to be, critical for the development of modern space missions. Longer flight times, continuous low-thrust propulsion, and multiple flybys are just a few of the modern features resulting in increasingly complex optimal control problems for trajectory designers to solve. In order to efficiently tackle such challenging problems, a variety of methods and algorithms have been developed over the last decades. The work presented in this dissertation aims at improving the solutions and the robustness of the optimal control algorithms, in addition to reducing their computational load and the amount of necessary human involvement. Several areas of improvement are examined in the dissertation. First, the general formulation of a Differential Dynamic Programming (DDP) algorithm is examined, and new theoretical developments are made in order to achieve a multiple-shooting formulation of the method. Multiple-shooting transcriptions have been demonstrated to be beneficial to both direct and indirect optimal control methods, as they help decrease the large sensitivities present in highly nonlinear problems (thus improving the algorithms' robustness), and increase the potential for a parallel implementation. The new Multiple-Shooting Differential Dynamic Programming algorithm (MDDP) is the first application of the well-known multiple-shooting principles to DDP. The algorithm uses a null-space trust-region method for the optimization of quadratic subproblems subject to simple bounds, which permits to control the quality of the quadratic approximations of the objective function. Equality and inequality path and terminal constraints are treated with a general Augmented Lagrangian approach. The choice of a direct transcription and of an Augmented Lagrangian merit function, associated with automated partial computations, make the MDDP implementation flexible, requiring minimal user effort for changes in the dynamics, cost and constraint functions. The algorithm is implemented in a general, modular optimal control software, and the performance of the multiple-shooting formulation is analyzed. The use of quasi-Newton approximations in the context of DDP is examined, and numerically demonstrated to improve computational efficiency while retaining attractive convergence properties. The computational performance of an optimal control algorithm is closely related to that of the integrator chosen for the propagation of the equation of motion. In an effort to improve the efficiency of the MDDP algorithm, a new numerical propagation method is developed for the Kepler, Stark, and three-body problems, three of the most commonly used dynamical models for spacecraft trajectory optimization. The method uses a time regularization technique, the generalized Sundman transformation, and Taylor Series developments of equivalents to the f and g functions for each problem. The performance of the new method is examined, and specific domains where the series solution outperforms existing propagation methods are identified. Finally, because the robustness and computational efficiency of the MDDP algorithm depend on the quality of the first- and second-order State Transition Matrices, the three most common techniques for their computation are analyzed, in particular for low-fidelity propagation. The propagation of variational equations is compared to the complex step derivative approximation and finite differences methods, for a variety of problems and integration techniques. The subtle differences between variable- and fixed-step integration for partial computation are revealed, common pitfalls are observed, and recommendations are made for the practitioner to enhance the quality of state transition matrices.