Combining Multivariate Adaptive Regression Splines With A Response Surface Methodology For Simulation Based Design Optimization

Multivariate Adaptive Regression Splines (MARS) provide a flexible statistical modeling method that employs forward and backward search algorithms to identify the combination of basis functions that best fits the data. In optimization, MARS has been used successfully to estimate the value function in stochastic dynamic programming, and MARS could be potentially useful in many real world optimization problems where objective (or other) functions need to be estimated from data, such as in simulation optimization. Many optimization methods depend on convexity, but a nonconvex MARS approximation is inherently possible because interaction terms are products of univariate terms. In this dissertation, convex versions of MARS are proposed. In order to ensure MARS convexity, two major modifications are made: (1) coefficients are constrained such that pairs of basis functions are guaranteed to jointly form convex functions; (2) The form of interaction terms is appropriately changed. Finally, MARS convexity can be achieved by the fact that the sum of convex functions is convex. The implementation of MARS for approximating complex optimization functions can involve hundreds to thousands of state or decision variables. In particular, this research studies application to an inventory forecasting stochastic dynamic programming problem and an airline fleet assignment problem. Although one can simply attempt a MARS approximation over all the variables, prior research on the fleet assignment application indicates that many variables have little effect on the objective. Thus, a data mining step to conduct variable selection is needed. This step separates potentially critical variables from clearly redundant ones. In this dissertation, variants of two data mining tools are explored separately and in combination for variable selection: regression trees and multiple testing procedures based on false discovery rate.

Central to Dynamic Programming (DP) is the 'cost-to-go' or 'future value' function, which is obtained via solving the Bellman's equation, and central to many Approximate Dynamic Programming (ADP) methods is the approximation of the future value function. The exact DP algorithm seeks to compute and store a table consisting of one cost-to-go value for each point in the state space, and its usefulness is limited by the curse of dimensionality, which renders the methodology computationally intractable in the face of real life problems with high-dimensional state spaces and in the face of continuous state variables. ADP methodology seeks to address and redress the issue of the curse of dimensionality by not seeking to compute the future value function exactly and at each point of the state space; rather opting for an approximation of the future value function in the domain of the state space. Existing ADP methodologies have successfully handled 'continuous' state variables through discretization of the state space and estimation of the cost-to-go or future value function and have been challenged in scenarios involving a mix of 'continuous' and 'categorical' or qualitative state variables. The first part of this dissertation research seeks to develop a flexible, nonparametric statistical modeling method which can capture complex nonlinearity in data comprised of a mix of continuous and categorical variables and can be used to approximate future value functions in stochastic dynamic programming (SDP) problems with a mix of continuous and categorical state variables. This dissertation proposes a statistical modeling method, called 'TreeMARS' which combines the versatility of tree-models with the flexibility of multivariate adaptive regression splines (MARS). An extension of the proposed model, called 'Boosted TreeMARS', is also presented. Comparisons are made to the tree-regression model that uses a similar concept, but only permits the use of linear regression at the terminal nodes. Comparisons are presented on a 10-dimensional simulated data set. The second part of the dissertation research, seeking statistical parsimony, proposes a sequential statistical modeling methodology utilizing the 'sequential' concept from Design and Analysis of Computer Experiments (DACE) to make the grid 'only fine enough' for the 'efficient' discretization and then use MARS methods to approximate future value functions. This methodology can be extended in the future to use tree-based MARS models to approximate future value functions involving a mix of continuous and categorical state variables. This sequential grid discretization is nothing but sequential exploration of the state space and this concept of sequential exploration of the state space provides a statistically parsimonious ADP methodology which 'adaptively' captures the important variables from the state space and builds approximations around these quantities while seeking approximation of the future value functions, by using adaptive and flexible modeling.

Abstract: "There is a problem faced by experimenters in many technical fields, where, in general, the response variable of interest is y and there is a set of predictor variables x1, x2, ..., x[subscript k]. For example, in Dynamic Network Analysis (DNA) Response Surface Methodology (RSM) might be useful for sensitivity analysis of various DNA measures for different kinds of random graphs and errors. In Social Network Problems usually the underlying mechanism is not fully understood, and the experimenter must approximate the unknown function g with appropriate empirical model y = f(x1, x2, ..., x[subscript k]) + [epsilon], where the term [epsilon] represents the error in the system. Usually the function f is a first-order or second-order polynomial. This empirical model is called a response surface model. Identifying and fitting from experimental data an appropriate response surface model requires some use of statistical experimental design fundamentals, regression modeling techniques, and optimization methods. All three of these topics are usually combined into Response Surface Methodology (RSM). Also the experimenter may encounter situations where the full model may not be appropriate. Then variable selection or model-building techniques may be used to identify the best subset of regressors to include in a regression model. In our approach we use the simulated annealing method of optimization for searching the best subset of regressors. In some response surface experiments, there can be one or more near-linear dependences among regressor variables in the model. Regression model builders refer to this as multicollinearity among the regressors. Multicollinearity can have serious effects on the estimates of the model parameters and on the general applicability of the final model. The RSM is also extremely useful as an automated tool for model calibration and validation especially for modern computational multi-agent large scale social-networks systems that are becoming heavily used in modeling and simulation of complex social networks. The RSM can be integrated in many large-scale simulation systems such as BioWar, ORA and is currently integrating in Vista, Construct, and DyNet. This report describes the theoretical approach for solving of these problems and the implementation of chosen methods."

Over the last few decades, uncertainty quantification in composite materials and structures has gained a lot of attention from the research community as a result of industrial requirements. This book presents computationally efficient uncertainty quantification schemes following meta-model-based approaches for stochasticity in material and geometric parameters of laminated composite structures. Several metamodels have been studied and comparative results have been presented for different static and dynamic responses. Results for sensitivity analyses are provided for a comprehensive coverage of the relative importance of different material and geometric parameters in the global structural responses.

This book introduces methods of robust optimization in multivariate adaptive regression splines (MARS) and Conic MARS in order to handle uncertainty and non-linearity. The proposed techniques are implemented and explained in two-model regulatory systems that can be found in the financial sector and in the contexts of banking, environmental protection, system biology and medicine. The book provides necessary background information on multi-model regulatory networks, optimization and regression. It presents the theory of and approaches to robust (conic) multivariate adaptive regression splines - R(C)MARS – and robust (conic) generalized partial linear models – R(C)GPLM – under polyhedral uncertainty. Further, it introduces spline regression models for multi-model regulatory networks and interprets (C)MARS results based on different datasets for the implementation. It explains robust optimization in these models in terms of both the theory and methodology. In this context it studies R(C)MARS results with different uncertainty scenarios for a numerical example. Lastly, the book demonstrates the implementation of the method in a number of applications from the financial, energy, and environmental sectors, and provides an outlook on future research.

This Research Topic is Volume III of a series. The previous volume can be found here: Spatial Modelling and Failure Analysis of Natural and Engineering Disasters through Data-based Methods - Volume II and Spatial Modelling and Failure Analysis of Natural and Engineering Disasters through Data-based Methods Natural and engineering disasters, which include landslides, rock fall, rainstorm, dam failure, floods, earthquakes, road and building disasters and wildfires, appear as results of the progressive or extreme evolution of climatic, tectonic and geomorphological processes and human engineering activities. It is significant to explore the failure mechanism and carry out spatial modeling of these engineering and natural disasters due to their serious harm to the safety of people's lives and property. The data-based methods, including advanced and successful remote sensing, geographic information systems, machine learning and numerical simulation techniques methods, are promising tools to analyze these complex disasters. Machine Learning models such as neurofuzzy logic, decision tree, artificial neural network, deep learning and evolutionary algorithms are characterized by their abilities to produce knowledge and discover hidden and unknown patterns and trends from large databases, whereas remote sensing and Geographic Information Systems appear as significant technology equipped with tools for data manipulation and advanced mathematical modeling. What is more, the numerical simulation can also be acknowledged as advanced technologies for discovering hidden failure mechanism of disasters. The main objective of this Research Topic is to provide a scientific forum for advancing the successful implementation of Machine Learning (ML) and numerical simulation techniques in operation rules, failure mechanism, spatial and time series prediction, susceptibility mapping, hazard assessment, vulnerability modeling, risk assessment and early warning of complex natural and engineering disasters.

This book deals with metal processing and its numerical modelling and simulation. In total, 21 papers from different distinguished authors have been compiled in this area. Various processes are addressed, including solidification, TIG welding, additive manufacturing, hot and cold rolling, deep drawing, pipe deformation, and galvanizing. Material models are developed at different length scales from atomistic simulation to finite element analysis in order to describe the evolution and behavior of materials during thermal and thermomechanical treatment. Materials under consideration are carbon, Q&T, DP, and stainless steels; ductile iron; and aluminum, nickel-based, and titanium alloys. The developed models and simulations shall help to predict structure evolution, damage, and service behavior of advanced materials.

Multivariate Adaptive Regression Splines (MARS) is a statistical modeling method used to represent high-dimensional data with interactions. It uses different algorithms to select the terms to be included in the approximation model that best represent the data. In addition, it performs a variable selection, therefore the most significant predictors are shown in the final model. Design and analysis of computer experiments (DACE) is a statistical technique for creating approximations (called metamodels) of computer models. For optimization problems in which there is an unknown function that must be approximated, DACE approach could be applied. In stochastic dynamic programming (SDP) for example, a metamodel can be used to approximate the unknown future value function.The goal of DACE is to efficiently predict the response value of a computer model. MARS has been used as a metamodel in DACE technique. MARS is a flexible model, however in optimization, certain characteristics may be desired, such as a convex or piecewise-linear structure. To satisfy these characteristics, different variants of MARS have been developed. By enabling these variants, MARS modeling facilitates the optimization process. These variations include the ability to model a convex function, a piecewise-linear function and to provide a smoothing option using a quintic routine.DACE has had an enormous contribution for studying complex system, however one of consistent concerns for the researchers is computational time. As researchers seek to study more and more complex systems, corresponding computer models continue to push the limits of computing power. To overcome this drawback, efficient sequential approaches have been studied to reduce the computational effort. This research work focuses its efforts on the development of sequential approaches based on MARS model. The objective is to sequentially update the approximation function using current and new input data points. Additionally, by using less input data points, an accurate prediction of the unknown function could be obtained in a faster manner, and thus the complexity of the model structure is less. This could also facilitate the optimization process.Different case studies are shown in order to test the different MARS variants and sequential MARS approaches proposed in this dissertation. These cases include an inventory forecasting problem, an automotive crash safety design problem and an air pollution SDP problem.