A Metric Evaluation Approach For The Defense Acquisition Workforce Improvement Act

The Defense Acquisition Workforce Improvement Act (DAWIA) was enacted in 1990 to improve the quality and professionalism of the Department of Defense acquisition workforce. To assess the effectiveness of DAWIA, actual outcomes of the law must be compared to its original objectives. A particular difficulty in evaluating public policies is that they usually cannot be measured in terms of output, such as dollars, hours, or physical units. The primary objective of this study was to find and introduce a performance measurement approach suitable for identifying effective metrics. A second objective was to establish a link between metrics and outcomes. Using the performance measurement approach as a tool, an analysis attempted to link acquisition workforce metrics with specific outcomes. To explore this issue, a literature review of relevant organizational and management texts on public policy analysis, performance measurement and program evaluation was conducted. Four suitable frameworks for performance measurement were found and evaluated. The preferred approach for evaluating DAWIA was determined to be a combination of two performance measurement approaches. The new approach is called Metric Assessment and Measurement Approach. It includes valuable features of the two approaches, and a newly developed metric template for evaluating metrics.

For every weapons system being developed, the U.S. Department of Defense (DOD) must make a critical decision: Should the system go forward to full-scale production? The answer to that question may involve not only tens of billions of dollars but also the nation's security and military capabilities. In the milestone process used by DOD to answer the basic acquisition question, one component near the end of the process is operational testing, to determine if a system meets the requirements for effectiveness and suitability in realistic battlefield settings. Problems discovered at this stage can cause significant production delays and can necessitate costly system redesign. This book examines the milestone process, as well as the DOD's entire approach to testing and evaluating defense systems. It brings to the topic of defense acquisition the application of scientific statistical principles and practices.

Numerous reform initiatives have attempted to improve the acquisition process. The success or failure of these initiatives have often been based on subjective determinations. In order to determine the true effect of these initiatives we must be able to measure the effect of these initiatives on the acquisition process. Measurement requires the development of metrics. This study explores the use of metrics for acquisition reform, using the Defense Acquisition Workforce Improvement Act (DAWIA), Public Law 101- 510, as a case study. This study identifies the objectives of DAWIA. Using the Policy Effectiveness Model, the study develops and proposes metrics for DAWIA objectives in an effort to measure the implementation and effectiveness of this important and far-reaching acquisition reform legislation.

The Department of Defense (DoD) recently adopted evolutionary acquisition, a dynamic strategy for the development and acquisition of its defense systems. Evolutionary defense systems are planned, in advance, to be developed through several stages in a single procurement program. Each stage is planned to produce a viable system which could be fielded. The system requirements for each stage of development may be specified in advance of a given stage or may be decided at the outset of that stage's development. Due to the different stages that comprise an evolutionary system, there exists a need for careful reexamination of current testing and evaluation policies and processes, which were designed for single-stage developments. The Office of the Under Secretary of Defense for Acquisition, Technology and Logistics (USD-AT&L) and the Director of Operational Testing and Evaluation (DOT&E) asked the Committee on National Statistics (CNSTAT) of the National Academies to examine the key issues and implications for defense testing from the introduction of evolutionary acquisition. The CNSTAT was charged with planning and conducting a workshop to study test strategies for the evolutionary acquisition. The committee reviewed defense materials defining evolutionary acquisition and interviewed test officials from the three major test service agencies to understand the current approaches used in testing systems procured through evolutionary acquisition. The committee also examined possible alternatives to identify problems in implementation. At the workshop that took place on December 13-14, 2004, the committee tried to answer many questions including: What are the appropriate roles and objectives for testing in an evolutionary environment?, Can a systematic, disciplined process be developed for testing and evaluation in such a fluid and flexible environment?, and Is there adequate technical expertise within the acquisition community to fully exploit data gathered from previous stages to effectively combine information from various sources for test design and analysis?. Testing of Defense Systems in an Evolutionary Acquisition Environment provides the conclusions and recommendations of the CNSTAT following the workshop and its other investigations.

During the past decade and a half, the National Research Council, through its Committee on National Statistics, has carried out a number of studies on the application of statistical methods to improve the testing and development of defense systems. These studies were intended to provide advice to the Department of Defense (DOD), which sponsored these studies. The previous studies have been concerned with the role of statistical methods in testing and evaluation, reliability practices, software methods, combining information, and evolutionary acquisition. Industrial Methods for the Effective Testing and Development of Defense Systems is the latest in a series of studies, and unlike earlier studies, this report identifies current engineering practices that have proved successful in industrial applications for system development and testing. This report explores how developmental and operational testing, modeling and simulation, and related techniques can improve the development and performance of defense systems, particularly techniques that have been shown to be effective in industrial applications and are likely to be useful in defense system development. In addition to the broad issues, the report identifies three specific topics for its focus: finding failure modes earlier, technology maturity, and use of all relevant information for operational assessments.