Test and evaluation (T&E) of Department of Defense (DoD) acquisition programs is essential to fielding a lethal, suitable, resilient, survivable, agile, and responsive Joint Force. T&E capitalizes on the knowledge of the operational environment (notionally depicted in Figure 1) and the latest advances in science and technology to inspire trust and confidence in our warfighting capabilities and help avoid operational failures in battle. In June 2022, the Director, Operational Test and Evaluation (DOT&E) delivered the Strategy Update 2022 to set a course for transformative changes to the T&E infrastructure, tools, processes, and workforce. Specifically, the strategy details the path forward to continue to enable accurate characterization of the operational performance and limitations of the DoD to Prevail in Conflict and Defend the Homeland.
Additionally, the strategy emphasizes the need to implement the DoD Data Management Strategy and to accelerate delivery of T&E data to acquisition decision-makers to Build Enduring Advantage. It also focuses on increasing the DoD’s ability to illuminate and help mitigate vulnerabilities at scale, frequency, and depth to improve survivability in a contested environment and build Defense/Resilience. And it aligns with the DoD Responsible Artificial Intelligence Strategy and Implementation Pathway, emphasizing the need for continuous evaluation of the operational and ethical performance of software-reliant systems designed to change over time. Lastly, the DOT&E Strategy Update 2022 focuses on fostering an agile and enduring T&E enterprise workforce to enable the DoD’s Readiness and Training. This article summarizes the intent of the strategy and focuses on those lines of efforts that could benefit the aircraft survivability community.
STRATEGIC DRIVERS
Existing T&E infrastructure, tools, and processes have largely been rooted in 20th century frameworks focused on the acquisition of hardware and built around a contested environment that is primarily driven by kinetic threat effects. The era of information, advanced algorithms, and high-performance computing is driving us to transform T&E to a data-centric enterprise optimized for the acquisition of largely software-reliant systems that need to be survivable in a multi-domain operating environment comprising both kinetic and nonkinetic threats.
For example, modern software engineering introduces never-before-seen capabilities and vulnerabilities that change at never-before-seen dynamic rates. This includes artificial intelligence (AI) and machine-learning (ML) tools designed to generate, exchange, process, validate, fuse, and analyze vast amounts of data at machine speeds. Machine-speed warfare, in turn, integrated across all combat domains and environments, requires intent focus on the T&E of mission threads that make up the system-of-systems and the exponentially larger attack surface. The sheer volume of systems, and their extensive reliance on each other to form effective kill webs, will require tools, processes, infrastructure, and a workforce that facilitate continuous and automated T&E, resulting in the ultimate strategic driver—the “speed-to-field.”
STRATEGIC PILLARS
The DOT&E Strategy Update 2022 defines five strategic pillars to deliver on the Director’s intent to transform T&E and enable the delivery of the world’s most capable warfighting capability. Table 1 summarizes these pillars, their associated high-level key actions, and the desired end state.
Table 1. Strategic Pillars of T&E Transformation and Capability Delivery
Pillar 1: Test the Way We Fight
The first pillar focuses on two lines of effort (LOE). The first LOE centers on the operational environment, a key prerequisite of operational and Live Fire Test and Evaluation (LFT&E) that could also be leveraged for experimentation, training, and mission rehearsals. The second LOE centers on the operational performance of the Joint Force that intentionally expands on the T&E requirements tailored to evaluating individual acquisition program to focus on the evaluation of mission and mission threads, comprising multiple systems working together to achieve the desired effect. Figure 2 notionally illustrates the importance of interoperability and the array of factors that could affect mission success.
LOE 1.1: Standardize the Development of a Scalable and Adaptive Representation of the Joint, Multi-Domain Operating Environment
Why This Matters: Accurate evaluation of next-generation warfighting capabilities requires an adequate representation of the theater-representative operating environment during test, training, and mission rehearsal. It also requires equipment, both physical and digital, that can adequately measure technical and operational performance of emerging or fielded warfighting capabilities in that environment. The DoD has an array of test and training ranges and capabilities managed, funded, and operated by different stakeholders. To enable efficient and structured modernization and sustainment of existing range capabilities while also transforming the ranges to meet the demands of the future, it is important to have an accurate and common picture of existing and required future range capabilities. It will be equally important to ensure this common picture is accurate, digitized, and transparent to key T&E stakeholders to enable collaboration in developing joint/interoperable solutions, avoiding redundancies, and increasing capability delivery and efficiencies. Defining the requirements for and developing an accurate representation of the future operating environment will be critical for an evaluation of aircraft survivability and air-superiority in multi-domain operations.
LOE 1.2: Implement Measures, Tools, and Processes to Efficiently Evaluate Mission Threads, Kill Webs, and System-of-Systems Performance
Why This Matters: Real-world mission scenarios involve the use of multiple systems of varying complexities and pedigrees working together to achieve the desired lethal effect. The emergence of highly network-centric concepts, greater dependency on connectivity, and the use of large amounts of data from a wide array of shooters and sensors across multiple domains, at machine speeds, warrants a review of our T&E processes within individual acquisition programs. Evaluating warfighting capability is further challenged by asynchronous updates and continuous evolution of the various components that comprise these system-of-systems operations. These evolutions demonstrate an inherent need to continually characterize the interoperability of such systems and their effectiveness as would be employed by the Combatant Commands. With the emergence of joint all-domain command-and-control solutions and the concept of kill webs, it is important to define the process and the required T&E tools that would effectively measure the success rates of mission threads, concepts, and solutions. For example, an array of different types of aircraft could be tasked to support such complex missions, requiring a well-thought-out process for evaluating each aircraft’s contribution to that mission’s success and any relevant mission-critical vulnerabilities that would have to be accounted for in aircraft design modifications, mission planning, or execution.
Pillar 2: Accelerate the Delivery of Weapons That Work
The second pillar focuses on embracing digital technologies and solutions to introduce efficiencies in T&E, plans, execution, and analyses by way of two related LOEs: (1) improving the way we store and access T&E data so we can then increase their usage by (2) developing tools and methods to increase our analytical and inference capabilities.
LOE 2.1: Develop and Implement an Enterprise-Level T&E Data Management Solution
Why This Matters: Data are strategic assets that fuel automation and algorithms designed to alleviate our workload, speed up our processes, help us achieve new insights, and achieve T&E at scale and speed. As data-driven and complex systems continue to proliferate, it is important to develop T&E data and interface standards, stores, and platforms to ensure that the data are credible, trustworthy, available, and secure across the T&E enterprise. The T&E community must demonstrate its compliance with and contribution to the DoD Data Management Strategy and enable research, development, test, and evaluation (RDT&E) data collection, storage, visibility, sharing, accessibility, ingestion, and security across commercial and Government stakeholders to expedite data analysis, optimize T&E planning and execution, and enable more automated T&E. This compliance translates to availability of data stores, knowledge management tools, and data fusion/analytics tools that will enable the new fluid and iterative nature of T&E demanded by software-and data-reliant systems. Data are also critical to verify and validate digital tools. Lastly, all data (contractor, developmental, operational, and Live Fire) must be effectively leveraged to adapt, inform, and optimize T&E plans—no data should be left on the table.
For example, aircraft survivability evaluations rely heavily on contractor, developmental, operational, and Live Fire test data, as well as modeling and simulation (M&S) outputs, to support the evaluation of aircraft susceptibility, vulnerability, force protection, and recoverability, including the evaluation of residual mission capability. Adequate evaluation of aircraft survivability hinges on these various sources of data—collected throughout the acquisition program as the system matures through its development process—requiring a capability not only to capture and store those data but also to credibly piece those different data elements together in a meaningful way.
LOE 2.2: Integrate T&E in Model-Based Engineering to Operationalize and Optimize the Shift-Left Approach
Why This Matters: Modern model-based engineering and adaptive inference processes offer integrated, holistic approaches to generating and managing knowledge of system performance throughout the life cycle. Early test data from system components, for example, can be integrated into a larger system model to predict mission-level performance early in development. Advanced performance inference techniques (e.g., Bayesian or similar) can be used to carry forward data from early prototypes through evaluation of production-representative systems. Moreover, model-based engineering can eliminate manual workflows through automation that enables generation and distribution of up-to-date dynamic reports on systems and their status in the acquisition life cycle.
The aircraft survivability community will have to leverage heavily model-based system engineering and other digital tools and technologies to enable full spectrum survivability evaluations, as required by the Fiscal Year 2022 National Defense Authorization Act (NDAA), Section 223. Full-spectrum aircraft survivability evaluations are intended to enable the survivability of the aircraft in a multi-domain operational environment, accounting for both kinetic and nonkinetic threats—such as cyber; directed energy weapons (DEW); electromagnetic spectrum (EMS) fires; chemical, biological, radiological, and nuclear (CBRN) threats; and any combination thereof. Moreover, full-spectrum survivability evaluation is intended to leverage digital technologies required to enable such evaluation throughout the life cycle of the acquisition program, as both the fielded system and the threat(s) evolve over time at more dynamic rates.
Pillar 3: Improve the Survivability of the DoD in a Contested Environment
The third pillar was brought in as its own pillar specifically to focus on LFT&E and survivability-related challenges and the importance of evaluating synergistic kinetic and nonkinetic effects. In short, the same amount of rigor must be applied to survivability evaluation against cyber, EMS, and similar threats as has been historically applied to kinetic threat effects. While this requirement/resolution may appear to be simple, the dynamic nature of these increasingly advanced threats and complexity of their potential effects make them much more involved and challenging than traditional kinetic threats.
LOE 3.1: Standardize and Automate Mission-Based Risk Assessments
Why This Matters: Seamless integration of various systems and technologies working together across multiple domains introduces a potential for vulnerabilities that cannot be evaluated one system or one threat at a time. As discussed under the first pillar, testing must consider the mission thread, specifically the composition of weapon systems; networks; critical infrastructure; equipment; and tactics, techniques, and procedures (TTPs). A rapid and accurate mission-based survivability assessment would define specific steps to enhance mission assurance and identify the defenses required against threats to those missions.
While there are several ways to potentially do this type of assessment, one example of a mission-based risk assessment design is shown in Figure 3. The approach is based on the Leveson and Thomas System-Theoretic Process Analysis framework, which is composed of (1) scope of analysis, (2) model of control, (3) identification of failures, and (4) identification of loss scenarios. This mission-based risk assessment process is directly applicable to aircraft survivability evaluation. However, bounding the scope of the assessment is essential to establishing a reliable process. The mission-based risk assessment would also have to be conducted iteratively during the life cycle of the system. In addition, this process requires early engagement, starting with the development of request for proposals and other contractual requirements, so mission engineering artifacts (including models of the control and other functional models needed to define mission-critical functions and establish unacceptable losses) are readily available to the operational and LFT&E community for an adequate mission-based risk assessment.
LOE 3.2: Emphasize Cyber and EMS Survivability
Why This Matters: The weapon systems of today and the future are defined by both software and hardware. Battle networks are central to the kill web, and information technology is at the heart of cyber, space, and EMS warfare. The complex interactions between software and hardware can sometimes be difficult to predict or evaluate. Our challenge is to evaluate cyber-physical systems against advanced cyber and EMS threats at scale and speed. Attack surfaces are growing exponentially, reaching into supply chains, software factories, and pipelines; the EMS; and an array of cloud solutions. We therefore must aggressively pursue verified and validated digital tools and transformative technologies to manage cyber, EMS, and advanced kinetic threat survivability T&E and assess the effectiveness of countermeasures and other self-defense solutions.
Figure 4 shows a notional overarching framework for evaluating full-spectrum survivability effects, emphasizing cyber and EMS survivability within the traditional survivability evaluation approaches.
As illustrated in the figure, in step 1, the system under test (SUT) intelligence inputs are fed into a suite of effects M&S tools within an overarching full-spectrum effects M&S framework (the center portion of the figure). Kinetic, cyber, EMS, directed energy, and CBRN M&S tools will each characterize and visualize the system; identify and prioritize vulnerabilities; plan attacks (leveraging threat intelligence data) and probabilistically analyze results; and assess functional defeat and mission impact.
In step 2, adversary objectives and SUT survivability requirements are translated into functional defeat metrics (the top portion of Figure 4). Mission planning is carried out to develop an overall attack plan, comprising a combination or sequence of different domain attacks on the SUT (e.g., a cyber attack followed by an EMS attack followed by a kinetic attack). For step 3, the mission planning engine appropriately defines attack inputs to each of the individual effects M&S tools, which then execute the attacks and compute SUT functional defeat. Moving to step 4, individual functional defeat results are fed back into the mission planning engine, where adjustments to the coordinated attack plan can be made; and in step 5, this iterative process continues until overall functional defeat is optimized with respect to the prescribed SUT functional defeat metrics.
Finally, in step 6, full spectrum effects M&S shares a symbiotic relationship with T&E activities throughout the system life cycle. In one direction, the M&S framework can be used to augment (or potentially replace) T&E activities, which may be cost-prohibitive or even untestable if full spectrum effects T&E infrastructure is not in place. In the other direction, data collected through T&E can be fed back into M&S framework activities to improve model fidelity and simulation accuracy.
LOE 3.3: Evaluate Operational Performance in a Contested Space Environment
Why This Matters: Space is increasingly congested and highly contested, with a broad array of rapidly evolving threats. Reliance on space-based capabilities has sharpened the DoD’s—and our adversaries’—focus on deploying both offensive and defensive weapons in space. Because the DoD must operate in this contested environment, the T&E enterprise must be ready to accurately evaluate space-based and space-dependent systems’ operational performance, including survivability against current and anticipated threats. Furthermore, for aircraft survivability evaluations, additional focus needs to be placed on mission-critical functions that could affect aircraft survivability and air superiority.
Pillar 4: Pioneer the T&E of Weapon Systems Built to Change Over Time
The fourth pillar is focused on addressing the T&E challenges associated with complex, largely software-reliant systems, the operational performance of which could be affected by incremental and frequent software upgrades and/or frequent and dynamic changes to the operating environment. Related to software-reliant systems, this pillar also focuses on the challenges brought by AI and ML capabilities. All elements of this pillar are counting on advances of the digital ecosystems, which start with the development of credible digital twins.
LOE 4.1: Increase the Use of Credible Digital Twins in T&E
Why This Matters: The combination of new domains and operational constraints makes verified and validated digital technologies the necessary, practical approach for development and T&E of certain systems where live T&E is not possible or practical. For example, digital twins that can be subjected to repeated cyber attacks—as the system itself, the threats it will face, and adversary TTPs and procedures change over time—will help developers and program managers improve system cyber survivability at an increased pace. A digital twin is a high-fidelity digital representation of a physical object. These types of models allow us to find out how real-world objects might behave under different conditions or requirements. The defining feature of a digital twin is the ongoing data integration between the digital model and its physical unit counterpart. Digital twins have begun to incorporate transmission of real-time data sensed by the real-world object. These new, higher-resolution sensor data allow the digital twin to reason about future behaviors, then transmit feedback to the physical object. This ability could be particularly useful in enabling
continuous monitoring of operational performance of systems as they evolve over time. Unfortunately, while digital twins create new opportunities for T&E to determine the performance of continuously evolving systems, they also create new verification, validation, and accreditation challenges. Moreover, for the purposes of aircraft survivability evaluation, additional work is required to determine, for example, what is considered an adequate digital twin of an aircraft.
LOE 4.2: Evaluate the Operational and Ethical Performance of AI-Based Systems
Why This Matters: AI-based systems have accelerated the need to re-engineer T&E to enable continuous assessment once fielded. The T&E enterprise must monitor and evaluate the drift in deployed AI models’ behavior, which could occur when real-world data deviate from the training data used to create the model. Testing also must demonstrate with confidence that AI-based systems are responsible, ethical, equitable, traceable, reliable, and governable. Ethical and safe use of AI is necessary to reduce risks to U.S. strategic initiatives, reputation, operations, legal standing, and privacy issues. Due to their reliance on ever-changing data, however, AI-based systems are uncertain by nature. Emerging approaches that have the potential to address such uncertainty propagation deserve further investigation. Additional research is also needed to re-envision the T&E process with increased AI and automation tools to support, for example, aircraft survivability T&E professionals and identify opportunities where AI can assist them—relieving them of tedious tasks so they can better focus on tasks that require the creativity and innovation that only humans can provide.
LOE 4.3: Advance the Evaluation of Software-Reliant Systems’ Operational Performance
Why This Matters: Modern warfighting systems are increasingly software-reliant. They are developed through complex software pipelines filled with a myriad of tools intended to ensure automatically that the product is effective and secure. Yet, developers more and more frequently use open-source and third-party software, which raises risk from the security and sustainability perspectives. It is important to identify new approaches to address change propagation within software-reliant systems. For example, the aircraft survivability community needs to influence and measure the development and cyber defense of software pipelines and factories up front with accredited tools, techniques, and procedures. Automated testing should be embraced at every level, and a rigorous standard of testing should continue to continue to be implemented at the speed of relevance.
Pillar 5: Foster an Agile and Enduring T&E Enterprise Workforce
The fifth pillar is focused on preparing the T&E community, including the aircraft survivability community, for the challenges of the future. On one hand, we need to clearly define what we need in terms of capacity and skillsets and how to measures and track our readiness. On the other hand, we need a feasible continuous learning and a recruitment strategy that will allow us to prepare, retain, and augment our existing workforce while also being able to reach out to on-demand experts, as needed.
LOE 5.1: Identify and Track T&E Workforce Competencies and Capabilities
Why This Matters: A structured approach for the collective development and sustainment of the T&E enterprise workforce will enhance workforce agility and response to emerging T&E requirements. Dedicated T&E skill codes and qualifiers to track T&E professionals’ knowledge, skills, and abilities would improve the Department’s awareness of the T&E workforce’s overall health and development. An infrastructure to make data-driven workforce planning decisions would enable the T&E enterprise to forecast, track, and address gaps in the T&E workforce’s collective capabilities. It would also enable unified development of the T&E enterprise workforce, as well as its agility to move among the requirements developers, technology developers, buyers, and across the Service T&E communities. The requirements for the aircraft survivability community must also be accounted for in this approach.
LOE 5.2: Assess and Address Critical T&E Workforce Professional Development Needs
Why This Matters: T&E professionals of the future require access, bandwidth, and clear requirements to engage in continuous learning opportunities. Providing these tools/needs will better prepare them for advances in T&E operational and technical capabilities needed to perform their duties. It is important to establish enterprise-wide baseline education and training needs and the ability to identify all T&E-related course offerings to strengthen workforce capabilities. The T&E learning apparatus should change as quickly as the T&E operating environment, resulting in easily adaptable courses, content, and training and workforce demands. It is also important to establish a continuum of cutting-edge learning opportunities that can tie training and education to specific job and career outcomes across the enterprise; this establishment will improve and incentivize T&E learning and workforce retention. To compete with private sector organizations for top-tier talent and promote retention, the T&E enterprise will need to invest in workforce experiences that appeal to a diverse T&E workforce in terms of skill development, rotational opportunities, and leadership roles.
DOT&E STRATEGY IMPLEMENTATION PLAN
Requirements, intelligence, and the acquisition pathways drive the T&E process. Changes in capabilities, such as kill webs, complex all-domain environments, and gaps newly identified by intelligence reports, will steer acquisition decisions and commensurate T&E responses. Based on the requirements, intelligence, and mandates sourced from the six acquisition pathways, the T&E community will collaborate to identify and develop the T&E capabilities necessary to test and evaluate systems in the acquisition pipeline. These T&E activities will realize the goals of the five aforementioned strategic pillars that will in turn inform T&E policy and guidance with the potential to inform operational and system requirements, system development, and acquisition contracts.
The operating model depicted in Figure 5 highlights the significance of collaboration across an array of stakeholders that drive and affect the future of the T&E enterprise. The model identifies layers of governance, as well as key stakeholders, to demonstrate how the T&E community can align on and execute against the five strategic pillars. It is imperative that we work together and promote a pioneering spirit, as well as a culture of continuous learning, agility, transparency, and co-ownership, to use our combined talents most effectively. The operating model promotes the establishment of interdisciplinary teams of experts to forge and accelerate research and development needed to transform T&E tools, processes, infrastructure, and human capital.
MOBILIZE TO ACTION
The Department faces a shifting threat landscape and the need to swiftly leverage advanced technologies to increase the lethality, suitability, resiliency, survivability, agility, and responsiveness of our future Joint Force. To continue to deliver credible warfighting capability at the speed of need, the T&E enterprise must rethink the way we do business. We must become more agile, efficient, and effective to adequately account for the technology disruptors as we face an inflection point in the scope, scalability, and capabilities of our infrastructure, tools, processes, and workforce.
This DOT&E Strategy Update drives the T&E enterprise into the future through a series of coordinated enterprise-wide activities designed to achieve the future vision that is embedded in the strategy. It sets the framework to leverage ongoing government-based activities, as well as the best practices of industry, academia, and our allies to develop a future-ready T&E enterprise. Our immediate next step is to establish a governance charter and launch multi-disciplinary teams for each of the five pillars with a clear direction on what they need to accomplish, when, and why.
The T&E enterprise of the future must be agile, motivated by scenario/mission thread approaches, Joint warfighting concepts, and the power of digital tools and technologies. It will be strengthened by the effect of these changes on our ability to support the Warfighter. It will be empowered by continuous learning and supported by abundant access to state-of-the art skills and technologies.
By implementing this strategy, the T&E enterprise will stay ahead of the adversary and continue to advocate for the Warfighter and the mission, as defined by the National Defense Strategy 2022. We welcome you to join us in this important journey.
ABOUT THE AUTHORS
Dr. Sandra Hobson is the Deputy Director, Operational Test and Evaluation for Strategic Initiatives, Policy and Emerging Technologies. She holds B.S. and Ph.D. degrees in aerospace engineering from the U.S Naval Academy and the University of Maryland, respectively.
Mr. Paul Lowe is the Executive Officer for Director, Operational Test and Evaluation for Strategic Initiatives, Policy and Emerging Technologies. He holds B.S. and M.S. degrees in mechanical engineering from the University of Maryland Baltimore County and the Georgia Institute of Technology, respectively.
Mr. Nilo Thomas is the Software and Cyber Advisor for Director, Operational Test and Evaluation for Strategic Initiatives, Policy and Emerging Technologies. He holds a a B.S. in aerospace engineering from New Mexico State University
Dr. Jeremy Werner is Science Advisor to the Director, Operational Test and Evaluation for Strategic Initiatives, Policy and Emerging Technologies. He holds B.S. and Ph.D. degrees in physics from the University of California, Los Angeles, and Princeton University, respectively.
Mr. Garry Bishop is the Deputy Director, Operational Test and Evaluation for Land and Expeditionary Warfare. A retired Army colonel, he holds a B.S. degree in engineering from the U.S. Military Academy, a master’s degree in industrial engineering from New Mexico State University, and a master’s degree in national security strategies from the National Defense University.
