The 2021 National Defense Authorization Act (NDAA), now incorporated into the test and evaluation (T&E) chapter of Title 10, mandated that the Secretary of Defense “enable assessments of full spectrum survivability and lethality of each covered system with respect to kinetic and non-kinetic threats.” That mandate to assess covered systems, broadly defined in the NDAA to include “any warfighting capability that can degrade, disable, deceive, or destroy forces or missions,” was the impetus that led the Director, Operational Test and Evaluation (DOT&E) to develop the concept of the Mission-Based Risk Assessment (MBRA), a process for scoping and justifying full-spectrum survivability tests within the context of the risk to the mission.
The MBRA Process
A process for MBRA must address the cost and schedule implications of full-spectrum survivability testing. Conducting unbounded operational testing of all plausible threats in all possible combinations is simply not feasible within the budget, resource, and time constraints of any program. Further, the threats posed by adversary capabilities are constantly evolving, expanding the already intractable combinatorial test space over the program acquisitions process. Therefore, a means of bounding those combinations to only those that are likely to affect a system’s mission was needed. MBRA was developed to address this need.
The four major questions that MBRA attempts to answer are as follows:
- How might full-spectrum conditions affect system operation?
- What are the potential mission-critical functions and how could they be exploited by adversaries?
- How do these factors impact the system’s ability to complete its mission?
- What scope of operational testing will be required to verify survivability, suitability, and effectiveness?
Fundamentally, MBRA is meant to provide a defensible scope for full-spectrum survivability and lethality testing by providing information on mission risk from adversary threats, alone and in plausible combinations, to program stakeholders.
As currently envisioned in DOT&E policy, MBRAs would be managed by the program T&E working-level integrated product teams (WIPT), also known as integrated test teams (ITT), who would be responsible for planning, executing, and evaluating the result of MBRAs. The WIPT would plan one or more MBRAs over the course of a program’s acquisition life cycle, scoping each MBRA based on the available mission, system, and adversary threat information available.
The output of an MBRA is a recommended set of prioritized system test vignettes for full-spectrum T&E traceable to adversary threats, mission-critical functions, and system design elements. These recommendations would then be used by program stakeholders to populate, or revise, the program test and evaluation master plan (TEMP). While a DOT&E effort focused on supporting operational and live fire testing, MBRAs can also provide useful information to the program engineering effort, as risks identified early in the program’s acquisition life cycle can be used to harden the system against adversary threat combinations that endanger mission-critical functions (as illustrated in Figure 1).
The idea of conducting mission-based risk is not new, and DOT&E has built upon the foundations laid by earlier work, particularly from the cyber test community. Mission-based cyber risk assessments (MBCRA), such as cyber tabletops (CTT) and the Air Force Mission-based Risk Assessment Process for Cyber (MRAP-C), are well-established techniques for prioritizing testing based on evaluating mission risk from the intersection of adversary cyber threats, system vulnerabilities, and system component impacts on mission-critical functions. Further, MBCRAs are often conducted iteratively across the program acquisitions life cycle, providing both early low-fidelity insights into mission risk and later high-fidelity risk assessment as more information on the system design and implementation coalesce. MBRA can be seen as an evolution of these threat-area-specific (e.g., kinetic, cyber, electromagnetic spectrum operation, etc.) risk assessment processes to address threats across the full spectrum of adversary capabilities. The term “evolution” does not, however, necessarily imply replacement, and organizations already conducting MBCRAs or other domain-specific risk assessments could use those assessments as inputs to program MBRAs, minimizing program and organizational impact, through the program (as shown in Figure 2).
While the objectives and high-level policy mandates for conducting MBRAs in support of full-spectrum survivability and lethality T&E exist, a process for evaluating whether a risk assessment process is, or is not, an MBRA has not yet been precisely defined. It is thus difficult for programs and service test organizations to comply with policy, realize the intended benefits, or provide comparable data on the impacts to program execution. Accordingly, the remainder of this article is focused on the work of the DOT&E MBRA working group to define a set of evaluation criteria for MBRAs to mitigate this identified information gap.
Criteria to Close the Information Gap
Acknowledging the need for more concrete guidance on how to implement MBRA, either through new risk assessment processes or through modification of existing processes, the DOT&E Director for Strategic Initiatives, Policy, and Emerging Technology (SIPET) created a strategic objective under DOT&E Strategic Pillar Three, “Improve the survivability of DoD in a contested environment,” to “standardize and automate [MBRAs].” In support of this objective, a multidisciplinary working group was created, led by the Johns Hopkins University Applied Physics Laboratory (JHU APL) and consisting of subject-matter experts from both industry and Government. The primary goal of this working group is to derive a set of measurable criteria that Service test organizations can use to assess existing risk assessment processes for gaps or to create new risk assessment processes that both meet the objectives of MBRAs and fit within their specific program requirements.
Initial efforts by the working group have generated a set of preliminary measures grouped into six categories (as shown in Figure 3):
- Mission Analysis
- System Analysis
- Full-Spectrum Risk Analysis
- OT&E/LFT&E Scope and Design Recommendations
- Governance
- Data Management.
Mission Analysis
Mission Analysis focuses on those elements of the process related to defining mission-critical functions and unacceptable mission losses. The mission definition and mission characterization provided can be tied back to a larger Mission Engineering effort with documentation that Warfighter representatives and key stakeholders have approved. Mission measures and metrics should be tied together through a complete mission thread analysis. Unacceptable mission losses are determined using these mission threads and provide justification that the selected set of losses is correct and complete.
System Analysis
System Analysis ties system design elements to mission-critical functions and mission threads in line with the Mission Engineering Guide, version 2.0, published by the Office of the Under Secretary of Defense for Research and Engineering in 2023. System contribution to the mission is defined in a complete mission engineering thread analysis that Warfighter representatives and key stakeholders have approved. System behaviors contributing to mission losses are identified and crosswalk the mission thread and mission engineering thread analyses. System components and interfaces affected by threat effects are considered to streamline scenario analysis. Full-spectrum system risk scenarios map kinetic and nonkinetic threat effects to mission loss through documented risk scenarios.
Full-Spectrum Risk Analysis
Full-spectrum risk analysis encompasses the concrete risk assessment processes used, ranging from simple tabletop exercises to complex modeling and simulation of contested mission environments. Risk analysis considers kinetic and nonkinetic threat effects and/or system behaviors that could lead to mission loss. All five threat domains are considered separately and also in plausible combinations. This category includes identification and documentation of the effects that would disrupt mission success, as well as tools tailored to address missions and scenarios and program measures/dashboards.
OT&E/LFT&E Scope and Design Recommendations
OT&E/LFT&E scoping and recommendations address how risk assessment results are captured, reported, and tailored to specific stakeholder communities. Recommendations for OT&E and LFT&E scope and design should be presented to stakeholders that balance risk against cost, schedule, assumptions, and constraints.
Governance
Governance encapsulates and expands upon the responsibilities of the WIPT/ITT found in policy. The program’s individual MBRA process should be clearly defined and documented. The plans to conduct MBRAs, including plans for MBRAs to handle changes, should be integrated with project schedules. The Service-specific MBRA governing structure generally promotes consistency between programs.
Data Management
Data Management addresses data quality and availability requirements to support an MBRA effort. Data quality considers not only the precision and accuracy of data used to perform the MBRA but also the pedigree and descriptive metadata. Data availability addresses the access and approval for use of data to perform the MBRA. All these issues are in the context of the DoD Data Strategy’s VAULTIS principles.
Working Group Refinement
As preliminary individual criteria within each organizational category and their measures are defined, the working group has reached out to the broader service test community for comments and recommendations. While this effort has been spearheaded by DOT&E, the Service test organizations and the programs they oversee have a vested stakeholder interest in the criteria development process and hands-on knowledge from the programs they oversee. Their insight will thus help to shape the final set of criteria and measures.
Once the criteria and their measures are complete, they will be included in a DOT&E-published MBRA Guidebook, which will help clarify how to implement MBRAs within existing and future programs in compliance with policy and in support of the full-spectrum survivability and lethality testing requirements in Title 10. This guidebook will be a living document and after initial publication will be regularly updated based on feedback from programs and test organizations that implement MBRAs on real systems, as well as the results of gap analyses conducted against existing risk analysis process.
For further information on MBRA objectives, policy requirements, or the working group, please contact the SIPET Pillar Three lead at DOT&E, Mr. Tom Walrond, or any of the authors of this article.
About the Authors
Mr. Ryan Brunton is a member of the Senior Professional Staff at JHU APL. He has more than 20 years of experience in software design and implementation, computer security, open-source intelligence analysis, HLA-based simulation development, and enterprise software design and development. Mr. Brunton holds a bachelor’s degree in computer science from the University of California, San Diego and a master’s degree in homeland security from San Diego State University.
Ms. Lynne Donaldson is a Senior Systems and Systems Security Engineer at JHU APL. She has more than 35 years of experience in engineering, information and cyber security, policy implementation, and technical business administration. Ms. Donaldson holds a bachelor’s degree in systems engineering from JHU APL and in information security policy management from the University of Fairfax, as well as an MBA with a specialization in information security from the University of Rhode Island.
Mr. Jeffrey Painter is a career Operations Research Analyst with the National Security Analysis Department (NSAD) at JHU APL. He has contributed to the breadth of analysis across NSAD, including work with special operations modernization, contested deployment, and Joint experimentation. Mr. Painter holds a bachelor’s degree in mechanical engineering from the United States Military Academy and a master’s degree in operations research from the Air Force Institute of Technology.
Mr. Randy Saunders is a Principal Staff Analyst at JHU APL. He has more than 35 years of experience in the analysis, model-based design, implementation, and integration of high-fidelity simulations for military and government customers. Mr. Saunders holds a master’s degree in engineering from Harvey Mudd College and in computer science from the University of Southern California.
FAQs About MBRA
How is it MBRA different than other risk assessments?
MBRA offers a more dynamic, holistic, and system-oriented perspective, with consideration of the full spectrum threat and scenario challenges. Unlike conventional risk assessment methods that often focus on isolated components and stove-piped domains, MBRA assesses the tested system using multiple combinatory threat scenarios that challenge multiple attack surfaces simultaneously to determine total risk to mission.
Why do we need yet another process? This seems duplicative and adds little value. Isn’t the intent met by current mission and system engineering activities?
Decision support improvements that result from MBRA will provide decision-makers with actionable insights by identifying critical vulnerabilities, prioritizing risk mitigation strategies, and offering recommendations to enhance system design, testing, tactics, and policies. MBRA will help determine a defensible scope of operational and live-fire test and evaluation that complies with full-spectrum survivability testing requirements.
Won’t this add cost and time to the schedule? Is this an unfunded mandate?
For a program that is currently using mature processes for determining risk to operational missions, MBRA may have little impact to cost and schedule. This effort seeks to minimize impact to cost and schedule by providing a means to identify how current practices meet the intent of MBRA analysis and risk determination.
If design details are not available for my program, can I still conduct an MBRA?
MBRAs are meant to be applicable at any stage of the acquisitions life cycle, including early in the mission engineering process, when few design details are known. However, confidence in the assessed risk will increase with the quality and quantity of design and implementation data available.
