Verification and Validation with Innoslate

Verification and Validation (V&V)
with Innoslate®
STEVEN H. DAM, PH.D., ESEP
PRESIDENT, SPEC INNOVATIONS
571-485-7805
STEVEN.DAM@SPECINNOVATIONS.COM
1

Overview
1. V&V in the Early Stages of the Lifecycle?
2. Preparing for V&V
3. Applying LML to V&V
4. What’s a Verifiable Requirement?
5. Developing a V&V Plan Using MBSE
2

1. V&V IN THE EARLY STAGES OF THE LIFECYCLE?
3

What is V&V?
•Verification: “the process of establishing the truth, accuracy, or
validity of something” [Google definition]
•Validation: “the action of checking or proving the validity or
accuracy of something” [Google definition]
4
In USG Work these have taken on more
specific meanings

What Is Verification?
•Meets specification-level requirements
•Proves that the solution-dependent requirements are satisfied
•Proves that solution was or is being built according to agreed
requirements
5
Frequently the terms “Unit Test” and
“Developmental Test and Evaluation” apply to
verification

What Is Validation?
•Relates to customer and other stakeholder satisfaction
•Proves solution-independent requirements are being
satisfied
•Usually performed at the enterprise and/or system level
•In process validation helps ensure system will ultimately be
part of the accepted solution, in the target environment
6
Frequently the term “Operational Test
and Evaluation” applies to validation

The Lifecycle
7
Architecture
Development
System
Design
Hardware/Software
Acquisition
Integration
and Test
Operational
T&E and
Transition
Future Operations
and Maintenance
Demolition
and Disposal
Program
Management
Current Operations
and Maintenance

Requirements
Analysis
Functional Analysis
and Allocation
Synthesis
System Analysis and
Control
Best Use:
Architecture
Development
(To-Be)
Systems Engineering During Design Phase
8
Best Use:
“Classical SE”
Best Use: Reverse
Engineering (As-Is)
Adapted from EIA-632
• V&V lives in Systems Analysis
and Control if we use the
MIL-STD-499B process
• We prefer another process
for that phase

Coming Up the Vee
9
I&V Planning
Integration
Verification &
Validation
DesignMonitoring

What Do We Need to Do Early?
11
Architecture
Development
System
Design
Hardware/Software
Acquisition
Integration
and Test
Operational
T&E and
Transition
Future Operations
and Maintenance
Demolition
and Disposal
Program
Management
Current Operations
and Maintenance
?
?
?

What Do We Need?
•Ontology that includes V&V information
•Verifiable Requirements
•V&V Plan
12
To do any of these things you
need to understand how to
prepare for V&V, conduct V&V
activities, and report V&V results

LML’s Simplified Schema
• Action
• Artifact
• Asset
o Resource
• Characteristic
o Measure
• Connection
o Conduit
o Logical
• Cost
•Decision
•Input/Output
•Location
o Physical, Orbital, Virtual
•Risk
•Statement
o Requirement
•Time
14
Supports capturing information
throughout the lifecycle

V&V Extension to LML
•In developing Innoslate’s new Test Center capability, we discovered
the need to add a subclass to Action: Test Case
•You may discover that you want to add other classes, attributes, and
relationships
•It’s easy to do so using the Schema Editor, but recognize that you
may not have readily available views and reports
15

Test Case (Subclass of Action)
•A Test Case entity represents a set of conditions utilized
to determine whether a system works as intended
•Properties (Attributes)
16
Name Type Description
Status ENUMERATION
Status represents the current state of this Test
Case.
Actual
Result
BIG_TEXT
Actual Results represents the results that are
captured after executing a Test Case.
Expected
Result
BIG_TEXT
Expected Result represents the result expected
when executing this Test Case.
Set Up BIG_TEXT
Set Up represents the prerequisites that must be
fulfilled before executing this Test Case.

4. WHAT’S A VERIFIABLE REQUIREMENT?
17

Requirements Analysis
18
Source
Documents
External Interface
Database
User Needs
Decompose Requirements
Critical Issue?
Requirement
Verifiable?
Determine Options and Perform
Trade Studies
See System Analysis and
Control for details
Resolve Issues with Customer
YES
NO
Coordinate Changes to Make
Requirement Verifiable
NO
Review Statements and Risks
with Customer
Update Knowledgebase
YES
Identify Risks and Plan Mitigation
Updated
Requirements
Traceability Matrix
Preliminary Test
Requirements
Standards Selected
Change
Requests
SE
Knowledgebase
Trace Requirements to
Actions/Assets/Measures

Characteristics of Good Requirements
• Each individual requirement should be:
o Correct: Describes the user’s true intent and is legally possible
o Complete: Express a whole idea
o Clear: Unambiguous and not confusing
o Consistent: Not in conflict with other requirements
o Verifiable: Provable (within realistic cost and schedule) that the system meets the requirement
o Traceable: Uniquely identified, and able to be tracked to predecessor and successor lifecycle
items/objects
o Feasible: Able to be implemented with existing technology, and within cost and schedule
o Modular: Can be changed without excessive impact on other requirements
o Design: Does not impose a specific solution on design; says “what”, Independent not “how”
19

What Makes a Requirement Verifiable?
•Need to answer the following questions:
o Can the requirement be met?
 By analysis, demonstration, inspection, modeling & simulation, or test?
o Will the user accept the results?
o Avoid words such as “excessive,” “sufficient,” “resistant,” etc.
 Make the requirement quantifiable
o Even “suitability” can be quantified
20

Requirements Demo
•Using the FireSAT
sample project we will
show
• Analyzed requirements
quality
• Derived verification
requirements
21
Source
Documents
External
Interface
Database
User Needs
Decompose Statements
Critical
Issue?
Statement
Verifiable?
Determine Options and
Perform Trade Studies
See System
Analysis and
Control for details
Resolve Issues with
Customer
YES
NO
Coordinate Changes to
Make Statement Verifiable
NO
Review Statements and
Risks with Customer
Update Knowledgebase
YES
Identify Risks and Plan
Mitigation
Updated
Requirements
Traceability Matrix
Preliminary
Test
Requirements
Standards
Selected
Change
Requests
SE
Knowledgebase

5. DEVELOPING A V&V PLAN USING MBSE
22

Verification Process
23
SE Knowledgebase
Review Test Results with ERB
Develop Deployment Plan and
Conduct Beta Testing
Assess Risk and Plan Mitigation
Updated SE
Knowledgebase
Return Item to “Integrate Solution”
(if necessary)
Beta Test
Report
Deficiencies?
YES
NO
Workaround
Identified
Verify to Requirements
Verification
Requirements
From Design &
Analysis Phase
Perform Regression Testing
Acceptable
Regression Tests?
YES
NO
Deployment Plan
Submit Changes to CM Process
Regression Test
Report
Inspection and Test
Report
Updated Test
Plan

Create a Test Plan
24
1 Select Test
Plans from
Menu
2 Answer questions (use our template or create your own)
3 Fill-in
resulting
template

Test Planning Demo I
SE Knowledgebase
Review Test Results with ERB
Develop Deployment Plan and
Conduct Beta Testing
Assess Risk and Plan Mitigation
Updated SE
Knowledgebase
Return Item to “Integrate Solution”
(if necessary)
Beta Test
Report
Deficiencies?
YES
NO
Workaround
Identified
Verify to Requirements
Verification
Requirements
From Design &
Analysis Phase
Perform Regression Testing
Acceptable
Regression Tests?
YES
NO
Deployment Plan
SubmitChanges to CM Process
Regression Test
Report
Inspection and Test
Report
Updated Test
Plan
• Show Test Plan in
Documents View
• Discuss techniques
used to complete the
test plan
• Show relationships
between test plan
elements and the
design
25

Create a Test Process or Case
26
• Use
Action
Diagram
to create
process
• Link test
process to
test plan

Make Sure the Test Case Works Using Simulation
27
Explore the variation of individual steps to execute the model over
many iterations using the Monte Carlo Simulator
Watch test case execute at all levels using
the Discrete Event Simulator

Test Planning Demo II
•Show test cases
•Built model using
Action Diagram
•Run simulators to
verify models
•Show traced test
cases back to
verification
requirements
28
SE
Knowledgebase
Develop/Revise Context
Diagram
Determine Options and
Perform Trade Studies
See System Analysis and
Control for details
Review Model and Risks with
Customer
Identify Risks and Plan
Mitigation
Updated SE
Knowledgebase
Develop Series of Scenarios
for Analysis
Create/Update System
Behavior Model
Analyze Behavior Model
Performance
Behavior Model
• Control Flow
• Data Flow (Activity Model)
• Performance Criteria
Allocate Actions to Assets and
Input/Outputs to Conduits
Updated SE
Knowledgebase
Detailed
Operational
Concept
Operational
Requirements
Document
(ORD)

Use the Test Center
•Capture as part of test plan
•Link back to requirements
29

Other Testing Activities
• Test Equipment and Facilities (Assets)
• Test Organization (Assets) and Roles/Responsibilities (Statements)
• Risks and mitigations (Many)
• Criteria (Characteristics/Measures)
• Schedule (Timeline and/or process simulation output)
• Cost (WBS linked to processes)
• Data capture (actual Measures and Times)
• Artifacts
• Location
• Decisions (assumptions, issues, etc.)
30

What Did We Cover?
1. V&V in the Early Stages of the Lifecycle?
2. Preparing for V&V
3. Applying LML to V&V
4. What’s a Verifiable Requirement?
5. Developing a V&V Plan Using MBSE
32

Verification and Validation with Innoslate

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Verification and Validation with Innoslate

Similar to Verification and Validation with Innoslate (20)

More from Elizabeth Steiner

More from Elizabeth Steiner (20)

Recently uploaded

Recently uploaded (20)

Verification and Validation with Innoslate