Trade-off Analytics
Creating and Exploring the System Tradespace
Wiley Series in Systems Engineering and Management Series

Presents information to create a trade-off analysis framework for use in government and commercial acquisition environments

This book presents a decision management process based on decision theory and cost analysis best practices aligned with the ISO/IEC 15288, the Systems Engineering Handbook, and the Systems Engineering Body of Knowledge. It provides a sound trade-off analysis framework to generate the tradespace and evaluate value and risk to support system decision-making throughout the life cycle. Trade-off analysis and risk analysis techniques are examined. The authors present an integrated value trade-off and risk analysis framework based on decision theory. These trade-off analysis concepts are illustrated in the different life cycle stages using multiple examples from defense and commercial domains.

• Provides techniques to identify and structure stakeholder objectives and creative, doable alternatives
• Presents the advantages and disadvantages of tradespace creation and exploration techniques for trade-off analysis of concepts, architectures, design, operations, and retirement
• Covers the sources of uncertainty in the system life cycle and examines how to identify, assess, and model uncertainty using probability
• Illustrates how to perform a trade-off analysis using the INCOSE Decision Management Process using both deterministic and probabilistic techniques 

Trade-off Analytics:  Creating and Exploring the System Tradespace is written for upper undergraduate students and graduate students studying systems design, systems engineering, industrial engineering and engineering management. This book also serves as a resource for practicing systems designers, systems engineers, project managers, and engineering managers.

Gregory S. Parnell, PhD, is a Research Professor in the Department of Industrial Engineering at the University of Arkansas. He is also a senior principal with Innovative Decisions, Inc., a decision and risk analysis firm and has served as Chairman of the Board. Dr. Parnell has published more than 100 papers and book chapters and was lead editor of Decision Making for Systems Engineering and Management, Wiley Series in Systems Engineering (2nd Ed, Wiley 2011) and lead author of the Handbook of Decision Analysis (Wiley 2013).  He is a fellow of INFORMS, the INCOSE, MORS, and the Society for Decision Professionals.

List of Contributors xix

About the Authors xxi

Foreword xxxi

Preface xxxiii

Acknowledgments xli

About the Companion Website xlv

1 Introduction to Trade-off Analysis 1
Gregory S. Parnell, Matthew Cilli, Azad M. Madni and Garry Roedler

1.1 Introduction 2

1.2 Trade-off Analyses Throughout the Life Cycle 3

1.3 Trade-off Analysis to Identify System Value 3

1.4 Trade-off Analysis to Identify System Uncertainties and Risks 6

1.5 Trade-off Analyses can Integrate Value and Risk Analysis 6

1.6 Trade-off Analysis in the Systems Engineering Decision Management Process 8

1.7 Trade-off Analysis Mistakes of Omission and Commission 9

1.7.1 Mistakes of Omission 12

1.7.2 Mistakes of Commission 15

1.7.3 Impacts of the Trade-Off Analysis Mistakes 18

1.8 Overview of the Book 20

1.8.1 Illustrative Examples and Techniques Used in the Book 24

1.9 Key Terms 24

1.10 Exercises 25

References 26

2 A Conceptual Framework and Mathematical Foundation for Trade-Off Analysis 29
Gregory S. Parnell, Azad M. Madni and Robert F. Bordley

2.1 Introduction 29

2.2 Trade-Off Analysis Terms 30

2.3 Influence Diagram of the Tradespace 31

2.3.1 Stakeholder Needs System Functions and Requirements 33

2.3.2 Objectives 33

2.3.3 System Alternatives 34

2.3.4 Uncertainty 36

2.3.5 Preferences and Evaluation of Alternatives 37

2.3.6 Resource Analysis 44

2.3.7 An Integrated Trade-Off Analyses 44

2.4 Tradespace Exploration 46

2.5 Summary 46

2.6 Key Words 47

2.7 Exercises 48

References 48

3 Quantifying Uncertainty 51
Robert F. Bordley

3.1 Sources of Uncertainty in Systems Engineering 51

3.2 The Rules of Probability and Human Intuition 52

3.3 Probability Distributions 56

3.3.1 Calculating Probabilities from Experiments 56

3.3.2 Calculating Complex Probabilities from Simpler Probabilities 58

3.3.3 Calculating Probabilities Using Parametric Distributions 59

3.3.4 Applications of Parametric Probability Distributions 62

3.4 Estimating Probabilities 66

3.4.1 Using Historical Data 66

3.4.2 Using Human Judgment 68

3.4.3 Biases in Judgment 70

3.5 Modeling Using Probability 72

3.5.1 Bayes Nets 72

3.5.2 Monte Carlo Simulation 75

3.5.3 Monte Carlo Simulation with Dependent Uncertainties 76

3.5.4 Monte Carlo Simulation with Partial Information on Output Values 77

3.5.5 Variations on Monte Carlo Simulation 78

3.5.6 Sensitivity Analysis 78

3.6 Summary 81

3.7 Key Terms 81

3.8 Exercises 83

References 86

4 Analyzing Resources 91
Edward A. Pohl, Simon R. Goerger and Kirk Michealson

4.1 Introduction 91

4.2 Resources 92

4.2.1 People 92

4.2.2 Facilities 95

4.2.3 Costs 95

4.2.4 Resource Space 99

4.3 Cost Analysis 99

4.3.1 Cost Estimation 102

4.3.2 Cost Estimation Techniques 108

4.3.3 Learning Curves 120

4.3.4 Net Present Value 125

4.3.5 Monte Carlo Simulation 130

4.3.6 Sensitivity Analysis 134

4.4 Affordability Analysis 135

4.4.1 Background 136

4.4.2 The Basics of Affordability Analysis Are Not Difficult 137

4.4.3 DoD Comparison of Cost Analysis and Affordability Analysis 138

4.4.4 Affordability Analysis Definitions 139

4.4.5 “Big A” Affordability Analysis Process Guide 141

4.5 Key Terms 147

4.6 Excercises 149

References 152

5 Understanding Decision Management 155
Matthew Cilli and Gregory S. Parnell

5.1 Introduction 155

5.2 Decision Process Context 156

5.3 Decision Process Activities 157

5.3.1 Frame Decision 159

5.3.2 Develop Objectives and Measures 163

5.3.3 Generate Creative Alternatives 171

5.3.4 Assess Alternatives via Deterministic Analysis 180

5.3.5 Synthesize Results 183

5.3.6 Develop Multidimensional Value Model 187

5.3.7 Identify Uncertainty and Conduct Probabilistic Analysis 190

5.3.8 Assess Impact of Uncertainty 192

5.3.9 Improve Alternatives 196

5.3.10 Communicating Trade-Offs 197

5.3.11 Present Recommendation and Implementation Plan 197

5.4 Summary 199

5.5 Key Terms 199

5.6 Exercises 200

References 201

6 Identifying Opportunities 203
Donna H. Rhodes and Simon R. Goerger

6.1 Introduction 203

6.2 Knowledge 205

6.2.1 Domain Knowledge 205

6.2.2 Technical Knowledge 205

6.2.3 Business Knowledge 205

6.2.4 Expert Knowledge 206

6.2.5 Stakeholder Knowledge 206

6.3 Decision Traps 207

6.4 Techniques 210

6.4.1 Interviews 210

6.4.2 Focus Groups 213

6.4.3 Surveys 215

6.5 Tools 219

6.5.1 Concept Map 219

6.5.2 System Boundary 220

6.5.3 Decision Hierarchy 220

6.5.4 Issues List 221

6.5.5 Vision Statement 221

6.5.6 Influence Diagram 222

6.5.7 Selecting Appropriate Tools and Techniques 223

6.6 Illustrative Examples 223

6.6.1 Commercial 223

6.6.2 Defense 226

6.7 Key Terms 228

6.8 Exercises 230

References 230

7 Identifying Objectives and Value Measures 233
Gregory S. Parnell and William D. Miller

7.1 Introduction 233

7.2 Value-Focused Thinking 234

7.2.1 Four Major VFT Ideas 235

7.2.2 Benefits of VFT 235

7.3 Shareholder and Stakeholder Value 236

7.3.1 Private Company Example 237

7.3.2 Government Agency Example 237

7.4 Challenges in Identifying Objectives 238

7.5 Identifying the Decision Objectives 239

7.5.1 Questions to Help Identify Decision Objectives 239

7.5.2 How to Get Answers to the Questions 240

7.6 The Financial or Cost Objective 241

7.6.1 Financial Objectives for Private Companies 241

7.6.2 Cost Objective for Public Organizations 242

7.7 Developing Value Measures 243

7.8 Structuring Multiple Objectives 243

7.8.1 Value Hierarchies 244

7.8.2 Techniques for Developing Value Hierarchies 245

7.8.3 Value Hierarchy Best Practices 247

7.8.4 Cautions about Cost and Risk Objectives 248

7.9 Illustrative Examples 248

7.9.1 Military Illustrative Example 248

7.9.2 Homeland Security Illustrative Example 250

7.10 Summary 250

7.11 Key Terms 252

7.12 Exercises 253

References 255

8 Developing and Evaluating Alternatives 257
C. Robert Kenley, Clifford Whitcomb and Gregory S. Parnell

8.1 Introduction 257

8.2 Overview of Decision-making Creativity and Teams 258

8.2.1 Approaches to Decision-Making 258

8.2.2 Cognitive Methods for Creating Alternatives 260

8.2.3 Key Concepts for Building and Operating Teams 260

8.3 Alternative Development Techniques 263

8.3.1 Structured Creativity Methods 263

8.3.2 Morphological Box 266

8.3.3 Pugh Method for Alternative Generation 270

8.3.4 TRIZ for Alternative Development 271

8.4 Assessment of Alternative Development Techniques 275

8.5 Alternative Evaluation Techniques 276

8.5.1 Decision-Theory-Based Approaches 276

8.5.2 Pugh Method for Alternative Evaluation 276

8.5.3 Axiomatic Approach to Design (AAD) 277

8.5.4 TRIZ for Alternative Evaluation 280

8.5.5 Design of Experiments (DOE) 280

8.5.6 Taguchi Approach 282

8.5.7 Quality Function Deployment (QFD) 283

8.5.8 Analytic Hierarchy Process AHP 287

8.6 Assessment of Alternative Evaluation Techniques 290

8.7 Key Terms 290

8.8 Exercises 290

References 293

9 An Integrated Model for Trade-Off Analysis 297
Alexander D. MacCalman, Gregory S. Parnell and Sam Savage

9.1 Introduction 297

9.2 Conceptual Design Example 298

9.3 Integrated Approach Influence Diagram 300

9.3.1 Decision Nodes 300

9.3.2 Uncertainty Nodes 303

9.3.3 Constant Node 310

9.3.4 Value Nodes 314

9.4 Other Types of Trade-Off Analysis 322

9.5 Simulation Tools 322

9.5.1 Monte Carlo Simulation Proprietary Add-Ins 324

9.5.2 The Discipline of Probability Management 324

9.5.3 SIPmath^TM Tool in Native Excel 324

9.5.4 Model Building Steps 325

9.6 Summary 329

9.7 Key Terms 330

9.8 Exercises 331

References 335

10 Exploring Concept Trade-Offs 337
Azad M. Madni and Adam M. Ross

10.1 Introduction 337

10.1.1 Key Concepts Concept Trade-Offs and Concept Exploration 341

10.2 Defining the Concept Space and System Concept of Operations 345

10.3 Exploring the Concept Space 346

10.3.1 Storytelling-Enabled Tradespace Exploration 346

10.3.2 Decisions and Outcomes 347

10.3.3 Contingent Decision-Making 347

10.4 Trade-off Analysis Frameworks 348

10.5 Tradespace and System Design Life Cycle 349

10.6 From Point Trade-offs to Tradespace Exploration 351

10.7 Value-based Multiattribute Tradespace Analysis 351

10.7.1 Tradespace Exploration and Sensitivity Analysis 353

10.7.2 Tradespace Exploration and Uncertainty 354

10.7.3 Tradespace Exploration with Spiral Development 356

10.7.4 Tradespace Exploration in Relation to Optimization and Decision Theory 356

10.8 Illustrative Example 359

10.8.1 Step 1: Determine Key Decision-Makers 359

10.8.2 Step 2: Scope and Bound the Mission 360

10.8.3 Step 3: Elicit Attributes and Utilities (Preference Capture) 360

10.8.4 Step 4: Define Design Vector Elements (Concept Generation) 362

10.8.5 Step 5: Develop Model(s) (Evaluation) 362

10.8.6 Step 6: Generate the Tradespace (Computation) 364

10.8.7 Step 7: Explore the Tradespace (Analysis and Synthesis) 365

10.9 Conclusions 369

10.10 Key Terms 371

10.11 Exercises 372

References 372

11 Architecture Evaluation Framework 377
James N. Martin

11.1 Introduction 377

11.1.1 Architecture in the Decision Space 378

11.1.2 Architecture Evaluation 379

11.1.3 Architecture Views and Viewpoints 380

11.1.4 Stakeholders 382

11.1.5 Stakeholder Concerns 382

11.1.6 Architecture versus Design 383

11.1.7 On the Uses of Architecture 384

11.1.8 Standardizing on an Architecture Evaluation Strategy 384

11.2 Key Considerations in Evaluating Architectures 385

11.2.1 Plan-Driven Evaluation Effort 386

11.2.2 Objectives-Driven Evaluation 387

11.2.3 Assessment versus Analysis 387

11.3 Architecture Evaluation Elements 389

11.3.1 Architecture Evaluation Approach 389

11.3.2 Architecture Evaluation Objectives 390

11.3.3 Evaluation Approach Examples 391

11.3.4 Value Assessment Methods 391

11.3.5 Value Assessment Criteria 393

11.3.6 Architecture Analysis Methods 394

11.4 Steps in an Architecture Evaluation Process 396

11.5 Example Evaluation Taxonomy 398

11.5.1 Business Impact Factors 398

11.5.2 Mission Impact Factors 398

11.5.3 Architecture Attributes 399

11.6 Summary 400

11.7 Key Terms 400

11.8 Exercises 402

References 402

12 Exploring the Design Space 405
Clifford Whitcomb and Paul Beery

12.1 Introduction 405

12.2 Example 1: Liftboat 406

12.2.1 Liftboat Fractional Factorial Design of Experiments 406

12.2.2 Liftboat Design Trade-Off Space 409

12.2.3 Liftboat Uncertainty Analysis 411

12.2.4 Liftboat Example Summary 411

12.3 Example 2: Cruise Ship Design 411

12.3.1 Cruise Ship Taguchi Design of Experiments 411

12.3.2 Cruise Ship Design Trade-Off Space 412

12.3.3 Cruise Ship Example Summary 416

12.4 Example 3: NATO Naval Surface Combatant Ship 417

12.4.1 NATO Surface Combatant Ship Stakeholder Need 418

12.4.2 NATO Surface Combatant Ship Box–Behnken Design of Experiments 420

12.4.3 NATO Surface Combatant Ship Cost-Effectiveness Trade-Off 421

12.4.4 NATO Surface Combatant Ship Design Tradespace 421

12.4.5 NATO Surface Combatant Ship Design Trade-Off 422

12.4.6 NATO Surface Combatant Ship Trade-Off Summary 430

12.5 Key Terms 431

12.6 Exercises 433

References 435

13 Sustainment Related Models and Trade Studies 437
John E. MacCarthy and Andres Vargas

13.1 Introduction 437

13.2 Availability Modeling and Trade Studies 439

13.2.1 FMDS Background 439

13.2.2 FMDS Availability Trade Studies 449

13.2.3 Section Synopsis 453

13.3 Sustainment Life Cycle Cost Modeling and Trade Studies14 454

13.3.1 The Total System Life Cycle Model 454

13.3.2 The O&S Cost Model 456

13.3.3 Life Cycle Cost Trade Study 459

13.4 Optimization in Availability Trade Studies 464

13.4.1 Setting Up the Optimization Problem 464

13.4.2 Instantiating the Optimization Model 465

13.4.3 Discussion of the Optimization Model Results 468

13.4.4 Deterministic Sensitivity Analysis 469

13.5 Monte Carlo Modeling 471

13.5.1 Input Probability Distributions for the Monte Carlo Model 471

13.5.2 Monte Carlo Simulation Results 472

13.5.3 Stochastic Sensitivity Analysis 473

13.6 Chapter Summary 475

13.7 Key Terms 476

13.8 Exercises 478

References 482

14 Performing Programmatic Trade-Off Analyses 483
Gina Guillaume-Joseph and John E. MacCarthy

14.1 Introduction 483

14.2 System Acceptance Decisions and Trade Studies 485

14.2.1 Acceptance Decision Framework 486

14.2.2 Calculating the Confidence That a System Is “Good” 491

14.2.3 Acceptance Test Design and Trade Studies 493

14.2.4 A “Delay Fix and Test” Cost Model 499

14.2.5 The Integrated Decision Model 504

14.2.6 Conclusions 511

14.3 Product Cancelation Decision Trade Study 512

14.3.1 Introduction 512

14.3.2 Significance 513

14.3.3 Defining Failure 514

14.3.4 Developing the Predictive Model 519

14.3.5 Research Results 522

14.3.6 Model Implementation In Industry 528

14.3.7 Predictive Model Deployment in Industry 530

14.3.8 When the Decision Has Been Made to Cancel the System 536

14.3.9 Conclusion 537

14.4 Product Retirement Decision Trade Study 538

14.4.1 Introduction 538

14.4.2 Legacy HR Systems 539

14.4.3 The US NAVY Retirement and Decommission Program for Nuclear-Powered Vessels 544

14.4.4 Decision Analysis for Decommissioning Offshore Oil and Gas Platforms in California 551

14.4.5 System Retirement and Decommissioning Strategy 559

14.4.6 Conclusion 561

14.5 Key Terms 562

14.6 Exercises 564

References 566

15 Summary and Future Trends 571
Gregory S. Parnell and Simon R. Goerger

15.1 Introduction 571

15.2 Major Trade-Off Analysis Themes 572

15.2.1 Use Standard Systems Engineering Terminology 572

15.2.2 Avoid the Mistakes of Omission and Commission 572

15.2.3 Use a Decision Management Framework 572

15.2.4 Use Decision Analysis as the Mathematical Foundation 573

15.2.5 Explicitly Define the Decision Opportunity 573

15.2.6 Identify and Structure Decision Objectives and Measures 574

15.2.7 Identify Creative Doable Alternatives 574

15.2.8 Use the Most Appropriate Modeling and Simulation Technique for the Life Cycle Stage 575

15.2.9 Include Resource Analysis in the Trade-Off Analysis 575

15.2.10 Explicitly Consider Uncertainty 575

15.2.11 Identify the Cost Value Schedule and Risk Drivers 575

15.2.12 Provide an Integrated Framework for Cost Value and Risk Analyses 576

15.3 Future of Trade-Off Analysis 576

15.3.1 Education and Training of Systems Engineers 577

15.3.2 Systems Engineering Methodologies and Tools 577

15.3.3 Emergent Tradespace Factors 580

15.4 Summary 581

References 581

Index 583

Trade-off Analytics:  Creating and Exploring the System Tradespace is written for upper undergraduate students and graduate students studying systems design, systems engineering, industrial engineering and engineering management. This book also serves as a resource for practicing systems designers, systems engineers, project managers, and engineering managers.

Gregory S. Parnell, Ph.D, is a Research Professor in the Department of Industrial Engineering at the University of Arkansas. He is also a senior principal with Innovative Decisions, Inc., a decision and risk analysis firm and has served as Chairman of the Board. Dr. Parnell has published more than 100 papers and book chapters and was lead editor of Decision Making for Systems Engineering and Management, Wiley Series in Systems Engineering (2nd Ed, Wiley 2011) and lead author of the Handbook of Decision Analysis (Wiley 2013). He is a fellow of INFORMS, INCOSE, MORS, and the Society for Decision Professionals.