Open Access

Update to recommended best practice for verification and validation of ICME methods and models for aerospace applications

  • Bradford A Cowles1Email author,
  • Daniel G Backman2 and
  • Roland E Dutton3
Integrating Materials and Manufacturing Innovation20154:2

DOI: 10.1186/s40192-014-0030-8

Received: 7 November 2014

Accepted: 20 December 2014

Published: 23 January 2015

Abstract

Background

Rigorous verification and validation practices are regarded essential for successful implementation of Integrated Computational Materials Engineering (ICME) for aerospace applications. Guidelines and a recommended best practice for ICME verification and validation were developed in 2011 under Air Force Research Laboratory sponsorship and were published in 2012. These included general guidelines for ICME verification and validation, a recommended best practice, practitioner aides in the form of checklists for planning and status assessment, a Tool Maturity Level assessment guide, and a recommended approach for risk vs. consequence assessment for specific ICME applications. ICME development programs and expert practitioners have since reviewed and used the guidelines and provided constructive feedback for improvements.

Findings

This feedback was used to generate a comprehensive revision to the original guidelines and recommended best practice, as well as to the corresponding practitioner aides. The System-Level and Model-Level Checklists were simplified and improved, as was the Tool Maturity Level assessment guide. A template for a one-page summary chart was also generated to facilitate communication of status.

Conclusions

Experience with several ICME-related programs and expert practitioners reinforced the conclusion that guidelines and a recommended best practice are important tools to ensure that rigorous, comprehensive verification and validation practices are employed during ICME development and implementation efforts.

Keywords

ICME Verification Validation Tool maturity level System-level checklist Model-level checklist

Findings

Background and hypothesis

It is believed that broad development and implementation of Integrated Computational Materials Engineering (ICME) offers potential for significant benefits to all aspects of aerospace materials and processes engineering - in the form of greatly reduced time, cost and risk of technology development, validation, and sustainment. One major challenge for broad acceptance and implementation is that of verification and validation (V&V) of ICME models and methods. The US Air Force Research Laboratory (AFRL) sponsored efforts related to development of guidelines for ICME V&V. The approach taken to develop this recommended best practice followed three guiding tenets:
  1. 1.

    Utilize current, recognized practices for V&V as the basis. This was predominately the American Society of Mechanical Engineers (ASME) V&V Guide - 2006 [1].

     
  2. 2.

    Facilitate alignment with established, gated practices for product and technology development. These included the Technology and Manufacturing Readiness Level processes (TRL and MRL, respectively) and both the Integrated Product Development (IPD) and Defense Acquisition Guidebook processes for system or product development.

     
  3. 3.

    Provide simple, useful guidance and tools to aid practitioners in planning and assessing V&V activities and results for ICME development.

     

The rationale and approach for developing these guidelines and practitioner aides were described in a journal article in 2012 [2]. The initial versions of the recommended best practice, practitioner aides such as V&V checklists, a Tool Maturity Level (TML) assessment approach, and a recommended approach to assess risk vs. consequences of ICME application were appended as additional files accessible through the IMMI journal reference.

It was expected that the initial guidelines and practitioner aides would benefit from a revision cycle following some application experience. Several AFRL- and Defense Advanced Research Projects Agency (DARPA)-sponsored programs have since reviewed and/or applied the guidelines. Feedback from these programs and from individual experts was used to generate a comprehensive revision to the original guidelines and practitioner aides. This effort was performed under AFRL sponsorship of the Metals Affordability Initiative (MAI) program ‘GE-12: Verification and Validation (V&V) of ICME Models and Methods’ (Agreement Number: FA8650-08-2-5251).

Methods and approach

Feedback was solicited from numerous individuals engaged in ICME development programs representing aerospace propulsion companies and from within AFRL. Participants represented three programs where the guidelines and practitioner aides were used directly, in whole or in part. In these cases, technology and/or intended product applications were well defined, as was the intended role of the ICME elements under development. Responses were also solicited from selected individuals with prior ICME development and implementation experience. Feedback was collected over an 18-month period, from May 2012 through October 2013.

Feedback was generally consistent and focused on the need to simplify wording, reduce the number of line items in the practitioner aides, and clarify the recommended process flow for V&V planning and execution. The feedback was collectively reviewed, consolidated, and used to generate revisions to the ICME V&V guidelines, the recommended best practice, and the associated practitioner aides. The text document containing the revised guidelines and recommended best practice [3] and the associated Excel™ file containing the revised practitioner aides [4] were released by AFRL for public distribution in January 2014. These references are appended as Additional files 1 and 2.

Results

The general form of the guidelines and recommended best practice for ICME V&V planning and execution was preserved in the updated version, as itemized in the Background and hypothesis section, above. The recommended process flow chart, as revised based on the effort described above, is shown in Figure 1.
Figure 1

Flow chart of the simplified ICME V&V planning and assessment process.

The flow chart indicates the key steps in the process and where the practitioner aides may be helpful. The flow chart also indicates the expected iterative nature of developing an ICME V&V plan based on a specific technology or product application. A more detailed flow chart, and associated descriptions of the steps involved, is presented in the appended files [3,4].

Specific changes and improvements that were included can be summarized as follows:
  • System-Level and Model-Level Checklists: These practitioner aides were simplified in wording and number of line items. The supporting text was significantly revised and expanded to more clearly communicate intent and recommended process for use of this checklist.

  • TML assessment guide: This practitioner aide was revised by adding numerous supporting comments in the Excel™ comment tabs, adjusting some of the category headers based on feedback, and by adding some line items in the category boxes for selected TML levels.

  • Recommended Best Practice for ICME V&V: The two recommended process flow charts were revised. The simplified (‘3-step’) process flow chart was updated to illustrate the iterative and interactive nature recommended for key steps in the V&V planning process. The detailed process flow chart was revised to improve flow and reflect the expected actual sequence of activities in a V&V planning process. In addition, the step-by-step instructions were updated and recommended activities clarified.

  • Use of Phenomena Identification and Ranking Technique (PIRT) (note that often ‘Table’ is used instead of ‘Technique’ in this acronym): Instructions and an ICME-relevant example for the use of this technique provided by Southwest Research Institute (SwRI) were included in the guidelines and recommended best practice. Use of PIRT is a flexible approach to outline and describe model hierarchy and parameters, assess their relative importance, and communicate or illustrate this to the project team or reviewers and is now included as a recommended tool for use.

  • Development of a ‘Dashboard’ chart: A simple, one-page chart in presentation format was developed, suitable for succinctly summarizing the status of ICME V&V planning and execution for a specific project. The ‘Dashboard’ reflects the status of checklist items, TML, and any risk mitigation items in a format suitable for reporting and presentation.

  • Application examples were expanded and improved: A more detailed example of use of the ‘Risk vs. Consequences’ approach for prediction of forging residual stress predictions was included. A detailed example of uncertainty assessment for an ICME computational model for aluminum was expanded from the prior version and related to the TML assessment guide.

  • Alignment of the guidelines and recommended best practice with current gated processes: Background and supporting text was revised, and figures showing alignment of the recommended best practice with TRL, MRL, IPD, and the Department of Defense (DoD) Acquisition Guide were revised and improved.

One key conclusion of the effort to revise and update these guidelines was that such guidelines are, in fact, essential to the successful communication of V&V activities across the supply chain and between engineering, manufacturing, and product disciplines. The perception of what constitutes adequate V&V for ICME applications was observed to vary significantly amongst practitioners and was often based on whether the practitioner was a supplier, an original equipment manufacturer, or a customer. The adoption of a common approach to V&V was found to facilitate the successful development and implementation of ICME methodologies.

Availability of supporting data

The data sets supporting the results of this article are included within the article and its additional files.

Abbreviations

AFRL: 

Air Force Research Laboratory

ASME: 

American Society of Mechanical Engineers

DARPA: 

Defense Advanced Research Projects Agency

DoD: 

Department of Defense

ICME: 

Integrated Computational Materials Engineering

IPD: 

Integrated Product Development (sometimes …Deployment)

MAI: 

Metals Affordability Initiative

MRL: 

Manufacturing Readiness Level

PIRT: 

Phenomena Identification and Ranking Technique

SwRI: 

Southwest Research Institute

TML: 

Tool Maturity Level

TRL: 

Technology Readiness Level

V&V: 

verification and validation

Declarations

Acknowledgements

The authors would like to thank John Warren (GE) and David Riha (SwRI) for their contributions to the development of this material.

Authors’ Affiliations

(1)
Cowles Consulting, LLC
(2)
Backman Materials Consulting, LLC
(3)
Materials and Manufacturing Directorate, Air Force Research Laboratory

References

  1. The American Society of Mechanical Engineers (2006) Guide for verification and validation in computational solid mechanics. ASME V&V 10-2006. ASME International, New YorkGoogle Scholar
  2. Cowles BA, Backman DG, Dutton RE (2012) Verification and validation of ICME methods and models for aerospace applications. Integrating Materials and Manufacturing Innovation 1:2. http://www.immijournal.com/content/1/1/2
  3. Cowles BA, Backman DG (2013) Verification and validation of ICME methods and models for aerospace applications – V&V guidelines and recommended best practice, revision 1. Summary document prepared for AFRL/RXM. Cowles Consulting, LLC, Tolland. Appended as Additional file 1
  4. Cowles BA, Backman DG (2013) Verification and validation of ICME methods and models for aerospace applications – V&V guidelines and recommended best practice, revision 1. Spreadsheet tools and practitioner aides prepared for AFRL/RXM. Cowles Consulting, LLC, Tolland. Appended as Additional file 2

Copyright

© Cowles et al.; licensee Springer. 2015

This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited.