Credibly reaching a reliability target using a model initially constructed by expert elicitation
 Lawrence E Pado^{1}Email author
DOI: 10.1186/s401920140020x
© Pado.; licensee Springer. 2014
Received: 20 February 2014
Accepted: 29 May 2014
Published: 18 June 2014
Abstract
The Defense Advanced Research Projects Agency Defense Science Office (DARPA/DSO) is sponsoring Open Manufacturing (OM), an initiative to develop new technologies, new computational tools, and rapid qualification to accelerate the manufacturing innovation timeline. Certification Methodology to Transition Innovation (CMTI), an OM program, has developed a methodology to quantify the effect of manufacturing variability on product performance to address the risk to cost and performance associated with failure to take manufacturing capability and material and fabrication/assembly variation into account early in the design process. An important aspect of this program is the use of Bayesian networks (BN) to evaluate risk. The BN is used as a graphical representation of the contributing factors that lead to manufacturing defects. The reliability of the final product is then analyzed using the contributing factors. There are many types of programs where there is little relevant data to support the probabilities needed to populate the BN model. This is very likely the case for new programs or at the end of long programs when obsolescence challenges servicing a product when original vendors are no longer in business. In these cases, probabilities must be obtained from expert opinion using a technique called expert elicitation. Even under objective ‘Good Faith’ opinions, the expert himself has a lot of uncertainty in that opinion. This paper details an approach to obtaining credible model output based on the idea of having a hypothetical expert whose unconscious bias influences the model output and discovering and using countermeasures to find and prevent these biases. Countermeasures include replacing point probabilities with beta distributions to incorporate uncertainty, 95% confidence levels, and using a multitude of different types of sensitivity analyses to draw attention to potential trouble spots. Finally, this paper uses a new technique named ‘confidence level shifting’ to optimally reduce epistemic uncertainty in the model. Taken together, the set of tools described in this paper will allow an engineer to cost effectively determine which areas of the manufacturing process are most responsible for performance variance and to determine the most effective approach to reducing that variance in order to reach a target reliability.
Keywords
Credibility Expert elicitation Confidence level shifting Monte Carlo Uncertainty quantification Targeted testing Unitized testing Uncertainty reduction Epistemic uncertainty Reliability targetsBackground
The Defense Advanced Research Projects Agency Defense Science Office (DARPA/DSO) is sponsoring Open Manufacturing (OM), an initiative to develop new technologies, new computational tools, and rapid qualification to accelerate the manufacturing innovation timeline. Certification Methodology to Transition Innovation (CMTI), one of the programs in the OM portfolio, has developed a methodology to quantify the effect of manufacturing variability on product performance to address the risk to cost and performance associated with failure to take manufacturing capability and material and fabrication/assembly variation into account early in the design process.
Motivation
The goal motivating this research is to first credibly ascertain the reliability of a product by including the effects of variability and defects in manufacturing as well as uncertainty in the environment. Note that credibility is a key requirement and is made more difficult when using expert elicitation to determine the value of model parameters used to calculate the reliability. Secondly, upon evaluation of the manufacturing process, it is very likely that product reliability will fall short of the desired target. The set of tools described in this paper will allow the engineer to cost effectively determine which areas of the manufacturing process are most responsible for performance variance and to determine the most effective approach to reducing that variance in order to reach a target reliability. These benefits will be explained in detail in the sections entitled ‘Techniques to meet a target POF with a 95% confidence level’ and ‘Putting it all together  an example using credibility tools’.
An exemplar problem
POF is partially a function of the probability of defects occurring during the manufacture of a part. A subset of all possible defects that can be introduced by the manufacturing process, tooling, etc., and that are thought to contribute to failure under acknowledged conditions, were identified. The focus of this work was on quantifying and reducing manufacturing defects. Although material and environmental variability were accounted for, examination of the costs and benefits of reducing the variability of those factors were beyond the scope of this research. They are however important factors and will be considered in future work. For the hatstiffened panel, the defects recognized were wrinkles (nugget/noodle fiber waviness), noodle void/porosity/geometry, lower radius thickening, upper radius thinning, and top crowning. If there is any doubt as to if a defect can affect performance, it should be included in the analysis.
Once the defects of interest were identified, the stepbystep manufacturing process was analyzed to determine which steps or combinations of steps could possibly produce one or more of those defects. Additionally, options that could affect the probability of introducing defects were identified such as tooling choices, manufacturing capability levels, and manufacturing process alternatives.
In order for the network to calculate these total defect and POF probabilities, however, the probability that each individual manufacturing step can induce a defect and that each individual quality assurance test can find it, if it exists, must be provided.
Methods
An approach to determining the credibility of models
For those programs where there is little relevant data to support the probabilities needed to populate the BN model, expert elicitation must be used to provide them. Even under objective ‘Good Faith’ opinions, the expert himself has a lot of uncertainty in that opinion. On a given day, the expert may even choose different probabilities than the one he had chosen earlier. Given that the output of the model has realworld consequences, possibly in terms of customer acceptance of the product, there may be bias when choosing probabilities  especially if the model has not shown that it can meet a target POF.
The chosen way to approach this issue is to make the potential bias explicit by figuring out the best way to modify model inputs to get a desired result. Once this methodology is known, the idea is to reverse engineer it to find countermeasures and establish the credibility of the model.
The expert's first approach  using and adjusting point probabilities
The expert's initial approach is to make estimates of the probabilities as objectively as possible. If the target POF is reached, then the expert is done. If not, he will find the point probabilities that the output is most sensitive to and adjust them as little as possible such that the target is reached. Note that point probabilities are probabilities that are assumed to be known with absolute certainty and are represented by a single scalar value. The reasoning behind this strategy is that adjusting the probabilities that do not have much of an effect would require large unrealistic changes to have a significant effect.
Derivativebased approach to sensitivity
With this technique in hand, the expert would calculate the sensitivity of POF to changing the probabilities within every manufacturing step or QA test node of the network. After sorting from most sensitive to least sensitive, he could calculate exactly how much to change the top few nodes to reach his target POF.
Quantifying uncertainty in the model probability parameters  the beta distribution
The beta distribution is quite flexible and can represent uniform (a = 1, b = 1), ramp, symmetrical, and asymmetrical distributions. It is simple to implement Bayesian learning using newly introduced data using this distribution. For each step in the manufacturing process, if a flaw occurs during that step, the parameter ‘a’ merely needs to be incremented by one. Likewise, if no flaw is introduced during that step, the parameter ‘b’ should be incremented by 1. For quality assurance (QA) tests, if a QA test does not miss a flaw that exists, then ‘b’ should be incremented. If the test misses a flaw that exists, ‘a’ should be incremented.
Using expert opinion elicitation to determine the parameters of the beta distribution
The goal of expert opinion elicitation is to determine the parameters of beta distribution such that it accurately represents the expert's opinion about the most likely value of the probability to be specified (the mode) as well as his uncertainty in that opinion. In this methodology, this is accomplished by having the expert express his uncertainty in terms of the number of samples he has seen. The following basic example will build the readers intuition about this process.
Finally after 30 flips, and obtaining 30 heads in a row, it is clear that this is not a fair coin but is very highly weighted towards coming up heads. The updated beta distribution is shown in Figure 4C. Note that even after 30 flips, it is not a sure thing that a heads result will always be obtained. Also note that the distribution is getting narrower and narrower representing an increase in the certainty of the coin's weighting value.
Using the above example as an intuitive example of the meaning of ‘samples seen’, the expert can be asked to provide a level of uncertainty in terms of samples seen. For the exemplar problem, what proportion of wrinkles has been observed during the debulking process (i.e., the probability of a wrinkle occurring during the debulking process)? Note that this is represented by the node named ‘Wrinkle induced during debulking’ in Figure 3.
This process is continued for every node in the network until every parameter of the network is represented by a beta distribution.
Determining a 95% confidence value on the model output using Monte Carlo methods
A Monte Carlo analysis entails pulling a single sample from each distribution within the model, populating the model with these new samples, and then running the model to get a single answer. This process is then repeated thousands of times to collect enough data to establish the distribution of the output parameter of interest such as POF and enables calculating its CL.
Finding the 95% CL using histogram data is a straightforward process involving sorting the POF data from lowest value to highest value and then selecting the value for which 95% of the data is that value or smaller.
The expert's second approach  adjusting mode and certainties of PDFs
Using probability distributions instead of point probabilities and instituting a 95% confidence level to introduce conservatism is a good first step for establishing credibility. The expert's bias, if it exists, may now show up as a lower mode value or as a higher level of certainty in that mode value. This type of bias must be detected if it is to be countered.
Countermeasure to the expert's second approach  sensitivity analysis on the mode with respect to POF
The countermeasure to the impact of the expert's mode and certainty selection is to perform a sensitivity analysis of the 95% confidence level to changes in the mode.
There are two conditions that have to be met before the 95% confidence level shows significant sensitivity to the mode of a node:
The node has to have already been shown to be important through the use of sensitivity analysis. If the output has no sensitivity to the node, then the mode of the node is inconsequential.
The sensitivity of 95% CL to a mode increases as the certainty in the value of probabilities increases. As discussed above, for beta distributions, certainty is a function of the number of samples expressed. A sample size of 2 will result in no sensitivity to mode with the sensitivity increasing as the number of samples increases.
Where:
where ${\mathit{S}}_{95\%{\mathrm{CL}}_{\mathit{i}}}$ = the sensitivity of the 95% confidence level of the model output of interest to a change in mode of node i, Δ 95% CL = the change in the 95% confidence level due to a change in mode of node i, and Δ Mode_{ i } = the change in mode of node i.
The following process can be used to calculate ${\mathit{S}}_{95\%{\mathrm{CL}}_{\mathit{i}}}$
Calculate the baseline 95% CL by running a Monte Carlo analysis on the baseline model.
Choose a node i.
Increase the mode of node i by a delta value.
Calculate the new 95% confidence level by running a Monte Carlo analysis.
Calculate the sensitivity as per Equation 6.
Repeat this process for each node i of interest.
Sensitivity calculations of 95% CL to a change in mode for an illustrative exemplar
Nodei  NumSamples  Old_mode  New_mode  Delta_mode  95POF  New95POF  Delta_POF  Sen 

Wrinkle introduced during debulking  120  0.114  0.227  0.114  4.80 × 10^{−5}  8.07 × 10^{−5}  3.26 × 10^{−5}  0.00029 
QA test finds debulking wrinkle if it exists  2  0.100  0.200  0.100  4.80 × 10^{−5}  4.80 × 10^{−5}  0.00E + 00  0.00000 ^{a} 
UltraSonic inspection finds wrinkle  1,000  0.001  0.002  0.001  4.80 × 10^{−5}  6.25 × 10^{−5}  1.45 × 10^{−5}  0.01453 ^{b} 
As shown in Table 1, the credibility of the mode value in the first two nodes listed in the table is very high as even a very large change in the mode value would have very little effect on the 95% CL POF. In fact, the mode value of node ‘QA test finds debulking wrinkle if it exists’ has absolutely no effect on the value of 95% CL POF due to having the maximum uncertainty in its value. It should be noted, however, that the uncertainty in the mode value has a large impact on the variance of the output, as will be discussed in more detail below. The final node, ‘UltraSonic inspection finds wrinkle,’ with a sensitivity of 0.01453, indicates that it would have been possible for the expert to significantly change the 95% CL POF by changing the mode. More specifically, the 95% CL changes by 1.45 × 10^{−5} for every 0.001 change in the mode. This means, for example, that if the mode was originally 0.008 and the expert lowered it to 0.001, the 95% CL would have been 7 × 1.45 × 10^{−5} higher or 1.5 × 10^{−4} instead of the reported 4.8 × 10^{−5} from Table 1. The consequence of this observation is that the expert should be required to provide documented proof of the 1,000 sample size or else he should be required to reduce his reported sample size.
Techniques to meet a target POF with a 95% confidence level
With these procedures in place, the expert may find that it is not possible to meet the target value of product reliability (i.e., a low enough probability of failure). What guidance can be provided to the expert to cost effectively increase reliability?
The goal is to raise the ‘certain’ reliability costeffectively. The word ‘certain’ here is used to indicate that a reported low reliability may be due in part to a lack of process knowledge, while the other portion is due to variability in the manufacturing process coupled with a lack of suitable quality assurance tests. These ideas are captured by the following two types of uncertainty [9]:
Aleatory variability is the natural randomness in a process. Aleatory uncertainty cannot be reduced thru data collection. For example, the knowledge of what number will turn up on a sixsided die. This type of uncertainty can be reduced through better process control and through quality assurance testing. In the die analogy, this is equivalent to reducing the number of sides on the die or weighting the die to come up favorably.
Epistemic uncertainty is the scientific uncertainty in the model of the process. It is due to limited data and knowledge. This uncertainty can be reduced through more data collection, better expert knowledge, or through analytical means.
Reducing aleatory uncertainty through improved process control to lower randomness is application dependent and will not be discussed in this paper other than to note that the identification of the processes that drive uncertainty in the output is invaluable.
This section will discuss improving reported reliability by reducing epistemic uncertainty through targeted testing.
The most direct way to reduce uncertainty in the output (thus reducing 95% CL) is by reducing the variance in the output. Thus, the goal at this stage is to discover which nodes are most responsible for variance in the output. Once that is known, the focus should be on reducing the variance of those nodes. This may involve breaking a single mode into multiple subnodes to increase the level of detail of a particular process.
Saltelli et al. [4] have developed a technique to efficiently determine which variables in a probabilistic model contribute the most to variance in the output. This technique is called variancebased global sensitivity analysis and herein will also be referred to as the Saltelli method.
It is illuminating to compare point or derivativebased sensitivity analysis (previously discussed) and to which Equation 1 refers, with Saltelli global sensitivity analysis.
Conventional derivative (point)based sensitivities
Do not take into account uncertainty in the parameters.
Do provide good information about a parameter at its most likely value.
Global sensitivity analysis (GSA) (the Saltelli method)
Does take into account uncertainty in the parameters
Is capable of determining which factors have a major effect on the variance of the POF calculation
Is capable of determining which factors interact with others in an important way (synergistic effects)
Is especially useful for determining the small subset of parameters that are important
Is essentially a variance decomposition algorithm  it determines to some degree what portion of the output variance is due to variance in a particular parameter
The Saltelli process produces two sensitivity measures for each variable. S_{ i } indicates the main effect of variable i, and S_{ T }_{ i } indicates the total effect of variable i. There are a few characteristics of these two types of sensitivities that are important to know. S_{ i } indicates by how much one could reduce (on average) the output variance if variable i could be fixed. It is a measure of the main effect. S_{ T }_{ i } is useful in determining two important aspects of a variable. This first is if it has interactions with other variables. This can be measured by (S_{ T }_{ i } − S_{ i }). The second is if the variable is noninfluential and can safely be ignored by setting it to a fixed value when performing time consuming analyses. This is indicated by S_{ T }_{ i } = 0.
Saltelli global sensitivity analysis of the exemplar problem
Factor name  Hat_max_load_FE  Hat_max_load_TE  Mode  NumSamples 

QA test finds debulking wrinkle if it exists  0.407223  0.633979  0.010  20 
Wrinkle introduced during debulking  0.210948  0.525119  0.010  120 
Wrinkle intro. during final cloth overwrap  0.056164  0.187127  0.010  120 
QA test finds final cloth overwrap wrinkle  0.033471  0.172301  0.020  120 
QA test finds bagging wrinkle  0.024144  0.154588  0.010  120 
Wrinkle introduced during release film  0.029566  0.152148  0.010  120 
QA test finds release film wrinkle  0.025807  0.15214  0.010  120 
Wrinkle introduced during bagging  0.033863  0.142558  0.010  120 
Radius thickening intro. during cloth overwrap  0.002573  0.126621  0.020  35 
QA test finds debulking radius thickening  0.002591  0.12661  0.010  2 
Reducing epistemic uncertainty using confidence level shifting (CLS)
To begin the CLS process, a Monte Carlo procedure will be run for the baseline network and the 95% confidence level of POF will be calculated before any NRTs have been applied to a beta distribution. Note that each beta distribution represents a probability (a factor). Next, a NRT is applied to a single factor and the Monte Carlo analysis is rerun and a new 95% confidence level of POF will be calculated. This provides enough information to calculate a Δ 95% confidence level for POF which is calculated as the original 95% confidence level for POF minus the newly calculated original 95% confidence level for POF. This term can be expressed more compactly as Δ 95% POF or even more simply as ΔPOF. If the cost of performing the test is known, another term, ΔPOF/$, can be defined which is the amount of change in 95% POF per dollar spent. The metric can be used to determine what data should be collected to most cost effectively drive the 95% POF value to the left.
Note that a single complete fabrication of a part with appropriate inspection will simultaneously provide a data point for every node in the network. Partial part constructions can be accomplished to gather data for just the most important nodes. The ratio of decrease in POF to the cost of running a trial is the measure by which it is decided which trials to run. Note that a test that simultaneously provides data for multiple nodes is called a ‘unitized test’. Unitized tests are a time and costefficient technique for generating data to reduce epistemic uncertainty.
Results of targeted testing analysis using confidence level shifting
Data  MS final cloth overwrap wrinkle  QA final cloth overwrap wrinkle  MS during debulking wrinkle  Q during debulking wrinkle  MS during bagging wrinkle  Q during bagging wrinkle  MS placing release film wrinkle  Q placing release film wrinkle  Lowest POF 

0  0  0  0  0  0  0  0  0  4.80 × 10^{−4} 
25  0  0  0  25  0  0  0  0  3.28 × 10^{−4} 
50  0  0  0  50  0  0  0  0  2.92 × 10^{−4} 
75  0  25  0  50  0  0  0  0  2.75 × 10 − 4 
100  0  25  0  75  0  0  0  0  2.56 × 10^{−4} 
125  25  25  0  75  0  0  0  0  2.45 × 10^{−4} 
150  25  25  0  75  0  25  0  0  2.31 × 10^{−4} 
175  25  25  0  75  0  25  25  0  2.18 × 10^{−4} 
200  25  25  0  75  0  25  25  25  2.13 × 10^{−4} 
225  25  50  0  75  0  25  25  25  2.05 × 10^{−4} 
250  25  50  0  100  0  25  25  25  1.94 × 10^{−4} 
275  25  50  0  100  0  25  25  50  1.88 × 10^{−4} 
300  25  75  0  100  0  25  25  50  1.80 × 10^{−4} 
Unitized test for efficient collection of testing data
Results and discussion
Putting it all together  an example using credibility tools
This section will provide an example of using the credibility tools discussed in this paper to reach a 95% confidence level probability of failure of 1 × 10^{−4} when starting with a manufacturing process for a threehatstiffened panel that has a 4.4% probability of failure under certain environmental conditions when no quality assurance testing is done.
To reach the stated goal of 1 × 10^{−4} POF or 0.9999 reliability, the $2,626 option is the most costeffective (not including scrap or rework). This option represents the case that all QA tests are off except for the ultrasonic inspection for wrinkle QA test.
While the mode of this analysis meets the goal of 1 × 10^{−4} (being 5.6 × 10^{−5}), the 95% confidence value in POF is slightly too large at 2.3 × 10^{−4}. Another issue with using only the ultrasonic inspection for wrinkle QA test is that it only catches the wrinkle after the part is complete, leading to a very high rejection rate of a finished part. According to the model, there is a 46% chance of a wrinkle defect. This means that nearly half of the completed parts would have to be rejected. This is unacceptable. To better understand what is causing the wrinkles, an analysis of manufacturing steps as modeled by the Bayesian network is undertaken.
By removing all QA tests and running a Saltelli global sensitivity analysis, the manufacturing steps most responsible for output variation can be found.
Saltelli global sensitivity analysis results
Title  Hat_max_load_FE  Hat_max_load_TE  Mode  NumSamples 

Wrinkle placing release film strips  0.435196  1.101102  0.2153  120 
Wrinkle induced during debulking  0.192996  0.950746  0.1136  120 
Wrinkle during bagging and pleating  0.713702  0.824299  0.2153  120 
Wrinkle applying final cloth over wrap  0.039608  0.776457  0.0100  120 
The relationship between p(wrinkle) and POF
Examining the effect of four inprocess QA checks
Efficacy and cost of QA tests for wrinkles
Node  Mode  NumSamples  Cost 

QA test finds bagging wrinkle  0.0400  120  $30 
QA test finds debulking wrinkle  0.1000  2  $315 
QA test finds release film wrinkle  0.0200  120  $30 
QA test finds final cloth overwrap wrinkle  0.0200  120  $165 
Identifying manufacturing process areas to target for improvements
Global sensitivity analysis of the manufacturing network that includes the four wrinkle direct QA tests
Factor name  Hat_max_load_FE  Hat_max_load_TE  Mode  NumSamples 

QA test finds debulking wrinkle if it exists  0.817142  0.88832  0.1000  2 
Wrinkle introduced during debulking  0.13929  0.279974  0.1136  120 
QA test finds bagging wrinkle  0.003301  0.096139  0.0400  120 
QA test finds release film wrinkle  0.00201  0.087771  0.0200  120 
Wrinkle introduced during bagging  0.000163  0.080021  0.2153  120 
Wrinkle introduced during release film  −0.003777  0.082574  0.2153  120 
QA test finds final cloth overwrap wrinkle  −0.005413  0.082446  0.0200  120 
Wrinkle intro. during final cloth overwrap  −0.005268  0.082371  0.0100  120 
Point sensitivity analysis of the manufacturing network that includes the four wrinkle direct QA tests
Node  POFHat_max_load_sens  Mode  NumSamples 

QA test finds bagging wrinkle  0.02038  0.040  120 
Wrinkle introduced during release film  0.02029  0.020  120 
QA test finds debulking wrinkle if t exists  0.01079  0.100  2 
Wrinkle introduced during debulking  0.00949  0.114  120 
Wrinkle introduced during bagging  0.00379  0.215  120 
Wrinkle introduced during release film  0.00189  0.215  120 
Wrinkle intro. during final cloth overwrap  0.00188  0.010  120 
QA test finds final cloth overwrap wrinkle  0.00094  0.020  120 
Table 6 shows that most of the variance is due to the debulking and bagging steps and also that there is synergy between those nodes and other nodes. The point sensitivity analysis of Table 7 shows that those nodes are also prominent in affecting POF. Based on these two tables, it appears that the most efficient means of decreasing POF is by improving the manufacturing debulking related steps and the QA of the debulking simultaneously (top two globally sensitive) to take advantage of the synergy between them as well as their high point sensitivity. If this is not enough, then the nodes related to bagging and the release film should be worked on next.
Global sensitivity analysis of the manufacturing network that includes four wrinkle direct QA tests and improved debulking steps
Factor name  Hat_max_load_FE  Hat_max_load_TE  Mode  NumSamples 

QA test finds bagging wrinkle  0.488152  0.625328  0.0400  120 
QA test finds release film wrinkle  0.298514  0.388214  0.0200  120 
Wrinkle introduced during bagging  0.135016  0.145427  0.2153  120 
Wrinkle introduced during release film  0.039348  0.123047  0.2153  120 
Wrinkle introduced during debulking  0.00898  0.122452  0.0100  120 
QA test finds debulking wrinkle if it exists  0.036094  0.115424  0.0100  20 
Wrinkle intro. during final cloth overwrap  0.006912  0.096334  0.0100  120 
QA test finds final cloth overwrap wrinkle  0.003504  0.095578  0.0200  120 
Radius thickening intro. during final cloth overwarp  0.003001  0.093529  0.0200  35 
At this stage, it may be costeffective to perform targeted testing to reduce epistemic uncertainty by using the confidence level shifting (CLS) analysis technique. The first step in the CLS process is to identify which nodes are causing the most variance on the output POF using global sensitivity analysis. Table 2 from the main body of this paper shows the results of this. With the steps and QA tests related to wrinkles balanced in terms of performance by engineers, all eight nodes related to wrinkles are found to be important. Figure 9 from this paper's CLS section shows that 95 successful unitized targeted tests can be run to drive the 95% CL POF to 1 × 10^{−4}.
Results of sensitivity calculations of 95% CL to a change in mode
Nodes  numSamples  Old_mode  New_mode  Delta_mode  95POF  New95POF  Delta_POF  Sen 

Wrinkle introduced during debulking  215  0.0055  0.0155  0.0100  9.26 × 10 ^{ −5 }  1.21 × 10 ^{ −4 }  2.81 × 10 ^{ −5 }  2.81 × 10 ^{ −3 } 
QA test finds debulking wrinkle if it exists  115  0.0055  0.0155  0.0100  9.26 × 10 ^{ −5 }  1.14 × 10 ^{ −4 }  2.11 × 10 ^{ −5 }  2.11 × 10 − 3 
QA test finds bagging wrinkle  215  0.0055  0.0155  0.0100  9.26 × 10 ^{ −5 }  1.11 × 10 ^{ −4 }  1.85 × 10 ^{ −5 }  1.85 × 10 ^{ −3 } 
QA test finds final cloth overwrap wrinkle  215  0.0032  0.0132  0.0100  9.26 × 10 ^{ −5 }  1.11 × 10 ^{ −4 }  1.81 × 10 ^{ −5 }  1.81 × 10 ^{ −3 } 
Wrinkle introduced during bagging  215  0.0055  0.0155  0.0100  9.26 × 10 ^{ −5 }  1.10 × 10 ^{ −4 }  1.75 × 10 ^{ −5 }  1.75 × 10 ^{ −3 } 
QA test finds release film wrinkle  215  0.0055  0.0155  0.0100  9.26 × 10 ^{ −5 }  1.10 × 10 ^{ −4 }  1.75 × 10 ^{ −5 }  1.75 × 10 ^{ −3 } 
Wrinkle introduced during release film  215  0.0055  0.0155  0.0100  9.26 × 10 ^{ −5 }  1.09 × 10 ^{ −4 }  1.60 × 10 ^{ −5 }  1.60 × 10 ^{ −3 } 
Wrinkle intro. during final cloth overwrap  215  0.0055  0.0155  0.0100  9.26 × 10 ^{ −5 }  1.06 × 10 ^{ −4 }  1.37 × 10 ^{ −5 }  1.37 × 10 ^{ −3 } 
Radius thickening intro. during debulking  30  0.0100  0.0200  0.0100  9.26 × 10^{−5}  9.55 × 10^{−5}  2.88 × 10^{−6}  2.88 × 10^{−4} 
QA test finds release film up. radius thickening  10  0.0100  0.0200  0.0100  9.26 × 10^{−5}  9.51 × 10^{−5}  2.50 × 10^{−6}  2.50 × 10^{−4} 
QA test finds bagging radius thickening  20  0.0050  0.0150  0.0100  9.26 × 10^{−5}  9.45 × 10^{−5}  1.93 × 10^{−6}  1.93 × 10^{−4} 
Summary of using credibility tools to reach a credible target 95% CL POF
In summary, an example project consisting of manufacturing a threehatstiffened panel was used as a case study for exercising the credibility tools detailed in this paper. The goal of the example was to analyze a manufacturing process in terms of the factors that contribute to its unreliability, to ensure that the expert opinion using to furnish the parameters of that model were credible, and then use a number of tools to determine the optimal way to create a more reliable product that met a target reliability number. Note that this was accomplished conceptually for illustrative purposes.
The examination started with noting the effect of any and all possible combinations of manufacturing options on the point reliability of the part. It was found that quality assurance tests had an extremely high impact on part reliability. After noting that a postmanufacturing QA test was effective but that it resulted in many costly part rejections, another analysis was undertaken which found four manufacturing steps that induced wrinkles also contributed the most to variance in the output. QA tests were directly applied within the model to address these four steps but it was found that they were not effective enough to reach the target 95% CL POF. An iterative process was then undertaken which involved identifying and then improving aspects of the manufacturing process until the process nearly reached the target POF. At this point, an effort to reduce epistemic uncertainty in the model was undertaken using confidence level shifting to identify target testing. This testing optimally reduced uncertainty in the model. Finally, a sensitivity analysis of the 95% confidence level POF to a change in mode was performed for each node of the model to indicate influential distributions that must be justified by documentation. At this point, it was found that with the targeted testing that had already occurred that the model was credible.
Conclusions
This paper details an approach to obtaining credible model output when model parameters are based on expert opinion. Although the model used as an example in this paper is a Bayesian network model, the approach and techniques described in this paper are completely transferrable to any model using uncertain parameters. This paper details an approach to obtaining credible model output based on the idea of having a hypothetical expert whose unconscious bias influences the model output and discovering and using countermeasures to find and prevent these biases. Countermeasures include replacing point probabilities with beta distributions to incorporate uncertainty and requiring 95% confidence levels to add conservatism. Multiple types of sensitivity analyses are used to identify parameters in the model that have the most influence over the model's output. This includes a derivative point probabilitybased sensitivity analysis that is a good indicator of relevance when all parameters are at their most likely values, a sensitivity analysis of 95% confidence level to a change in mode which is a good indication of influential distributions that must be justified by documentation and a variancebased global sensitivity analysis which is useful for identifying which model parameters contribute the most to output variance and which model parameters have synergy with other model parameters. Finally, this paper uses a new technique named ‘confidence level shifting’ to cost and time optimally reduce epistemic uncertainty in the model. This is useful when uncertainty in model parameters is inflating the 95% confidence level of a reported target output (such as probability of failure or probability of a defect) and needs to be brought down as cost effectively as possible.
Abbreviations
 Δ:

delta, change in value
 A:

beta distribution parameter expressing the number of flawed examples
 B:

beta distribution parameter expressing the number of flawless examples
 BN:

Bayesian networks
 CL:

confidence level
 CLS:

confidence level shifting
 CMTI:

Certification Methodology to Transition Innovation
 DARPA/DSO:

Defense Advanced Research Projects Agency Defense Science Office
 GSA:

global sensitivity analysis
 K:

expert confidence in estimate in terms of equivalent prior sample size
 Mode:

the most likely probability of a flaw
 NRT:

negative result test
 OM:

Open Manufacturing
 P :

proportion of flaws in the beta distribution
 P _{i} :

probability of node i inducing or failing to detect a defect
 POF:

probability of failure
 QA:

quality assurance
 RT:

radial thickening
 S _{95%CLi} :

sensitivity of the 95% CL of model output due to change in mode of node i
 S _{D} i :

derivativebased sensitivity measure
 Si :

effect due to variable i
 S _{T} _{ i } :

total effect due to variable i
 X _{i} :

model parameter
 Y :

model output
Declarations
Acknowledgements
This paper is sponsored by Defense Advanced Research Projects Agency, Defense Sciences Office under the Open Manufacturing Program, ARPA Order No. S587/00, Program Code 2D10, issued by DARPA/CMO under contract no. HR 001112C0034. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressly or implied, of the Defense Advanced Research Projects Agency of the U.S. Government. This paper was approved for public release, distribution unlimited as 1400070EOT.
Authors’ Affiliations
References
 Renieri G: High performance stiffened panel design concept. Annual Report. Office of Naval Research (ONR), Arlington, VA; 2013.Google Scholar
 Koller D, Friedman N: Probabilistic graphical models: principles and techniques. Press, MIT, Cambridge; 2009.Google Scholar
 Hahn GL, Pado LE, Thomas MJ, Liguore SL: Application of risk quantification approach to aerospace manufacturing using Bayesian networks. Paper presented at AIAA SciTech 2014. National Harbor, Maryland; 2014.Google Scholar
 Saltelli A, Ratto M, Andres T, Campolongo F, Cariboni J, Gatelli D, Saisana M, Tarantola S: Global sensitivity analysis. The Primer, Wiley, John & Sons, , Incorporated; 2008.Google Scholar
 Kruschke J: Doing Bayesian data analysis. Press, Academic, New York; 2011.Google Scholar
 Kruschke J: Beta distribution parameterized by mode instead of mean. 2012.Google Scholar
 Haldar A, Mahadevan S: Probability, reliability, and statistical methods in engineering design. Wiley, John & Sons, Incorporated, New York; 2000.Google Scholar
 Milton JS, Arnold JC: Probability and statistics in the engineering and computing sciences. Elsevier Academic Press, Burlington, MA; 1986.Google Scholar
 Abrahamson N: Seismic hazard assessment: problems with current practice and future developments Proceedings. First European Conference on Earthquake Engineering and Seismology, Geneva, Switzerland; 2006.Google Scholar
Copyright
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited.