Validation metrics for response histories: perspectives and case studies

Abstract

The quantified comparison of transient response results are useful to analysts as they seek to improve and evaluate numerical models. Traditionally, comparisons of time histories on a graph have been used to make subjective engineering judgments as to how well the histories agree or disagree. Recently, there has been an interest in quantifying such comparisons with the intent of minimizing the subjectivity, while still maintaining a correlation with expert opinion. This increased interest has arisen from the evolving formalism of validation assessment where experimental and computational results are compared to assess computational model accuracy. The computable measures that quantify these comparisons are usually referred to as validation metrics. In the present work, two recently developed metrics are presented, and their wave form comparative quantification is demonstrated through application to analytical wave forms, measured and computed free-field velocity histories, and comparison with Subject Matter Expert opinion.

Appendices

Appendix: Analytical wave form comparisons

The three cases presented in this Appendix are offered as further illustrations of the two considered validation metrics, as comparison cases for verification of local implementations of the metrics, and for comparison with possible future validation metrics.

1.1 Magnitude and decay rate differences

To illustrate magnitude and decay errors, Geers offers the following analytical function

$$c_{m} {\left( t \right)} = 0.8{\rm e}^{{- {\left( {t - \tau _{m}} \right)}/0.8}} \sin 2\pi {\left( {t - \tau _{m}} \right)}$$

(18)

where the parameters τ _m is used to adjust the time of arrival of the wave form; the functions are zero before the time-of-arrival. Figure 10 shows the wave forms proposed by Geers [1] and given by (15) and (18). Unfortunately, Geers did not specify the values of the time of arrival parameters, nor the integration duration used for the evaluation of the metrics. These parameters were estimated from the unlabeled axes plots presented by Geers as Figs. 1, 2 and 3 in his 1984 publication. The lack of a duration of application is not critical as the exponential decay of the wave forms minimizes late time contributions to the time integrals used by the Geers metrics. The times of arrival used in the present example are τ = 0.14 and τ _m = 0.12 with a time integration interval of 0 ≤ t ≤ 2 and time increments of Δt = 0.02.

Fig. 10

Idealized response histories for illustrating magnitude error

Table 4 presents a summary of the metric components for the two considered metrics, applied to this magnitude and decay rate comparison. The absolute value of the magnitude components of the Sprague and Geers (−28.4%) and Knowles and Geer (27.3%) metrics are nearly identical. However, this example illustrates an important difference between the two magnitude metrics. The Knowles and Geer magnitude metric does not report the sign of the magnitude difference, while a positive Sprague and Geers magnitude metric indicates the calculated response is greater in magnitude than the measurement, and the converse for a negative magnitude metric. This limitation of the Knowles and Geer magnitude metric appears minor when making a single comparison, it becomes more serious when a large suite of comparisons are made.

Table 4 Validation metric components for Geers magnitude and decay illustration

1.2 Phase error

To illustrate phase errors, Geers [1] offers the following analytical function:

$$c_{p} {\left( t \right)} = {\rm e}^{{- {\left( {t - \tau _{p}} \right)}}} \sin 1.6\pi {\left( {t - \tau _{p}} \right)}$$

(19)

Figure 11 shows the wave forms proposed by Geers and given by (15) and (19). The times-of-arrival used for the application of the metrics were τ = 0.14 and τ _p  = 0.04 with a time integration interval of 0 ≤ t ≤ 1.96 and time increments of Δt = 0.02.

Fig. 11

Idealized response histories for illustrating phase error

Table 5 presents a summary of the components of the two metrics, applied to this phase error comparison. For this case the Sprague and Geers magnitude metric is small at −0.7% and the phase metric is 16.2%. However, the magnitude metric for the Knowles and Gear metric is quite large at 43.2%. This large discrepancy in the magnitude metric values illustrates another difference between the two metrics. The Knowles and Gear metric does not address errors due to phase differences, rather such differences are included in the magnitude metric. Consider for example a sine and cosine wave forms with identical times-of-arrival. The Knowles and Gear magnitude metric would be about 100% while the Sprague and Geers magnitude would be nearly zero.

Table 5 Validation metric components for Geers phase difference illustration

The combining of magnitude and phase differences in the Knowles and Gear magnitude metric can be misleading. For the trivial case of the sine and cosine wave if these wave forms are interpreted as velocity or pressure wave forms, then their time integration is displacement and impulse, respectively. Imagine a structure responding to these two wave forms, although both wave forms produce the same integrated effect, i.e. displacement or impulse, the Knowles and Gear magnitude metric indicates there is a large difference, which would likely not be the case for the structure’s response.

1.3 Combined magnitude, decay rate, and phase errors

To illustrate combined magnitude, decay, and phase errors, Geers [1] offers the following analytical function:

$$c_{c} {\left( t \right)} = 1.2{\rm e}^{{- {\left( {t - \tau _{c}} \right)}/1.2}} \sin 1.6\pi {\left( {t - \tau _{c}} \right)}$$

(20)

Figure 12 shows the wave forms proposed by Geers and given by (15) and (20). The times-of-arrival used for the application of the metrics were τ = 0.14 and τ _c = 0.1 with a time integration interval of 0 ≤ t ≤ 1.96 and time increments of Δt = 0.02.

Fig. 12

Idealized response histories for illustrating combined errors in magnitude and phase

Table 6 presents a summary of metric components for the two metrics as applied to this magnitude, decay rate and phase error comparison. The Sprague and Geers magnitude metric is 30% with a 20% phase metric. The Knowles and Gear magnitude metric is 53% with a TOA metric of 22%. As noted above for the phase error case, the Knowles and Gear magnitude metric combines both magnitude and phase differences.

Table 6 Validation metric components for Geers combined magnitude and phase difference illustration

Validation metric numerical implementation

The validation metrics presented require the measurement and simulations results to be sampled at the same discrete times. This can most easily be addressed when the parameters of the validation exercise are specified, i.e. the experimental group provides the sampling interval for the measurements and the numerical simulation groups report their results at the specified sampling interval. When the sampling intervals vary, as occurred in the present suite of experiment to simulation comparisons, the simulation results can be interpolated to provide results with the same sampling rate used for the measurements.

2.1 Sprague and Geers metric

The time integrals used in the Sprague and Geers metrics are approximated by summations using simple trapezoidal integration, i.e.

$${\int\limits_a^b {f{\left( t \right)}}}{\rm d}t \approx \frac{{b - a}}{{2N}}{\sum\limits_{i = 1}^N {{\left[ {f{\left( {t_{i}} \right)} + f{\left( {t_{{i + 1}}} \right)}} \right]}}}$$

(21)

where N is the number of trapezoidal intervals such that Δt = (b − a)/N and f(t _i ) is the integrand evaluated at the indicated end of the interval.

All of the metric terms proposed by Sprague and Geers, i.e. (2) and (3), use ratios of the time integrals, so the coefficients preceding the summation in the trapezoidal integration cancel, i.e. from (2)

$$\frac{{\vartheta_{cc}}}{{\vartheta _{mm}}} \approx \frac{{{\sum_{i = 1}^N {{\left[ {c{\left( {t_{i}} \right)} + c{\left( {t_{{i + 1}}} \right)}} \right]}}}^{2}}}{{{\sum_{i = 1}^N {{\left[ {m{\left( {t_{i}} \right)} + m{\left( {t_{{i + 1}}} \right)}} \right]}}}^{2}}}$$

(22)

The use of uniform time sampling, or interpolation to uniform time sampling, greatly simplifies the metric component evaluations.

2.2 Knowles and Gear metric

The Knowles and Geer magnitude metric, (5), is expressed as a ratio of two series, and as in the Sprague and Geers metric, if uniform time sampling is used the magnitude metric evaluation is greatly simplified.

$$M_{{\rm KG}} = {\sqrt {\frac{{{\sum_{i = 1}^N {{\left( {\frac{{{\left| {m_{i}} \right|}}}{{m_{{\max}}}}} \right)}^{p}}}( \tilde{c}_{i} - m_{i})^{2}}} {{{\sum_{i = 1}^N {{\left( {\frac{{{\left| {m_{i}} \right|}}}{{m_{{\max}}}}} \right)}^{p}}}(m_{i})^{2}}}}}$$

(23)

Details of the Subject Matter Expert opinions and metric evaluations

3.1 SME responses

Table 7 presents the zero-to-one responses of the 11 SME, indicated by the letters A through L, on the agreement of the five waveform pairings; recall zero indicates poor agreement and one good agreement. The mean and standard deviation for the 11 responses are provided in the last two columns of the table. The generally low value of the standard deviations is perhaps indicative of the experts being selected from a group that has similar backgrounds/experiences with the selected waveforms, and has been working together for several years.

Table 7 Summary of Subject Matter Expert waveform pairings evaluations

3.1.1 Unsolicited SME comments

This section includes most of the unsolicited comments and qualification statements provided by the SME’s. The common thread is that since the waveforms to be compared are identified as velocity histories, some of the SME’s also used the corresponding (estimated) displacement to assess the waveforms. Also, the SME’s are aware that the metrics make no such assessment of displacement and that the intent of the SME questioning was to compare the SME evaluation with the K&G and S&G metrics. In retrospect, it would have been better not to identify the waveforms as velocity histories, as other types of wave forms, e.g. displacement and stress, may not have a corresponding integral aspect to their expert assessment.

The SME comments follow: general criteria used to judge the curves:

• Peak amplitude

• Displacement (area under curve), focusing especially on the positive phase

• Overall waveform shape

• Rise time and pulse width

• Arrival time

The Knowles and Gear metric puts a grater weight on what happens near the peak. I do too, but maybe not as much. For example, sometimes you have a very sharp and narrow spike at the peak, so the blind numerical peak value is not really representative. I tend to discount this. I put more weight on the overall shape of the main pulse, and maybe the period right after it (the main tail).

I also tend to put a fair amount of weight on displacements. Since these were not provided, I had to eyeball them. I do not know how much weight others put on displacements, but as far as I know none of the standard quantitative metrics put ANY weight on them, at least directly. (Displacements obviously affect the magnitude portions of the metrics, but indirectly.) I am not sure what you do about this.

3.1.2 Subjective evaluation of SME B’s response

Subject Matter Expert ‘B’ provide a response that consisted of two numbers for each waveform pairing, see Table 8, and the qualification statement:

Table 8 Summary of response from SME B and average of responses

My reply is based on the following assumptions. One of the waveforms is an experimental record and the other is a pre- or post-test prediction. My assessment also assumes that the gage records have been peer reviewed. From these records it is obvious that gage 1 is inconsistent (displacements) with gages 2–5 and therefore is less creditable.

When asked, SME B was unwilling to change the provided response to conform with the responses obtained from the other SME’s. The author decided to average the two numbers provided by SME B and include the average as the SME’s response. The overall low standard deviations of the SME responses perhaps justify this subjective decision.

3.2 Metric evaluations

This section presents the details of the time-of-arrival (phase), magnitude, and combined metric evaluations for the Knowles and Gear (Table 9) and Sprague and Geers (Table 10) metrics. The average of the combined metric, for the two non-symmetric evaluations, is used in the comparisons with the SME evaluations.

Table 9 Knowles and Gear metric evaluations of time-of-arrival (TOA), magnitude (MAG), and weighted metric (WM) for five waveform pairings

Table 10 Sprague and Geers metric evaluations of Phase, magnitude (MAG), and comprehensive metric (COMP) for five waveform pairings