A mathematical framework to optimize ATR systems with non-declarations and sensor fusion

https://doi.org/10.1016/j.cor.2006.09.012Get rights and content

Abstract

Combat identification is one example where incorrect automatic target recognition (ATR) output labels may have substantial decision costs. For example, the incorrect labeling of hostile targets vs. friendly non-targets may have high costs; yet, these costs are difficult to quantify. One way to increase decision confidence is through fusion of data from multiple sources or from multiple looks through time. Numerous methods have been published to determine a Bayes’ optimal fusion decision if decision costs are well known. This research presents a novel mathematical programming ATR evaluation framework. A new objective function inclusive of time is introduced to optimize and compare ATR systems. Constraints are developed to enforce both decision maker preferences and traditional engineering measures of performance. This research merges rejection and receiver operating characteristic (ROC) analysis by incorporating rejection and ROC thresholds as decision variables. The rejection thresholds specify non-declaration regions, while the ROC thresholds explore viable true positive and false positive tradeoffs for output target labels. This methodology yields an optimal ATR system subject to decision maker constraints without using explicit costs for each type of output decision. A sample application is included for the fusion of two channels of collected polarized radar data for 10 different ground targets. A Boolean logic and probabilistic neural network fusion method are optimized and compared. Sensitivity analysis of significant performance parameters then reveals preferred regions for each of the fusion algorithms.

Introduction

The purpose of this paper is to present the concepts of a new methodology to optimize receiver operating characteristic (ROC) and rejection thresholds, for an automatic target recognition (ATR) system subject to non-declarations. As introduced by Laine and Bauer [1] and refined within [2], this methodology is formulated to optimize a decision maker's primary objective, while constrained by other key performance preferences. This optimization framework may help determine optimal parameter settings and may be useful for evaluation and comparison of different classifiers. While the proposed methodology is applicable to many decisions with high-stake consequences, our primary research application and focus is ATR used for combat identification (CID). We consider the specific scenario in which a system attempts to identify potential enemy targets, in the presence of friend or neutral objects. The cost of incorrect declarations between these two classes may be substantial, with the most extreme misclassification leading to friendly fire. Thus, if a decision is too risky, given the current information available, a non-declaration or unknown label may also be an acceptable system output. By performing this research insight may be gained to help optimize ATR systems or other decision processes where after a “non-declaration” or “ND” label is obtained, subsequent collection and fusion of new data is performed.

In Section 2, we provide a brief background for ATR, ROC analysis and rejection. Section 3 presents the optimization framework. In Section 4, we provide an application of the framework to collected radar data, and fusion models, along with experimental results for optimization across ROC and rejection thresholds. Section 5 presents a sensitivity analysis of some of the key environmental ATR system variables. Finally, Section 6 provides a summary of our findings and recommendations for further research.

Section snippets

Background for ATR, ROC analysis and rejection

Many classification problems can be modeled at the top-level using two classes, whereby either the desired class is identified or not. For example, an ATR system may declare an unknown object as “target” or “non-target,” where “target” includes enemy assets and “non-target” may include clutter, neutral or friendly forces. Yet, before a target is declared and engaged, the USAF requires a minimum level of confidence [3], [4] to be obtained. Consequently, an ATR system forcing two decision labels

Mathematical programming formulation for ATR optimization

Since arguments have been presented in Section 2 suggesting an ATR system must provide a minimum of three classes, may have different acceptable levels for different error types, and must be accomplished in a timely manner, a new mathematical programming formulation is presented within this section to incorporate this information. This formulation also addresses the difficulty of obtaining relative error costs across critical and non-critical errors by not using explicit misclassification

Experiment and results

In this paper we use a designed experiment with an ATR system allowed to obtain up to five looks of each vehicle described within Table 2 to be identified as class “TOD,” “OH,” “FN” or “ND”. The objective is to maximize the true positive identifications per look, subject to constraints. Without consideration of physical or monetary budget constraints, the optimization framework from the previous section is used to determine optimal ROC and rejection thresholds with respect to the three

Sensitivity analysis of ATR optimization

Two primary goals are undertaken in this sensitivity analysis. The first goal seeks to determine where each fusion method may be preferred. Since the maximum TPR is estimated by evaluation of the fusion algorithms given limited data, determining where the fusion performance is relatively equivalent is also desired. The second goal of the sensitivity analysis will attempt to characterize the infeasibility regions associated with each of the fusion models and compare these conditions. This

Summary and conclusion

In this paper we presented a new optimization strategy for rejection and ROC thresholds to maximize a preferred objective while constrained by other requirements. An objective function to maximize PTP(x) per time was presented. Yet, further ATR system user input may be required to properly refine the objective function, to ensure “true positive IDs per time” is the best objective to optimize. An advantage of our optimization methodology is the development of acceptable constraints vs.

Acknowledgments

The authors would like to thank Mr. Charles Sadowski from USAF Air Combat Command (ACC/DR) for his insight of ATR, and also are grateful for the comments provided by the anonymous reviewers to significantly improve this paper. This research was jointly sponsored by ACC/DR and AFOSR under Grant #NMIPR045203616.

References (42)

  • G. Fumera et al.

    Reject option with multiple thresholds

    Pattern Recognition

    (2000)
  • D.F. Specht

    Probabilistic neural networks

    Neural Networks

    (1990)
  • T.I. Laine et al.

    A mathematical framework for the optimization of rejection and ROC thresholds in the fusion of correlated sensor data

    Proceedings of SPIE

    (2004)
  • Laine TI. Optimization of automatic target recognition with a reject option using fusion and correlated sensor data,...
  • Department of the Air Force (DAF). Air Warfare. AFDD2-1, Washington, DC: HQ USAF;...
  • Department of the Air Force (DAF). USAF intelligence targeting guide. AFP 14-210. Washington, DC: HQ USAF;...
  • Alsing SG. The evaluation of competing classifiers. PhD dissertation, Air Force Institute of Technology,...
  • Albrecht TW. Combat identification with sequential observations, rejection option, and out-of-library targets. PhD...
  • Department of the Air Force (DAF). Intelligence, surveillance, and reconnaissance operations. AFDD2-5.2. Washington;...
  • Hall DL, Llinas J, editors. Handbook of multisensor data fusion. New York: CRC Press;...
  • J.A. Swets

    Signal detection and recognition by human observers

    (1964)
  • J.A. Swets et al.

    Psychological science can improve diagnostic decisions

    Psychological Science in the Public Interest

    (2000)
  • C.K. Chow

    On optimum rejection error and reject tradeoff

    IEEE Transactions on Information Theory

    (1970)
  • P.A. Devijver et al.

    Pattern recognition—a statistical approach

    (1982)
  • Haspert KJ. Optimum ID sensor fusion for multiple target types. Institute for Defense Analysis (IDA) Document D-2451,...
  • T.D. Ross et al.

    Extensibility and other model-based ATR evaluation concepts

    Proceedings of SPIE

    (1997)
  • S.G. Alsing et al.

    Three-dimensional receiver operating characteristic trajectory concepts for the evaluation of target recognition algorithms faced with the unknown target detection problem

    Proceedings of SPIE

    (1999)
  • A. Wald

    Sequential analysis

    (1947)
  • W.W. Peterson et al.

    The theory of signal detectability

    Transactions of the IRE Professional Group on Information Theory

    (1954)
  • D. Mossman

    Three-way ROCs

    Medical decision making

    (1999)
  • D.J. Hand et al.

    A simple generalisation of the area under the ROC curve for multiple class classification problems

    Machine Learning

    (2001)
  • The views expressed in this paper are those of the authors and do not reflect official policy or position of the United States Air Force, Department of Defense, or United States Government.

    View full text