A mathematical framework to optimize ATR systems with non-declarations and sensor fusion☆
Introduction
The purpose of this paper is to present the concepts of a new methodology to optimize receiver operating characteristic (ROC) and rejection thresholds, for an automatic target recognition (ATR) system subject to non-declarations. As introduced by Laine and Bauer [1] and refined within [2], this methodology is formulated to optimize a decision maker's primary objective, while constrained by other key performance preferences. This optimization framework may help determine optimal parameter settings and may be useful for evaluation and comparison of different classifiers. While the proposed methodology is applicable to many decisions with high-stake consequences, our primary research application and focus is ATR used for combat identification (CID). We consider the specific scenario in which a system attempts to identify potential enemy targets, in the presence of friend or neutral objects. The cost of incorrect declarations between these two classes may be substantial, with the most extreme misclassification leading to friendly fire. Thus, if a decision is too risky, given the current information available, a non-declaration or unknown label may also be an acceptable system output. By performing this research insight may be gained to help optimize ATR systems or other decision processes where after a “non-declaration” or “ND” label is obtained, subsequent collection and fusion of new data is performed.
In Section 2, we provide a brief background for ATR, ROC analysis and rejection. Section 3 presents the optimization framework. In Section 4, we provide an application of the framework to collected radar data, and fusion models, along with experimental results for optimization across ROC and rejection thresholds. Section 5 presents a sensitivity analysis of some of the key environmental ATR system variables. Finally, Section 6 provides a summary of our findings and recommendations for further research.
Section snippets
Background for ATR, ROC analysis and rejection
Many classification problems can be modeled at the top-level using two classes, whereby either the desired class is identified or not. For example, an ATR system may declare an unknown object as “target” or “non-target,” where “target” includes enemy assets and “non-target” may include clutter, neutral or friendly forces. Yet, before a target is declared and engaged, the USAF requires a minimum level of confidence [3], [4] to be obtained. Consequently, an ATR system forcing two decision labels
Mathematical programming formulation for ATR optimization
Since arguments have been presented in Section 2 suggesting an ATR system must provide a minimum of three classes, may have different acceptable levels for different error types, and must be accomplished in a timely manner, a new mathematical programming formulation is presented within this section to incorporate this information. This formulation also addresses the difficulty of obtaining relative error costs across critical and non-critical errors by not using explicit misclassification
Experiment and results
In this paper we use a designed experiment with an ATR system allowed to obtain up to five looks of each vehicle described within Table 2 to be identified as class “TOD,” “OH,” “FN” or “ND”. The objective is to maximize the true positive identifications per look, subject to constraints. Without consideration of physical or monetary budget constraints, the optimization framework from the previous section is used to determine optimal ROC and rejection thresholds with respect to the three
Sensitivity analysis of ATR optimization
Two primary goals are undertaken in this sensitivity analysis. The first goal seeks to determine where each fusion method may be preferred. Since the maximum TPR is estimated by evaluation of the fusion algorithms given limited data, determining where the fusion performance is relatively equivalent is also desired. The second goal of the sensitivity analysis will attempt to characterize the infeasibility regions associated with each of the fusion models and compare these conditions. This
Summary and conclusion
In this paper we presented a new optimization strategy for rejection and ROC thresholds to maximize a preferred objective while constrained by other requirements. An objective function to maximize per time was presented. Yet, further ATR system user input may be required to properly refine the objective function, to ensure “true positive IDs per time” is the best objective to optimize. An advantage of our optimization methodology is the development of acceptable constraints vs.
Acknowledgments
The authors would like to thank Mr. Charles Sadowski from USAF Air Combat Command (ACC/DR) for his insight of ATR, and also are grateful for the comments provided by the anonymous reviewers to significantly improve this paper. This research was jointly sponsored by ACC/DR and AFOSR under Grant #NMIPR045203616.
References (42)
- et al.
Reject option with multiple thresholds
Pattern Recognition
(2000) Probabilistic neural networks
Neural Networks
(1990)- et al.
A mathematical framework for the optimization of rejection and ROC thresholds in the fusion of correlated sensor data
Proceedings of SPIE
(2004) - Laine TI. Optimization of automatic target recognition with a reject option using fusion and correlated sensor data,...
- Department of the Air Force (DAF). Air Warfare. AFDD2-1, Washington, DC: HQ USAF;...
- Department of the Air Force (DAF). USAF intelligence targeting guide. AFP 14-210. Washington, DC: HQ USAF;...
- Alsing SG. The evaluation of competing classifiers. PhD dissertation, Air Force Institute of Technology,...
- Albrecht TW. Combat identification with sequential observations, rejection option, and out-of-library targets. PhD...
- Department of the Air Force (DAF). Intelligence, surveillance, and reconnaissance operations. AFDD2-5.2. Washington;...
- Hall DL, Llinas J, editors. Handbook of multisensor data fusion. New York: CRC Press;...
Signal detection and recognition by human observers
Psychological science can improve diagnostic decisions
Psychological Science in the Public Interest
On optimum rejection error and reject tradeoff
IEEE Transactions on Information Theory
Pattern recognition—a statistical approach
Extensibility and other model-based ATR evaluation concepts
Proceedings of SPIE
Three-dimensional receiver operating characteristic trajectory concepts for the evaluation of target recognition algorithms faced with the unknown target detection problem
Proceedings of SPIE
Sequential analysis
The theory of signal detectability
Transactions of the IRE Professional Group on Information Theory
Three-way ROCs
Medical decision making
A simple generalisation of the area under the ROC curve for multiple class classification problems
Machine Learning
Cited by (7)
An analytical and probabilistic model with concordance for detecting mine-like objects with mirror symmetry
2021, Journal of Defense Analytics and LogisticsInformation Processing Simulation and Performance Evaluation for Optical Imaging Seeker
2019, Unmanned System TechnologiesA confidence paradigm for classification systems with out-of-library considerations
2012, Intelligent Decision TechnologiesBridging the fuzzy, neural and evolutionary paradigms for automatic target recognition
2011, Applied IntelligenceA confidence paradigm for classification systems
2011, Naval Research LogisticsAn Entropy-based Scheme for Automatic Target Recognition
2010, The Journal of Defense Modeling and Simulation: Applications, Methodology, Technology
- ☆
The views expressed in this paper are those of the authors and do not reflect official policy or position of the United States Air Force, Department of Defense, or United States Government.