Abstract
Despite algorithmic advances in the field of image processing as well as modern computer architectures, automated sensor data processing that guarantees reliable and robust information retrieval in dynamic environmental and varying flight conditions is still a challenging task within unmanned surveillance and reconnaissance missions. In our paper we will elaborate the reasons and propose a promising way out by adapting to variable environmental conditions and states of the UAS platform in terms of the dedicated usage of specialized sensor data processing chains.
However, these specialized chains must be used within their operation space. Otherwise, their performance in terms of detection precision and recall will degrade. To overcome this drawback, we propose to apply chain performance models based on Bayesian Networks (BNs). The evaluation of the BNs takes place during the flight depending on environmental influences. Accordingly, a performance probability can be predicted for each chain, which is used for an automatic chain selection.
We validate our approach within a real flight Search and Rescue scenario (SAR). To compare generalized and specialized chains, we conducted several flight experiments with an EO/IR mission sensor setup: a) to annotated and preselect/filter training-datasets, used for transfer-learning of the Convolutional Neural Networks (CNNs), and b) to derive test datasets in different environmental situations and with varying platform/sensor states. The sensor data comprises variations in illumination and meteorological conditions, photographic conditions (e.g., different sensor elevation angles and ground sample distances) as well as topographic conditions. We provide a comprehensive insight into the detection results derived. Based on these results we conclude that a targeted use of specialized CNNs can outperform generalized CNNs.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Larger CNNs refers to networks with higher resolution or multimodal input images and a denser network structure (increased number of nodes, layers).
- 2.
The ideas of specialized chains can furthermore be used to cope with the limited computational capacities on UAVs. Additionally adapted sensor requirements (e.g. reduced resolution/GSD, sampling/frame rate, number of channels/spectra, color depth) can be used to form runtime-optimized chain specializations/permutations but with usually lower performance.
- 3.
Depending on the application/use, separate metrics may also have different relevance with respect to an actual, situation-optimal evaluation of chain performance. For example, the selected threshold value of the classification generally influences the precision/false positive rate on the one hand and the recall on the other. A distinction of different performance criteria is therefore useful for the use of UAVs especially in safety-critical missions [21].
- 4.
Chain management was primary described within our Sensor-Perception-Management System approach (SPMS, [22]).
- 5.
In addition to the transmission of sensor images, detection, identification or tracking of objects (vehicles, persons, obstacles) can be selected as perception-task types.
- 6.
- 7.
For targeted stopping, starting, and prioritizing of chains, in addition to runtime parameters, task IDs, chain IDs and the currently available resources are stored by the PC in a database (bookkeeping). During the de-/activation of perception tasks via the perception solver as well as during the updating of auctions, these databases are evaluated for the scheduling of the chains, e.g., by stopping or prioritizing the executed processes (chain PIDs) for each task ID.
- 8.
Optional algorithms for object and obstacle detection are using to the LIDAR range-sensor, but not relevant for the evaluation within this paper.
- 9.
Traditional object recognition and detection methods for UAVs are using dedicated features such as SIFT (Scale-Invariant Feature Transform, [26, 27]), SURF (Speeded-Up Robust Features, [28]), and HOG (Histogram of Oriented Gradient, [29, 30]) or rely on the evaluation the foreground/background (e.g., color or edge based) and geometry constraints (BLOB, [31]). These methods use local features, thresholds and assumptions regarding the environment that were created by hand. Therefore, traditional image processing methods are error-sensitive or generally only valid for a very limited state space ([32]).
- 10.
Consequently, there are neither separate classification or detection modules that should be synchronized with each other nor region proposals necessary.
- 11.
E.g., with path-planning from a Mission-Management-system [13].
- 12.
Both foreground and background.
References
Tan, M., Pang, R., Le, Q.V.: EfficientDet: scalable and efficient object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10778–10787 (2020). https://doi.org/10.1109/CVPR42600.2020.01079
Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollar, P.: Focal loss for dense object detection. IEEE Trans. Pattern Anal. Mach. Intell. 42, 318–327 (2017). https://doi.org/10.48550/arxiv.1708.02002
Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: unified, real-time object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 779–788 (2015). https://doi.org/10.48550/arxiv.1506.02640
Bochkovskiy, A., Wang, C.-Y., Liao, H.-Y.M.: YOLOv4: optimal speed and accuracy of object detection (2020). https://doi.org/10.48550/arxiv.2004.10934
Wang, C.-Y., Bochkovskiy, A., Liao, H.-Y.M.: YOLOv7: trainable bag-of-freebies sets new state-of-the-art for real-time object detectors (2022). https://doi.org/10.48550/arxiv.2207.02696
Lin, T.-Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_48
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database, pp. 248–255 (2010). https://doi.org/10.1109/CVPR.2009.5206848
Everingham, M., Van Gool, L., Williams, C.K.I., Winn, J., Zisserman, A.: The pascal visual object classes (VOC) challenge. Int. J. Comput. Vis. 88, 303–338 (2010). https://doi.org/10.1007/S11263-009-0275-4
Wu, X., Li, W., Hong, D., Tao, R., Du, Q.: Deep learning for UAV-based object detection and tracking: a survey. IEEE Geosci. Remote Sens. Mag. 10, 91–124 (2021). https://doi.org/10.1109/mgrs.2021.3115137
Valappil, N.K., Memon, Q.A.: CNN-SVM based vehicle detection for UAV platform. Int. J. Hybrid Intell. Syst. 17, 59–70 (2021). https://doi.org/10.3233/HIS-210003
Srivastava, S., Narayan, S., Mittal, S.: A survey of deep learning techniques for vehicle detection from UAV images. J. Syst. Archit. 117 (2021). https://doi.org/10.1016/J.SYSARC.2021.102152
Tian, G., Liu, J., Yang, W.: A dual neural network for object detection in UAV images. Neurocomputing 443, 292–301 (2021). https://doi.org/10.1016/J.NEUCOM.2021.03.016
Russ, M., Stütz, P.: Application of a probabilistic market-based approach in UAV sensor & perception management. In: Information Fusion (16th FUSION) (2013)
Russell, S., Norvig, P.: Artificial Intelligence: A Modern Approach (2020)
Tanenbaum, A.S., van Steen, M.: Distributed Systems. CreateSpace Independent Publishing Platform (2017)
Russ, M., Schmitt, M., Hellert, C., Stütz, P.: Airborne sensor and perception management: experiments and results for surveillance UAS. In: AIAA Infotech@aerosp. Conference, pp. 1–16 (2013). https://doi.org/10.2514/6.2013-5144
Touvron, H., Cord, M., Sablayrolles, A., Synnaeve, G., Jégou, H.: Going deeper with image transformers. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 32–42 (2021). https://doi.org/10.48550/arxiv.2103.17239
Khan, A., Sohail, A., Zahoora, U., Qureshi, A.S.: A survey of the recent architectures of deep convolutional neural networks. Artif. Intell. Rev. 53(8), 5455–5516 (2020). https://doi.org/10.1007/s10462-020-09825-6
Vaddi, S., Kim, D., Kumar, C., Shad, S., Jannesari, A.: Efficient object detection model for real-time UAV application. Comput. Inf. Sci. 14, 45 (2021). https://doi.org/10.5539/CIS.V14N1P45
Wang, C.-Y., Bochkovskiy, A., Liao, H.-Y.M.: Scaled-YOLOv4: Scaling Cross Stage Partial Network (2021)
Hrabia, C.E., Hessler, A., Xu, Y., Brehmer, J., Albayrak, S.: EffFeu project: efficient operation of unmanned aerial vehicles for industrial fire fighters. In: Proceedings of the 2018 ACM International Conference on Mobile Systems, Applications and Services, DroNet 2018, pp. 33–38 (2018). https://doi.org/10.1145/3213526.3213533
Russ, M., Stütz, P.: Airborne sensor and perception management: a conceptual approach for surveillance UAS. In: Information Fusion (15th FUSION) (2012)
Sadaoui, S., Shil, S.K.: A multi-attribute auction mechanism based on conditional constraints and conditional qualitative preferences. J. Theor. Appl. Electron. Commer. Res. 11, 1–25 (2016). https://doi.org/10.4067/S0718-18762016000100002
Shil, S.K., Mouhoub, M., Sadaoui, S.: Winner determination in multi-attribute combinatorial reverse auctions. In: Arik, S., Huang, T., Lai, W.K., Liu, Q. (eds.) ICONIP 2015. LNCS, vol. 9491, pp. 645–652. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-26555-1_73
Bichler, M., Kalagnanam, J.: Configurable offers and winner determination in multi-attribute auctions. Eur. J. Oper. Res. 160, 380–394 (2005). https://doi.org/10.1016/j.ejor.2003.07.014
Xi, C.J., Guo, S.X.: Image target identification of UAV based on SIFT. Procedia Eng. 15, 3205–3209 (2011). https://doi.org/10.1016/J.PROENG.2011.08.602
Chen, X., Meng, Q.: Vehicle detection from UAVs by using SIFT with implicit shape model. In: Proceedings of the 2013 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2013, pp. 3139–3144 (2013). https://doi.org/10.1109/SMC.2013.535
Zhao, Y., Pei, H.: An improved vision-based algorithm for unmanned aerial vehicles autonomous landing. Phys. Procedia 33, 935–941 (2012). https://doi.org/10.1016/J.PHPRO.2012.05.157
Blondel, P., Potelle, A., Pégard, C., Lozano, R.: How to improve the HOG detector in the UAV context. IFAC Proc. 46, 46–51 (2013). https://doi.org/10.3182/20131120-3-FR-4045.00009
Zhang, G., Gao, F., Liu, C., Liu, W.: A pedestrian detection method based on SVM classifier and optimized Histograms of Oriented Gradients feature. In: 2010 Sixth International Conference on Natural Computation, pp. pp. 3257–3260. IEEE (2010). https://doi.org/10.1109/ICNC.2010.5582537
Jȩdrasiak, K., Nawrat, A.: Image recognition technique for unmanned aerial vehicles. In: Bolc, L., Kulikowski, J.L., Wojciechowski, K. (eds.) ICCVG 2008. LNCS, vol. 5337, pp. 391–399. Springer, Heidelberg (2009). https://doi.org/10.1007/978-3-642-02345-3_38
Ramachandran, A., Sangaiah, A.K.: A review on object detection in unmanned aerial vehicle surveillance. Int. J. Cogn. Comput. Eng. 2, 215–228 (2021). https://doi.org/10.1016/J.IJCCE.2021.11.005
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Ruß, M., Stütz, P. (2023). Airborne Sensor and Perception Management. In: Mazal, J., et al. Modelling and Simulation for Autonomous Systems. MESAS 2022. Lecture Notes in Computer Science, vol 13866. Springer, Cham. https://doi.org/10.1007/978-3-031-31268-7_11
Download citation
DOI: https://doi.org/10.1007/978-3-031-31268-7_11
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-31267-0
Online ISBN: 978-3-031-31268-7
eBook Packages: Computer ScienceComputer Science (R0)