Abstract
We present a bio-inspired fish simulation platform, which we call "Foids", to generate realistic synthetic datasets for an use in computer vision algorithm training. This is a first-of-its-kind synthetic dataset platform for fish, which generates all the 3D scenes just with a simulation. One of the major challenges in deep learning based computer vision is the preparation of the annotated dataset. It is already hard to collect a good quality video dataset with enough variations; moreover, it is a painful process to annotate a sufficiently large video dataset frame by frame. This is especially true when it comes to a fish dataset because it is difficult to set up a camera underwater and the number of fish (target objects) in the scene can range up to 30,000 in a fish cage on a fish farm. All of these fish need to be annotated with labels such as a bounding box or silhouette, which can take hours to complete manually, even for only a few minutes of video. We solve this challenge by introducing a realistic synthetic dataset generation platform that incorporates details of biology and ecology studied in the aquaculture field. Because it is a simulated scene, it is easy to generate the scene data with annotation labels from the 3D mesh geometry data and transformation matrix. To this end, we develop an automated fish counting system utilizing the part of synthetic dataset that shows comparable counting accuracy to human eyes, which reduces the time compared to the manual process, and reduces physical injuries sustained by the fish.
Supplemental Material
- John P Abraham, M Baringer, NL Bindoff, T Boyer, LJ Cheng, JA Church, JL Conroy, CM Domingues, JT Fasullo, J Gilson, et al. 2013. A review of global ocean temperature observations: Implications for ocean heat content estimates and climate change. Reviews of Geophysics 51, 3 (2013), 450--483.Google ScholarCross Ref
- University of Western Australia (UWA) Australian Institute of Marine Science (AIMS) and Curtin University. 2019. OzFish Dataset - Machine learning dataset for Baited Remote Underwater Video Stations. (2019). Google ScholarCross Ref
- Horst Bleckmann. 2004. 3-D-orientation with the octavolateralis system. Journal of Physiology-Paris 98, 1-3 (2004), 53--65.Google ScholarCross Ref
- Alexey Bochkovskiy, Chien-Yao Wang, and Hong-Yuan Mark Liao. 2020. Yolov4: Optimal speed and accuracy of object detection. arXiv preprint arXiv:2004.10934 (2020).Google Scholar
- Alisha A Brown, Marcia L Spetch, and Peter L Hurd. 2007. Growing in circles: Rearing environment alters spatial navigation in fish. Psychological Science 18, 7 (2007), 569--573.Google ScholarCross Ref
- Iain D. Couzin, Christos C. Ioannou, Güven Demirel, Thilo Gross, Colin J. Torney, Andrew Hartnett, Larissa Conradt, Simon A. Levin, and Naomi E. Leonard. 2011. Uninformed Individuals Promote Democratic Consensus in Animal Groups. Science 334, 6062 (2011), 1578--1580. arXiv:https://science.sciencemag.org/content/334/6062/1578.full.pdf Google ScholarCross Ref
- KF Cubitt, S Churchill, D Rowsell, DA Scruton, RS McKinley, et al. 2003. 3-dimensional positioning of salmon in commercial sea cages: assessment of a tool for monitoring behaviour. In Aquatic telemetry. Advances and applications. Proceedings of the fifth Conference on Fish Telemetry held in Europe, Ustica, Italy. 25--33.Google Scholar
- César Roberto de Souza, Adrien Gaidon, Yohann Cabon, and Antonio Manuel López Peña. 2017. Procedural Generation of Videos to Train Deep Action Recognition Networks. arXiv:1612.00881 [cs.CV]Google Scholar
- Ellen M Ditria, Rod M Connolly, Eric L Jinks, and Sebastian Lopez-Marcano. 2021. Annotated video footage for automated identification and counting of fish in unconstrained marine environments. Google ScholarCross Ref
- Andrew Dittman and Thomas Quinn. 1996. Homing in Pacific salmon: mechanisms and ecological basis. Journal of Experimental Biology 199, 1 (1996), 83--91.Google ScholarCross Ref
- Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. 2017. CARLA: An Open Urban Driving Simulator. arXiv:1711.03938 [cs.LG]Google Scholar
- Anders Fernö, Ingvar Huse, Jon-Erik Juell, and Åsmund Bjordal. 1995. Vertical distribution of Atlantic salmon (Salmo solar L.) in net pens: trade-off between surface light avoidance and food attraction. Aquaculture 132, 3-4 (1995), 285--296.Google ScholarCross Ref
- Ole Folkedal, Lars Helge Stien, Jonatan Nilsson, Thomas Torgersen, Jan Erik Fosseidengen, and Frode Oppedal. 2012. Sea caged Atlantic salmon display size-dependent swimming depth. Aquatic Living Resources 25, 2 (2012), 143--149.Google ScholarCross Ref
- Martin Føre, Tim Dempster, Jo Arve Alfredsen, Vegar Johansen, and David Johansson. 2009. Modelling of Atlantic salmon (Salmo salar L.) behaviour in sea-cages: A Lagrangian approach. Aquaculture 288, 3-4 (2009), 196--204.Google ScholarCross Ref
- Saeed Ghorbani, Kimia Mahdaviani, Anne Thaler, Konrad Kording, Douglas James Cook, Gunnar Blohm, and Nikolaus F. Troje. 2020. MoVi: A Large Multipurpose Motion and Video Dataset. arXiv:2003.01888 [cs.CV]Google Scholar
- Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. 2017. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision. 2961--2969.Google ScholarCross Ref
- James E Herbert-Read, Andrea Perna, Richard P Mann, Timothy M Schaerf, David JT Sumpter, and Ashley JW Ward. 2011. Inferring the rules of interaction of shoaling fish. Proceedings of the National Academy of Sciences 108, 46 (2011), 18726--18731.Google ScholarCross Ref
- A.L. Hodgkin and A.F. Huxley. 1952. A quantitative description of membrane current and its application to conduction and excitation in nerve. Journal of Physiology 117 (1952), 500--544.Google ScholarCross Ref
- I Huse and JC Holm. 1993. Vertical distribution of Atlantic salmon (Salmo salar) as a function of illumination. Journal of Fish Biology 43 (1993), 147--156.Google ScholarCross Ref
- Malthe Hvas, Ole Folkedal, and Frode Oppedal. 2021. Fish welfare in offshore salmon aquaculture. Reviews in Aquaculture 13, 2 (2021), 836--852.Google ScholarCross Ref
- Hochul Hwang, Cheongjae Jang, Geonwoo Park, Junghyun Cho, and Ig-Jae Kim. 2021. ElderSim: A Synthetic Data Generation Platform for Human Action Recognition in Eldercare Applications. IEEE Access PP (01 2021), 1--1. Google ScholarCross Ref
- Arne J Jensen, Bjørn O Johnsen, and Laila Saksgård. 1989. Temperature requirements in Atlantic salmon (Salmo salar), brown trout (Salmo trutta), and Arctic char (Salvelinus alpinus) from hatching to initial feeding compared with geographic distribution. Canadian Journal of Fisheries and Aquatic Sciences 46, 5 (1989), 786--789.Google ScholarCross Ref
- David Johansson, Kari Ruohonen, Anders Kiessling, Frode Oppedal, Jan-Erik Stiansen, Mark Kelly, and Jon-Erik Juell. 2006. Effect of environmental factors on swimming depth preferences of Atlantic salmon (Salmo salar L.) and temporal and spatial variations in oxygen levels in sea cages at a fjord site. Aquaculture 254, 1-4 (2006), 594--605.Google ScholarCross Ref
- Jon-Erik Juell. 1995. The behaviour of Atlantic salmon in relation to efficient cage-rearing. Reviews in fish biology and fisheries 5, 3 (1995), 320--335.Google Scholar
- Jon-Erik Juell and Håkan Westerberg. 1993. An ultrasonic telemetric system for automatic positioning of individual fish used to track Atlantic salmon (Salmo salar L.) in a sea cage. Aquacultural engineering 12, 1 (1993), 1--18.Google ScholarCross Ref
- Yael Katz, Kolbjørn Tunstrøm, Christos C. Ioannou, Cristián Huepe, and Iain D. Couzin. 2011. Inferring the structure and dynamics of interactions in schooling fish. Proceedings of the National Academy of Sciences 108, 46 (2011), 18720--18725. arXiv:https://www.pnas.org/content/108/46/18720.full.pdf Google ScholarCross Ref
- Yoshimi Kawai and Akiyoshi Wada. 2007. Diurnal sea surface temperature variation and its impact on the atmosphere and ocean: A review. Journal of oceanography 63, 5 (2007), 721--744.Google ScholarCross Ref
- Henri Korn and Donald S Faber. 2005. The Mauthner cell half a century later: a neurobiological model for decision-making? Neuron 47, 1 (July 2005), 13--28. Google ScholarCross Ref
- Jens Krause, Jean-Guy J Godin, and David Brown. 1996. Phenotypic variability within and between fish shoals. Ecology 77, 5 (1996), 1586--1591.Google ScholarCross Ref
- CC LINDSEY. 1978. Form, function, and locomotory habits in fish. Fish Physiology VII (1978), 1--100. https://ci.nii.ac.jp/naid/10024970754/Google Scholar
- Stefano Marras, Shaun S Killen, Jan Lindström, David J McKenzie, John F Steffensen, and Paolo Domenici. 2015. Fish swimming in schools save energy regardless of their spatial position. Behavioral ecology and sociobiology 69, 2 (2015), 219--226.Google Scholar
- Ollie Matthews, Koki Ryu, and Tarun Srivastava. 2020. Creating a Large-scale Synthetic Dataset for Human Activity Recognition. arXiv:2007.11118 [cs.CV]Google Scholar
- Thomas H. Miller, Katie Clements, Sungwoo Ahn, Choongseok Park, Eoon Hye Ji, and Fadi A. Issa. 2017. Social Status-Dependent Shift in Neural Circuit Activation Affects Decision Making. Journal of Neuroscience 37, 8 (2017), 2137--2148. arXiv:https://www.jneurosci.org/content/37/8/2137.full.pdf Google ScholarCross Ref
- AVS Murty. 1969. A theoretical approach to the attenuation coefficient of light in sea water. Indian Journal of Fisheries 16, 1&2 (1969), 151--155.Google Scholar
- Gabrielle A Nevitt, Andrew H Dittman, Thomas P Quinn, and William J Moody. 1994. Evidence for a peripheral olfactory memory in imprinted salmon. Proceedings of the National Academy of Sciences 91, 10 (1994), 4288--4292.Google ScholarCross Ref
- Lucy Odling-Smee and Victoria A Braithwaite. 2003. The role of learning in fish orientation. Fish and Fisheries 4, 3 (2003), 235--246.Google ScholarCross Ref
- Frode Oppedal, Tim Dempster, and Lars H Stien. 2011. Environmental drivers of Atlantic salmon behaviour in sea-cages: a review. Aquaculture 311, 1-4 (2011), 1--18.Google ScholarCross Ref
- DS Pavlov and AO Kasumyan. 2000. Patterns and mechanisms of schooling behavior in fish: a review. Journal of Ichthyology 40, 2 (2000), S163.Google Scholar
- Sahithi Podila and Ying Zhu. 2017a. Animating Escape Maneuvers for a School of Fish. In Proceedings of the 21st ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games (San Francisco, California) (I3D '17). Association for Computing Machinery, New York, NY, USA, Article 18, 2 pages. Google ScholarDigital Library
- Sahithi Podila and Ying Zhu. 2017b. Animating Multiple Escape Maneuvers for a School of Fish. In Proceedings of Graphics Interface 2017 (Edmonton, Alberta) (GI 2017). Canadian Human-Computer Communications Society / Société canadienne du dialogue humain-machine, 140 -- 147. Google ScholarDigital Library
- Bryan A. Reyes, Julie S. Pendergast, and Shin Yamazaki. 2008. Mammalian Peripheral Circadian Oscillators Are Temperature Compensated. Journal of Biological Rhythms 23, 1 (2008), 95--98. arXiv:https://doi.org/10.1177/0748730407311855 PMID: 18258762. Google ScholarCross Ref
- Craig Reynolds. 1999. Steering Behaviors For Autonomous Characters.Google Scholar
- Craig W Reynolds. 1987. Flocks, herds and schools: A distributed behavioral model. In Proceedings of the 14th annual conference on Computer graphics and interactive techniques. 25--34. Google ScholarDigital Library
- William Wallace Reynolds and Martha Elizabeth Casterlin. 1979. Behavioral thermoregulation and the "final preferendum" paradigm. American zoologist 19, 1 (1979), 211--224.Google Scholar
- Alzayat Saleh, Issam H. Laradji, Dmitry A. Konovalov, Michael Bradley, David Vazquez, and Marcus Sheaves. 2020. A realistic fish-habitat dataset to evaluate algorithms for underwater visual analysis. Scientific Reports 10, 1 (Sep 2020). Google ScholarCross Ref
- Daiki Satoi, Mikihiro Hagiwara, Akira Uemoto, Hisanao Nakadai, and Junichi Hoshino. 2016. Unified Motion Planner for Fishes with Various Swimming Styles. ACM Transactions on Graphics (Proc. SIGGRAPH) 35, 4 (2016). Google ScholarDigital Library
- Michael Sfakiotakis, David M Lane, and J Bruce C Davies. 1999. Review of fish swimming modes for aquatic locomotion. IEEE Journal of oceanic engineering 24, 2 (1999), 237--252.Google ScholarCross Ref
- Kingsley Stephens, Binh Pham, and Aster Wardhani. 2003. Modelling Fish Behaviour. In Proceedings of the 1st International Conference on Computer Graphics and Interactive Techniques in Australasia and South East Asia (Melbourne, Australia) (GRAPHITE '03). Association for Computing Machinery, New York, NY, USA, 71--78. Google ScholarDigital Library
- AM Sutterlin, KJ Jokola, and B Holte. 1979. Swimming behavior of salmonid fish in ocean pens. Journal of the Fisheries Board of Canada 36, 8 (1979), 948--954.Google ScholarCross Ref
- Penny Tarling, Mauricio Cantor, Albert Clapés, and Sergio Escalera. 2021. Deep learning with self-supervision and uncertainty regularization to count fish in underwater images. (2021). arXiv:2104.14964 [cs.CV]Google Scholar
- Xiaoyuan Tu and Demetri Terzopoulos. 1994. Artificial Fishes: Physics, Locomotion, Perception, Behavior. In Proceedings of the 21st Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '94). Association for Computing Machinery, New York, NY, USA, 43--50. Google ScholarDigital Library
- Gül Varol, Javier Romero, Xavier Martin, Naureen Mahmood, Michael J. Black, Ivan Laptev, and Cordelia Schmid. 2017. Learning from Synthetic Humans. In CVPR.Google Scholar
- C.A.G. Wiersma and K. Ikeda. 1964. Interneurons commanding swimmeret movements in the crayfish, Procambarus clarki (girard). Comparative Biochemistry and Physiology 12, 4 (1964), 509--525. Google ScholarCross Ref
- Steven J Zottoli. 1978. Comparison of Mauthner cell size in teleosts. Journal of Comparative Neurology 178, 4 (1978), 741--757.Google ScholarCross Ref
Index Terms
- Foids: bio-inspired fish simulation for generating synthetic datasets
Recommendations
Robust Fish Enumeration by Multiple Object Tracking in Overhead Videos
Pattern Recognition. ICPR International Workshops and ChallengesAbstractIn this paper, we design a framework called detection-tracking fish counting (DTFC) for overhead videos of fish moving in shallow water. DTFC includes three steps, fish tracking, counting zone generation and fish counting. In this work, we adopt ...
Automatic live fingerlings counting using computer vision
Highlights- Kalman Filter and Mixture of Gaussians are efficient for tracking fish.
- Data ...
AbstractFish counting is still a rudimentary process in most fisheries in Brazil. Current solutions are generally unaffordable for small and medium-size producers; hence, in order to provide a low-cost solution, this paper proposes a new ...
Numerical study of the correlation between fish school arrangement and propulsive performance
AbstractWe focus on the distances in flow direction and its vertical direction on three fish school arrangements, and correlate the fish school in uniform flow with fluid forces which can be divided into two components, drag and thrust, by three-...
Comments