Software Architecture with Flexibility for the Data-Intensive Fish4Knowledge Project

Abstract

The software architecture used in the Fish4Knowledge project allowed us to produce a system able to analyze 3 years of video footage and represent this data with an interface suitable for end-users (marine biologists/ecologists). To achieve this, the architecture design focused on four problems: (1) Data-Intensiveness, which allowed the system to process and store 528624 video clips of 10 min, resulting in 1445.41M fish observations. (2) Flexibility, which gave the developer freedom to design their own solutions within the larger system. (3) Dependency, which provided clear definitions of the output of the subsystems allowing researchers to work with each other’s outputs. (4) Trust, which was important for the end-user to understand the uncertainties in the system and how to deal with them. The overall design of the final system used the database as means of communication for the different software components in order to deal with these challenges In this case, developers only had to make a database connection to obtain data they depended on. The main lesson learned in this project is that stable database definitions and visualizations are very important in Data-Intensive projects.