Elsevier

Advances in Engineering Software

Volume 68, February 2014, Pages 49-55
Advances in Engineering Software

A generic framework for data acquisition and transmission

https://doi.org/10.1016/j.advengsoft.2013.11.003Get rights and content

Highlights

  • •

    We propose a configurable Data Acquisition and Transmission Framework (DATF).

  • •

    The DATF has 4 parts: a controller, a set of extractors, an assembler and a loader.

  • •

    We design the hierarchy of the four parts.

  • •

    Modular design and shared library technique make the DATF extensible and flexible.

Abstract

Online applications such as decision support systems depend on data collected from multiple sources. We develop a generic data acquisition and transmission framework by modularizing the repetitive functions. Other than data acquisition and distribution with necessary transformation, the framework can act as a middle storage when the sources cannot connect to the destination directly. We designed three types of data extractors to accommodate the data acquisition from the file system, the net protocol and the database. After being collected by the extractors, the data is processed by an assembler module, to fit the target’s data structure. The data is inserted into the database by a loader module, which gets data from the assembler module. The assembler module and the loader module are controlled by a monitor and controller module. These modules are highly configurable and they form a 3-Level hierarchy. Taking advantage of modular design and shared library technique, the framework is extensible and flexible.

Introduction

Online applications such as decision support systems demand a large amount of data, which is obtained from multiple sources and inserted into the target’s database or data warehouse [1], [2]. The process is often referred to Extraction, Transformation and Load (ETL) [3]. For the target applications, the ETL generally loads data into the database, especially the data warehouse [4]. Some functionalities in ETL are reusable in other applications. We are working on a highly configurable and extensible Data Acquisition and Transmission Framework (DATF) for general purpose.

Data acquisition and transmission frameworks have been established in many fields [5], [6], [7]. There are also mature products and relational database and flat files are common sources [3]. However, our application for a wind power base requires information from wind farms, wind turbines and etc. The source information carriers are database, file media and messages transferred via network.

We aim at reducing the workload and simplify the transplantation of the framework between different applications. Furthermore, it should be unnecessary to restart the whole program when adding or removing a function module. To achieve this, the functionalities are modularized, and each module can be configured easily by editing a text file.

Section snippets

Working model

Depending on the system configuration, the working models can be classified into two categories: direct connection and indirect connection.

In many cases, which can be classified as direct connection, there is no isolation between the target application and the data sources (Fig. 1). Being considered as a part of the target application, the DATF connect the data sources with the application by extracting data from the external sources and sending it to the target database after transformation if

System structure

The structure of the DATF consists of four major parts as shown in Fig. 3: the Monitor and Controller (MAC), the extractors, the assembler and the loader. The DATF integrates the pieces of data retrieved from the sources into a whole one which can be inserted into a data table of the target database. And then it inserts the final records into the table. We call the above two sections a task. An instance, usually a process, of the assembler cooperates with a loader instance to serve a task. We

Module design

To make the DATF extensible for different applications, we separate the control function from the business logic. The user only needs to provide a set of configuration files and several modules to complete a task.

For readability and simplicity, we focus on the data acquisition and transmission in this paper. Issues such as error handling and security policy are beyond the scope of this paper.

Example

The DATF was first introduced when we were developing the Monitoring System for Chinese Jiuquan Wind Power Base in Gansu province in 2011. The DATF loads running data of wind farms, step-up substations, wind turbines and anemometer towers. External databases store the data of the wind farms, step-up substations and anemometer towers. The data of the wind turbines are recorded in “E-text” files located on an FTP server.

The organization of data in the monitoring system contains two parts as shown

Conclusion

We present a generic data acquisition and transmission framework in this paper. We design two working models, i.e. direct connection and indirect connection. To implement the same functionalities of an ETL process in different applications, we propose a framework based on the modular design method, and the framework is suitable for both models. We further analyze the hierarchy of the modules in the framework, and reduce the transplanting work load by utilizing configuration files. The framework

References (22)

  • K. Mershad et al.

    A framework for secure and efficient data acquisition in vehicular ad hoc networks

    IEEE Trans Veh Technol

    (2013)

    • Cited by (2)

    • A holistic framework for engineering simulation platform development gluing open-source and home-made software resources

      2014, Advances in Engineering Software
      Citation Excerpt :

      In the development of the finite element analysis system TopFEM, a framework is presented to take advantage of the topological data structure together with object-oriented programming concepts to handle a variety of finite element problems, in an efficient, but generic fashion [2]. Modularizing the repetitive functions, data acquisition and transmission frameworks have been established in online applications field, which is extensible and flexible [3]. A Web-based MDO (Multidisciplinary Design Optimization) framework has been developed to assist in the analysis and optimization of highly complex system [4].

    • Acquisition of operational data in industrial laundry facilities

      2014, Chemical Engineering Transactions
    View full text
    Copyright © 2013 Elsevier Ltd. All rights reserved.