A robust framework for real-time distributed processing of satellite data

doi:10.1016/j.jpdc.2005.12.001

Journal of Parallel and Distributed Computing

Volume 66, Issue 3, March 2006, Pages 403-418

https://doi.org/10.1016/j.jpdc.2005.12.001 Get rights and content

Abstract

It is estimated that future satellite instruments such as the Advanced Baseline Imager (ABI) and the Hyperspectral Environmental Suite (HES) on the GOES-R series of satellites will provide raw data volume of about 1.5 Terabyte per day. Due to the high data rate, satellite ground data processing will require considerable computing power to process data in real-time. Cluster technologies employing a multi-processor system present the only current economically viable option. To sustain high levels of system reliability and operability in a cluster-oriented operational environment, a fault-tolerant data processing framework is proposed to provide a platform for encapsulating science algorithms for satellite data processing. The science algorithms together with the framework are hosted on a Linux cluster.

In this paper we present an architectural model and a system prototype for providing performance, reliability, and scalability of candidate hardware and software for a satellite data processing system. Furthermore, benchmarking results are presented for a selected number of science algorithms for the Geosynchronous Imaging Fourier Transform Spectrometer (GIFTS) instrument showing that considerable performance can be gained without sacrificing the reliability and high availability constraints imposed on the operational cluster system.

Section snippets

Shahram Tehranian holds a M.S. in Fluid Dynamics from the Royal Institute of Technology, Stockholm, Sweden, and a M.S. in Computer Science from George Mason University, Virginia, USA. His research interests are high performance scientific computing on parallel and distributed platforms and his recent research focus has been on developing high performance data processing and archival systems for the National Oceanic and Atmospheric Administration (NOAA) using Linux cluster technology.

References (29)

K. Hawick et al.
Distributed frameworks and parallel algorithms for processing large-scale geographic data
Parallel Computing
(2003)
R. Batchu et al.
MPI/FT: architecture and taxonomies for fault-tolerant, message-passing middleware for performance-portable parallel computing
Boost serialization library:...
A. Bouteiller et al.
MPICH-V2: a fault tolerant MPI for volatile nodes based on the pessimistic sender based message logging
Data redistribution and remote method invocation in parallel component architectures, 19th IEEE International Parallel...
R.K. Garcia et al.
Design studies for real-time distributed processing of infrared hyperspectral image data
R.K. Garcia et al.
Component-oriented design studies for efficient processing of hyperspectral infrared imager data
GIFTS Downlink Subsystem User's Guide, Version 1.2, Document Reference 1201-USRG-001, ASRC Aerospace Corp., Maryland,...
GOES_R First Hyperspectral Sounder Data Compression Meeting, January 2002,...
R. Gordon
Essential JNI: Java Native Interface
(1998)

W. Gropp, S. Huss-Lederman, A. Lumsdaine, E. Lusk, MPI—The Complete Reference, vol. 2, The MPI Extensions, MIT Press,...

P. Haggerty et al.

The NOAA MODIS/near-real-time System

K. Hawick, H. James, A. Silis, D. Grove, C. Pattern, J. Mathew, P. Coddington, K. Kerry, J. Hercus, F. Vaughan,...

High availability user's guide,...

Cited by (33)

Evaluating hedge fund downside risk using a multi-objective neural network
2019, Journal of Visual Communication and Image Representation
Citation Excerpt :
Among them, there are two important risk measurement methods: dispersion risk measurement and risk exposure measurement. As the understanding of downside risk has gradually deepened, variance, semi-variance, and lower partial moment methods have been derived to measure dispersion risk [27–31]. The semi-variance method improves on the bilateral variance method by assuming that only downward fluctuations form the risk.
Predicting the downside risk of a hedge fund is the foundation of risk measurement. These predictions also provide conditions that can be used for designing and implementing risk prevention measures. Hence, this paper proposes a big data hedge fund downside risk evaluation model based on a multi-objective neural network. First, two evaluation indexes are defined. Then, local search is applied to merge parent and descendant populations. Only those individuals from the Pareto front are optimized. Experimental results suggest that this model and method is feasible and valid. Specifically, the VaR model is unable to estimate the possible extreme risk of a hedge fund. In contrast, the CVaR model can accurately measure the risks under extreme market conditions. However, a combination of VaR and CVaR can help a fund manager avoid extreme risks to a hedge fund.
EDAWS: A distributed framework with efficient data analytics workspace towards discriminative services for critical infrastructures
2018, Future Generation Computer Systems
Citation Excerpt :
Big data research gains the popularity and it is successfully applied to a broad variety of sustainability fields (e.g., home security, building management, smart cities, emergency control). With the help of analyzing large data in distributed computing environment, people thus benefit a lot from big data researches [24]. Today’s data mining software is confronted with enormous challenges about how to make use of newly developed paradigms of data processing.
Critical infrastructure systems which are interrelated with people’s daily life perform functions in multiple domains. However, with the explosion of specialized textual information in such systems, providing discriminative services for users through potential knowledge discovery becomes an essential and technical concern. Once massive data analytics is conducted in standalone server, the performance will degenerate tremendously. Alternatively, people cannot conveniently get such discriminative (self-caring) services. To address these concerns, we propose the general solution of EDAWS: a Novel Distributed Framework with Efficient Data Analytics Workspace towards Discriminative Service for Critical Infrastructures, through leveraging the state-of-the-art software technologies and computing paradigms. We argue it from the following aspects: Firstly, the server-side platform facilitates native data capture, storage, index and data mining with a systematic organization. Secondly, a text-mining approach with index building in parallel is conducted for various functional business, by exploiting the potential of Lucene-based distributed cluster. Thirdly, with the widespread usage of tiny but powerful mobile devices, the server-side platform could be accessed by mobile-side clients remotely in a more convenient way. To demonstrate our solution, a case study of smart residence prototype towards discriminative services in terms of information retrieval, personalized information push, and hot topic discovery is thoroughly discussed. The extensively experimental studies are conducted for the prototype over various real-world datasets. Experimental results indicate that, data processing which runs on computing nodes has good scalability with data sizes and computing nodes, and the prototype passes from data to discriminative services successfully.
SunwayMR: A distributed parallel computing framework with convenient data-intensive applications programming
2017, Future Generation Computer Systems
Citation Excerpt :
Although MapReduce (e.g., Hadoop) gains popularity due to its simplicity, the main drawback of MapReduce is its inefficiency for processing structured (relational) data and graph data. In particular, Shahram, et al. propose a robust framework for real-time distributed processing of satellite data [10], with using the multiple-server redundancy backup manners to guarantee the framework’s reality. Nonetheless, it has few types of data to be processed.
Managing servers integration to realize distributed data computing framework is an important concern. Regardless of the underlying architecture and the actual distributed system’s complexity, such framework gives programmers an abstract view of systems to achieve variously data-intensive applications. However, some state-of-the-art frameworks need too much library dependencies and parameters configuration, or lack extensibility in application programming. Moreover, general framework’s precise design is a nontrivial work, which is fraught with challenges of task scheduling, message communication and computing efficiency, etc. To address these problems, we present a general, scalable and programmable parallel computing framework called SunwayMR, which only needs GCC/G++ environment. We argue it from the following aspects: (1) Distributed data partitioning, message communication and task organization are given to support transparent application execution on parallel hardware. By searching threads table of each node, the task gets an idle thread (with preferred node IP address) for executing data partition. A novel communication component, SunwayMRHelper, is employed to merge periodical results synchronously. Through identifying whether current node is master or slave, SunwayMR deals with the periodical task’s results differently. (2) As for optimizations, a simple fault tolerance is given to resume data-parallel applications, and thread-level stringstream is utilized to boost computing. To ensure ease-of-use, open Application Programming Interface (API) excerpts can be invoked by various of applications with fewer handwritten code than OpenMPI/MPI. We conduct extensively experimental studies to evaluate the performance of SunwayMR over real-world datasets. Results indicate that SunwayMR (runs on 16 computational nodes) outperforms Spark in various applications, and has good scaling with data sizes, nodes and threads.
Parallel techniques for information extraction from hyperspectral imagery using heterogeneous networks of workstations
2008, Journal of Parallel and Distributed Computing
Recent advances in space and computer technologies are revolutionizing the way remotely sensed data is collected, managed and interpreted. In particular, NASA is continuously gathering very high-dimensional imagery data from the surface of the Earth with hyperspectral sensors such as the Jet Propulsion Laboratory's airborne visible-infrared imaging spectrometer (AVIRIS) or the Hyperion imager aboard Earth Observing-1 (EO-1) satellite platform. The development of efficient techniques for extracting scientific understanding from the massive amount of collected data is critical for space-based Earth science and planetary exploration. In particular, many hyperspectral imaging applications demand real time or near real-time performance. Examples include homeland security/defense, environmental modeling and assessment, wild-land fire tracking, biological threat detection, and monitoring of oil spills and other types of chemical contamination. Only a few parallel processing strategies for hyperspectral imagery are currently available, and most of them assume homogeneity in the underlying computing platform. In turn, heterogeneous networks of workstations (NOWs) have rapidly become a very promising computing solution which is expected to play a major role in the design of high-performance systems for many on-going and planned remote sensing missions. In order to address the need for cost-effective parallel solutions in this fast growing and emerging research area, this paper develops several highly innovative parallel algorithms for unsupervised information extraction and mining from hyperspectral image data sets, which have been specifically designed to be run in heterogeneous NOWs. The considered approaches fall into three highly representative categories: clustering, classification and spectral mixture analysis. Analytical and experimental results are presented in the context of realistic applications (based on hyperspectral data sets from the AVIRIS data repository) using several homogeneous and heterogeneous parallel computing facilities available at NASA's Goddard Space Flight Center and the University of Maryland.
A High-Performance Infrastructure for Remote Sensing Data Applications Using HPC Paradigms
2022, International Journal of Electrical and Electronics Research
Double layer programming model to the scheduling of remote sensing data processing tasks
2019, Discrete and Continuous Dynamical Systems - Series S

View all citing articles on Scopus

Yongsheng Zhao holds a Ph.D. degree in Aerospace Engineering from University of Maryland, and a M.S. and B.S. in Aerospace Engineering from Northwestern Polytechnical University, China. His current work is to develop real time, high performance, and fault tolerant software on distributed platforms such as Linux Clusters, with application to NOAA's satellite ground data processing systems.

Tony Harvey received a BSc in Mathematics from London University in 1968. He has been involved in communications systems development from the early 1970s. Starting with terrestrial data and voice networks in Europe and then satellite ground systems in the USA, he has developed software for realtime communications systems for more than 30 years. In many of these systems he developed highly reliable software architectures in a variety of multi-processor environments. At AC Technologies, Tony was involved with the ground systems for NOAA's geostationary and polar orbiting satellites, implementing a one-way secure bridge for data flow out of the secure operations environment to provide public access to NOAA's weather data.

Anand Swaroop holds a Ph.D. in Physics from the Indian Institute of Technology, Delhi, India, and a M.S. in Physics from Lucknow University, India. He is Program Manager with NORTEL Government Solutions, Inc., and supports National Oceanic and Atmospheric Administration (NOAA) satellite ground systems software development utilizing new hardware and software technologies. His research interests are real time high performance scientific computing on parallel and distributed platforms and developing high performance data processing and archival systems using Linux cluster technology for next generation satellite instrument such as GIFTS on board geosynchronous satellite.

Keith McKenzie joined the NOAA National Environmental Satellite Data and Information Service (NESDIS) in 1980. He has worked as an engineering Project Manager managing GOES satellite ground system development projects since 1987. He is currently the manager of the Replacement Product Monitor (RPM), Modernized Sensor Processing System (MSPS), and GIFTS (level 0) calibration algorithm projects. He holds a M.S. in Telecommunications and Computers from George Washington University in Washington, D.C.

View full text

A robust framework for real-time distributed processing of satellite data

Abstract

Section snippets

Parallel Computing

MPI/FT: architecture and taxonomies for fault-tolerant, message-passing middleware for performance-portable parallel computing

MPICH-V2: a fault tolerant MPI for volatile nodes based on the pessimistic sender based message logging

Design studies for real-time distributed processing of infrared hyperspectral image data

Component-oriented design studies for efficient processing of hyperspectral infrared imager data

Essential JNI: Java Native Interface

The NOAA MODIS/near-real-time System