Elsevier

Parallel Computing

Volume 34, Issue 1, January 2008, Pages 32-46
Parallel Computing

Cluster versus grid for operational generation of ATCOR’s modtran-based look up tables

https://doi.org/10.1016/j.parco.2007.11.002Get rights and content

Abstract

A critical step in the product generation of satellite or airborne earth observation data is the correction of atmospheric features. Due to the complexity of the underlying physical model and the amount of coordinated effort required to provide, verify and maintain baseline atmospheric observations, one particular scientific modelling program, modtran, whose ancestor was first released in 1972, has become a de facto basis for such processing. While this provides the basis of per-pixel physical modelling, higher-level algorithms, which rely on the output of potentially thousands of runs of modtran are required for the processing of an entire scene. The widely-used atcor family of atmospheric correction software employs the commonly-used strategy of pre-computing a large look up table (lut) of values, representing modtran input parameter variation in multiple dimensions, to allow for reasonable running times in operation. The computation of this pre-computed look up table has previously taken weeks to produce a dvd (about 4 GB) of output. The motivation for quicker turnaround was introduced when researchers at multiple institutions began collaboration on extending atcor features into more specialized applications. In this setting, a parallel implementation is investigated with the primary goals of: the parallel execution of multiple instances of modtran as opaque third-party software, the consistency of numeric results in a heterogeneous compute environment, the potential to make use of otherwise idle computing resources available to researchers located at multiple institutions, and acceptable total turnaround time. In both grid and cluster environments, parallel generation of a numerically consistent lut is shown to be possible and reduce ten days of computation time on a single, high-end processor to under two days of processing time with as little as eight commodity CPUs. Runs on up to 64 processors are investigated and the advantages and disadvantages of clusters and grids are briefly explored in reference to the their evaluation in a medium-sized collaborative project.

Introduction

The ancestor of the modtran line of atmospheric modelling software was first released in 1972, based on band models developed in the 1950’s and 1960’s used to describe atmospheric transmission and absorption behaviour [1]. Over time, not only has the software has been continuously updated [2], [3], but also the underlying molecular spectroscopic database [4]. Although competing models also exist, modtran has become established as a de facto standard in fields related to atmospheric physics and remote sensing. While many applications employing modtran have usage patterns that require only tens of executions, others require hundreds or thousands. Examples of this usage include sensitivity studies [5], [6], and the radiometric and spectral calibration of imaging spectrometers using known characteristics of solar and atmospheric absorption features [7], [8], [9].

When modtran became more widely-used in operational, remotely sensed spectroscopy, invasive modifications of the typically end-user-compiled Fortran source code were developed by a group of users in order to introduce single-run parallelism, which resulted in efficient and scalable speedups [10]. For whatever reason, these code modifications were not officially incorporated into the standard modtran release, which has grown to over 80,000 lines of code. The code base has moved on, producing multiple releases since then, and these parallel modifications have effectively been lost to the wider scientific community.

Meanwhile, further standard applications, including atmospheric correction software such as employed by ISDAS [11] and the widely-disseminated atcor family of software [12], [13] were being built using modtran calculations as the basis of their own algorithms. Atmospheric correction of multispectral/hyperspectral imagery involve calculations that depend on a number of varying modtran-modifiable parameters defining atmospheric conditions (e.g., molecular absorber concentrations, aerosol scattering, optical depth) as well as observer and solar geometry, i.e., flight altitude, heading, ground elevation, view and solar zenith and azimuth angles. Therefore a large number of modtran calls are required to process a single satellite or airborne scene. In software such as isdas [14] and atcor [15], these computations are usually performed off-line, and stored in luts prior to actual atmospheric correction of a scene to enable reasonable operational hyperspectral data processing times. The use of luts support the processing of a large variety of unrelated scenes in a much shorter amount of time than would be needed for direct computation.

In the case of atcor, researchers from multiple institutions began to collaborate, in an informal way, on extending functionality into novel special-purpose areas, ranging from the haze removal of low-spectral, high-spatial IKONOS imagery to the processing of hyperspectral, wide field-of-view (fov) imagery as obtained from airborne (as opposed to space-bourne) instruments. This increased collaboration also increased the desire for higher turnaround on lut generation and, in turn, created the opportunity to pool the collaborators computing resources, in an informal way, to speed up lut generation.

In the lut generation usage of modtran, individual executions are independent of each other and input data parameters can be pre-computed ahead of time, leading to a workload that is “embarrassingly parallel” – i.e. there is no particular effort needed to segment the problem into a large number of parallel tasks. It is a common pattern in the employment of parallel computing for embarrassingly parallel problems to run multiple instances of a particular program over varying input parameters within a given problem space. This usage is explored here in both grid and cluster processing environments.

Section snippets

Method

The stated goals of this study include: the parallel execution of multiple instances of opaque third-party software, the consistency of numeric results in light of a heterogeneous compute environment, the ability to make use of otherwise idle computing resources available to researchers located at multiple institutions, and acceptable total turnaround time. These goals in addition to the parallel decomposition strategy are addressed individually.

Results

Numerical differences of the gnu-compiled modtran-generated results on the four different platforms are characterized in Fig. 5 by mean relative deviations [25] of a per-case “profile” across all 45,056 cases. The “profile” for a case, which is only used for comparison of that case with the other platforms, is defined as the mean of the subset of only those component radiance values that are eventually recorded in the resulting lut. All cases which produced bad i.e., negative, results (see

High-level granularity of modtran

Although it cannot be known what performance effect would have resulted from applying parallelization at a lower level of granularity i.e., within modtran itself, according to Table 1, a sufficiently fast turnaround time is clearly achievable using either of the aforementioned grid or cluster approaches on current hardware. With continuous hardware improvements, this situation is only expected to improve.

Numeric consistency evaluation

The most important information depicted in Fig. 5 is that other than the large number of

Conclusions

The application of grid and cluster parallelization in executing thousands of runs of third party software with varying input parameters has been investigated, specifically for the generation of atcor’s lut, widely-used in earth observation applications. The importance of validating the numerical consistency of the results for numerically intensive scientific software such as modtran, especially in light of heterogeneous computing environments, has been shown. In this case study, the potential

Acknowledgements

This work was supported in part by ESA/ESTEC contracts 16298/02/NL/US and 15449/01/NL/Sfe. The authors wish to acknowledge the support of the University of Zurich IT Services Department for access to the Matterhorn cluster [26]. J. Brazile acknowledges the support of Netcetera AG throughout the period of this work.

References (39)

  • A. Berk et al.

    modtran cloud and multiple scattering upgrades with applications to AVIRIS

    Remote Sensing of Environment

    (1998)
  • L. Rothman et al.

    The HITRAN molecular spectroscopic database: edition of 2000 including updates through 2001

    Journal of Quantitative Spectroscopy and Radiative Transfer

    (2003)
  • P. Wang et al.

    MODTRAN on supercomputers and parallel computers

    Parallel Computing

    (2002)
  • G.P. Anderson, J. Wang, M.L. Hoke, F.X. Kneizys, J.H. Chetwynd, L.S Rothman Jr., L.M. Kimball, R.A. McClatchey, E.P....
  • A. Berk, G.P. Anderson, P.K. Acharya, L.S. Bernstein, L. Muratov, J. Lee, M. Fox, S.M. Adler-Golden, J.H. Chetwynd,...
  • D. Schläpfer et al.

    Modeling the noise equivalent radiance requirements of imaging spectrometers based on scientific applications

    OSA Journal of Applied Optics

    (2002)
  • D. Schläpfer, J. Nieke, Operational simulation of at sensor radiance sensitivity using the MODO/MODTRAN environment,...
  • R.O. Green et al.

    On-orbit radiometric and spectral calibration characteristics of EO-1 Hyperion derived with an underflight of AVIRIS and in-situ measurements at Salar de Arizaro, Argentina

    IEEE Transactions on Geoscience and Remote Sensing

    (2003)
  • R.A. Neville, L. Sun, K. Staenz, Detection of spectral line curvature in imaging spectrometer data, in: S. Shen, P....
  • J. Brazile et al.

    Scene-based spectral response function shape discernibility for the APEX imaging spectrometer

    IEEE Geoscience and Remote Sensing Letters

    (2006)
  • K. Staenz et al.

    ISDAS – a system for processing/analyzing hyperspectral data

    Canadian Journal of Remote Sensing

    (1998)
  • R. Richter et al.

    Geo-atmospheric processing of airborne imaging spectrometry data. Part 2: atmospheric/topographic correction

    International Journal of Remote Sensing

    (2002)
  • D. Schläpfer et al.

    Geo-atmospheric processing of airborne imaging spectrometry data Part 1: parametric orthorectification

    International Journal of Remote Sensing

    (2002)
  • K. Staenz et al.

    Retrieval of surface reflectance from hyperspectral data using a look-up table approach

    Canadian Journal of Remote Sensing

    (1997)
  • R. Richter, Atmospheric/topographic correction for satellite imagery, Tech. Rep. DLR-IB 565-01/05, DLR, Wessling,...
  • D. Stern, Interactive Data Language. <http://www.ittvis.com>, 2007 (accessed December...
  • The MPI (Message Passing Interface) Forum. <http://www.mpi-forum.org/>, 2007 (accessed December...
  • D. Goldberg

    What every computer scientist should know about floating-point arithmetic

    ACM Computing Surveys

    (1991)
  • J. Barhen et al.

    Consistent uncertainty reduction in modeling nonlinear systems

    SIAM Journal on Scientific Computing

    (2004)
  • Cited by (19)

    • Advanced radiometry measurements and Earth science applicationswith the Airborne Prism Experiment (APEX)

      2015, Remote Sensing of Environment
      Citation Excerpt :

      The science grade APEX processor is designed to process large quantities of imaging spectrometer data, including calibration and housekeeping data (Schaepman, Schläpfer, Brazile, & Bojinski, 2002). Its design is based on an iterative prototyping approach and from beginning on includes considerations to build look-up tables for atmospheric compensation (Brazile et al., 2004), as well as optimized processing speed requirements (Brazile, Richter, Schläpfer, Schaepman, & Itten, 2008; Brazile et al., 2005). The science grade APEX processor is developed to integrate calibration data with actual measurements, allowing to reconstruct each geometrical position and each radiance of any given detector element (Fig. 4).

    • Crop planting and type proportion method for crop acreage estimation of complex agricultural landscapes

      2012, International Journal of Applied Earth Observation and Geoinformation
      Citation Excerpt :

      Atmospheric corrections, geometric corrections and the soil adjusted vegetation index (SAVI) are processed before classification, as described below and shown in Fig. 4. The MODTRAN algorithm removes the effects of atmospheric components, such as O3, CO2, water, and aerosols (Brazile et al., 2008). The meteorological observation data used in the atmospheric corrections, such as aerosol depth are obtained from the China Meteorological Administration.

    • A High-Performance Infrastructure for Remote Sensing Data Applications Using HPC Paradigms

      2022, International Journal of Electrical and Electronics Research
    • Gradient-based automatic lookup table generator for radiative transfer models

      2019, IEEE Transactions on Geoscience and Remote Sensing
    View all citing articles on Scopus
    View full text