Synchronization in federation community networks

https://doi.org/10.1016/j.jpdc.2009.10.006Get rights and content

Abstract

A large scale High Level Architecture (HLA)-based simulation can be constructed using a network of simulation federations to form a “federation community”. This effort is often for the sake of enhancing scalability, interoperability, composability and enabling information security. Synchronization mechanisms are essential to coordinate the execution of federates and event transmissions across the boundaries of interlinked federations. We have developed a generic synchronization mechanism for federation community networks with its correctness mathematically proved. The synchronization mechanism suits various types of federation community network and supports the reusability of legacy federates. It is platform-neutral and independent of federate modeling approaches. The synchronization mechanism has been evaluated in the context of the Grid-enabled federation community approach, which allows simulation users to benefit from both Grid computing technologies and the federation community approach. A series of experiments has been carried out to validate and benchmark the synchronization mechanism. The experimental results indicate that the proposed mechanism provides correct time management services to federation communities. The results also show that the mechanism exhibits encouraging performance in terms of synchronization efficiency and scalability.

Introduction

Distributed simulation technology facilitates the construction of a large scale simulation with simulation components of various types, which can be developed independently and distributed geographically. The High Level Architecture (HLA) defines the rules, interface specification and object model template to support reusability and interoperability amongst the simulation components, known as federates [12]. While the HLA serves as the de facto standard for distributed simulations, the Runtime Infrastructure (RTI) software provides services1to support and synchronize the interactions amongst different federates conforming to the HLA standard to sustain an overall simulation application, known as a federation as shown in Fig. 1(A).

In the case where the problem domain is particularly complex or involves multiple collaborative parties, the analysts often need to construct large scale HLA-based simulations which may involve a large number of federates and vast computing resources over a network or the Internet. Some typical examples are: military commission rehearsal, Internet gaming, biology simulation, and supply chain simulation. Sometimes such large scale simulations need to be constructed upon multiple simulation federations. Despite the tremendous advantages brought by the HLA technologies, the HLA standard does not explicitly sustain interoperation between federates across the boundaries of federations. To address this issue, a method has been proposed to harness a network of federations to achieve a common goal in the form of a “federation community” [24]. Fig. 1(B) illustrates an example of federation community network. In addition to the advantages of using flat federations,2 simulation developers and users can benefit from the federation community method in (1) improving the scalability of large and complex applications by reducing the bandwidth requirements through the partitioning of network load and the filtering out of irrelevant data amongst federations, (2) enhancing composability of simulation by enabling simulation development using legacy federations, (3) supporting interoperation between heterogeneous federations and RTIs, and (4) facilitating information security [2], [3], [4], [6], [23], [24]. The applications of a federation community have been extensively discussed in the existing literature, including the architecture for information hiding adopted in secured supply chain simulations developed by our team [4].

The emergence of Grid computing technologies meets the requirement of large amounts of distributed computational and data resources by the increasing size and complexity of simulation applications. The Grid provides a flexible, secure and coordinated resource sharing environment which can facilitate distributed simulation execution [27]. In the last few years, there has been an increasing interest in taking advantage of the Grid technologies to execute HLA simulations over the Internet. A few research groups have successfully enabled the Grid technologies to RTI, which either use middleware approaches to encapsulate vendor-provided RTIs [8], [27], [30], [31], [32] or implement the RTI directly using Grid services [14], [26]. While the existing Grid-enabled simulation techniques have proved highly advantageous, users can even benefit more if a whole federation is accessible through Grid services. Once a Grid-enabled federation community is constructed, the computational resources within administrative domains can be further exploited with the reusability3 of legacy federates maximized when enabling the Grid technology.

Simulation federates in a federation community should be provided with RTI services as in a flat federation [18] so that they can interact with each other and the RTIs involved, irrespective of whether these federates operate in the same federation or not. Support to the RTI services related to Federation Management, Declaration Management, Object Management (OM) and Data Distribution Management (DDM) has been well addressed to facilitate the execution of federation communities [2], [3], [21], [22]. Nevertheless, a more important and challenging issue, namely Time Management (TM) over federation community networks, has received little attention. Time management is concerned with the mechanisms for controlling the advancement of each federate along the federation time axis and synchronizing event (the terms “event” and “message” are used interchangeably in the rest of the paper) delivery amongst federates. Without a properly designed synchronization mechanism, the overall simulation execution upon a federation community is susceptible to state inconsistencies, e.g., the federates in different federations receiving messages for the same set of events in different orders. However, the current IEEE 1516 HLA standard is not intended to foster time management across the boundaries of federations [18]. There only exist a few preliminary or non-standard methods to address this issue. Hence, there exists a pressing need for a generic synchronization mechanism for HLA-based federation communities.

This paper proposes a generic approach to synchronizing federates and events within federation community networks. The proposed synchronization algorithms have been developed based on the gateway federate approach for constructing federation communities. The algorithms operate inside the gateway federates, and the time constraints from any federation are propagated through the whole network to coordinate the progresses of the whole federation community along the simulation time axis. The approach has also been successfully applied onto Grid-enabled federation communities.

A shorter version of this paper was presented in the 1st IEEE International Workshop on Advances of CyberInfrastructure [5] in the context of a Grid-enabled infrastructure for simulation of large and complex scenarios. The rest of this paper is organized as follows: Section 2 gives an overview of federation communities and related issues. Section 3 discusses existing work and analyses the challenges to be addressed. Section 4 details the algorithms for synchronizing Timestamp Order events crossing federation boundaries and proves the algorithms’ correctness. Section 5 presents the benchmarking experiments, which examine the correctness of the synchronization mechanism and evaluate its performance. In Section 6, we conclude with a summary and proposals on future work.

Section snippets

HLA federation communities

The technology for constructing federation communities has been well studied. Federation communities may be constructed for different objectives (see Section 1) and various architectures. Grid computing technologies can also be combined with federation community approaches to further benefit simulation users.

Time management in federation communities

In the context of HLA-based simulations, a federate can be both time regulating and time constrained, either of them, or neither. Time regulating federates may send time stamp ordered (TSO) events while time constrained federates are able to receive TSO events in time order. Each regulating federate must specify a “lookahead” value, and ensure that it will not generate any TSO event earlier than its current time plus lookahead [18]. After requesting a time advance from the RTI and getting

Synchronization algorithms for federation community networks

This section proposes a generic synchronization mechanism that takes advantage of the Gateway approach to constructing federation communities. Two novel synchronization algorithms are introduced which are employed by gateway federates, one for layered federations and another for star link federations (see Fig. 3). The basic idea of the proposed mechanism is to let a gateway throttle the time advance of the federations linked together by the gateway. The two proposed algorithms acquire the time

Experiments and results

In order to verify the correctness and investigate the overhead incurred in the proposed synchronization mechanism and its scalability, we carried out a number of experiments to benchmark the efficiency of synchronization in federation communities (multiple-layer federation community, peer-to-peer federation community and Grid-enabled federation community) in both LAN environments and a WAN environment between the UK (Birmingham) and Singapore.

The DMSO RTI NG 1.3 with Java bindings and its

Conclusions and future work

This paper is concerned with issues related to synchronization in federation community networks with alternative approaches for constructing federation communities and various architectures of federation community networks are introduced. The paper presents a mechanism for supporting synchronization crossing the boundaries of interlinked federations. We have investigated the issues and design of two generic synchronization algorithms. Based upon the gateway federate approach, the

Acknowledgment

This research was supported by the National Natural Science Foundation of China under the project grants No. 60804036 and No. 60805045. It is a collaborative work among the University of Birmingham, Nanyang Technological University, and Yanshan University under the Birmingham-Warwick Science City Research Interdisciplinary Alliance.

Dan Chen is a joint HEFCE Fellow with the University of Birmingham and the University of Warwick (United Kingdom). He is also a Professor with the Institute of Electrical Engineering, Yanshan University (China). He was a Postdoctoral Research Fellow with School of Computer Engineering at Nanayang Technological University (NTU, Singapore). He received a B.Sc. in applied physics from Wuhan University (China) and a M.Eng. in computer science from Huazhong University of Science and Technology

References (32)

  • D. Chen et al.

    A decoupled federate architecture for high level architecture-based distributed simulation

    Journal of Parallel and Distributed Computing

    (2008)
  • D. Chen et al.

    Large scale agent-based simulation on the grid

    Future Generation Computer Systems

    (2008)
  • C. Bouwens, D. Hurrell, D. Shen, Implementing ownership management services with a bridge federate, in: Proceedings of...
  • W. Cai, G. Li, S.J. Turner, B.S. Lee, L. Liu, Automatic construction of hierarchical federation architectures, in:...
  • W. Cai, G. Li, S.J. Turner, B.S. Lee, L. Liu, Implementation of federation management services over federation...
  • W. Cai, S.J. Turner, B.P. Gan, Hierarchical federations: An architecture for information hiding, in: Proceedings of the...
  • D. Chen, C. Bian, Towards hybrid grid infrastructure for large simulations, in: Proceedings of the 15th IEEE...
  • D. Chen, B.S. Lee, W. Cai, S.J. Turner, Design and development of a cluster gateway for cluster-based hla distributed...
  • D. Chen et al.

    Algorithms for HLA-based distributed simulation cloning

    ACM Transactions on Modeling and Computer Simulation

    (2005)
  • D. Chen et al.

    Towards fault-tolerant HLA-based distributed simulations

    Simulation: Transactions of the Society for Modeling and Simulation International

    (2008)
  • A. Cramp, J.P. Best, Time management in hierarchical federation communities, in: Proceedings of the 2002 Fall...
  • J.S. Dahmann et al.

    Standards for simulation: As simple as possible but not simpler, the high level architecture for simulation

    Simulation: Transactions of the Society for Modeling and Simulation International

    (1998)
  • DMSO. 2002, RTI 1.3—Next Generation Programmer’s Guide Version 5, DoD, DMSO,...
  • G. Fox, A. Ho, S. Pallickara, M. Pierce, W. Wu, Grids for the GiG and real time simulation, in: Proceedings of the 9th...
  • R.M. Fujimoto et al.

    Time management in the DoD high level architecture

    ACM SIGSIM Simulation Digest Archive

    (1996)
  • R.M. Fujimoto

    Time management in the high level architecture

    Simulation: Transactions of the Society for Modeling and Simulation International

    (1998)
  • Cited by (27)

    • Hybrid modelling and simulation of huge crowd over a hierarchical Grid architecture

      2013, Future Generation Computer Systems
      Citation Excerpt :

      The synchronization module is responsible for delivering these events in a correct order and coordinating time advances of all simulation models. More details about the synchronization algorithm are available at [27]. We have developed a large simulation system of a huge crowd’s evacuation and its interaction with vehicles in an urban area.

    • Towards enabling Cyberinfrastructure as a Service in Clouds

      2013, Computers and Electrical Engineering
      Citation Excerpt :

      Computing Clouds render users with services to access hardware, software and data resources, thereafter an integrated computing platform as a service. The Cloud computing paradigm distinguishes itself from other computing paradigms, such as Grid computing [9–11], Global computing, Internet Computing in the following aspects [12,13]: Utility computing model: Users obtain and employ computing platforms in computing Clouds as easily as they access a traditional public utility (such as electricity, water, natural gas, or telephone network).

    • Granular cloning: Intra-object parallelism in ensemble studies

      2018, SIGSIM-PADS 2018 - Proceedings of the 2018 ACM SIGSIM Conference on Principles of Advanced Discrete Simulation
    • Cloning agent-based simulation

      2017, ACM Transactions on Modeling and Computer Simulation
    • Large-Scale Simulation: Models, Algorithms, and Applications

      2017, Large-Scale Simulation: Models, Algorithms, and Applications
    View all citing articles on Scopus

    Dan Chen is a joint HEFCE Fellow with the University of Birmingham and the University of Warwick (United Kingdom). He is also a Professor with the Institute of Electrical Engineering, Yanshan University (China). He was a Postdoctoral Research Fellow with School of Computer Engineering at Nanayang Technological University (NTU, Singapore). He received a B.Sc. in applied physics from Wuhan University (China) and a M.Eng. in computer science from Huazhong University of Science and Technology (China). After that, he received another M.Eng. and a Ph.D. from NTU in year 2002 and 2006 respectively. His research interests include: computer-based modelling and simulation, high performance computing, and neuroinformatics. Recently, he has been working in dynamics analysis of large crowd and high performance computing in neuroinformatics.

    Stephen J. Turner is a Professor and the head of the Computer Science Division, School of Computer Engineering at Nanyang Technological University (NTU, Singapore). He joined NTU in 1999. Previously, he was a Senior Lecturer in Computer Science at Exeter University (UK). He received his MA in Mathematics and Computer Science from Cambridge University (UK) and his M.Sc. and Ph.D. in Computer Science from Manchester University (UK). His current research interests include: parallel and distributed simulation, distributed virtual environments, grid computing and multi-agent systems. He is steering committee chair of the Principles of Advanced and Distributed Simulation conference and advisory committee member of the Distributed Simulation and Real Time Applications symposium. He is also an area editor of ACM Transactions on Modeling and Computer Simulation (TOMACS).

    Wentong Cai is a Professor in the Division of Computer Science, School of Computer Engineering at Nanyang Technological University, Singapore. He is also the Director of the Parallel and Distributed Computing Centre. He received his Ph.D. in Computer Science from University of Exeter (UK) in 1991. His current research interests include: Parallel & Distributed Simulation, Multi-agent Systems, and Grid and Cluster Computing. He is an associate editor of the ACM Transactions on Modeling and Computer Simulation (TOMACS), and editorial board member of the Multiagents and Grid Systems—An International Journal.

    Georgios K. Theodoropoulos is a Reader in the School of Computer Science at the University of Birmingham (UK) where he has set up and leads the Distributed Systems Lab. He is also a founding Director of the Midlands e-Science Centre. He received his Diploma in Computer Engineering from the University of Patras (Greece) and his M.Sc. and Ph.D. in Computer Science from the University of Manchester (UK). His current research is in parallel and distributed simulation, distributed virtual environments, Grid computing and Peer-to-Peer systems. He is currently Area Editor of the Simulation: Transactions of the Society for Computer Simulation International (SCS) and editorial board member of the International Journal of Simulation and Process Modelling.

    Muzhou Xiong is a Ph.D. candidate in Huazhong University of Science and technology, P.R. China. Also he currently is a Research Associate in Nanyang Technological University, Singapore. He obtained his B.S. degree in computer science from Huazhong University of Science and Technology in 2002. His research interests include grid computing, network storage, automatic storage management, and parallel and distributed simulation. His recent research has focused on modeling of human behavior model for high dense crowds.

    Michael Lees is currently an Assistant Professor at Nanyang Technological University (Singapore). He completed his Ph.D. at Nottingham University (UK) in 2006 and went on to post-doctoral positions at Nottingham University and Birmingham University. He received his BSc from Edinburgh University (UK) in Artificial Intelligence and Computer Science. His research interests include: Agent-based modeling and simulation, large-scale distributed simulation and high-performance computing.

    View full text