Skip to main content
Springer Nature Link
Log in

The Ignite Distributed Collaborative Scientific Visualization System

  • Chapter
  • First Online:
The GENI Book

  • 675 Accesses

Abstract

We describe the Ignite Distributed Collaborative Visualization System (IDCVS), a system which permits real-time interaction and visual collaboration around large data sets on thin devices for users distributed about the wide area. The IDCVS provides seamless interaction and immediate updates even under heavy load and when users are widely separated: the design goal was to fetch a 1 MB data set from a server and render it within 150 ms, for a user anywhere in the world, and reflect changes made by a user in one location to all other users within the bound given by inter-user network latency. Scientific collaboration and interaction is the initial use case for the IDCVS, since eScience is characterized by large data sets. The visualizer can be used for any application where the data can be visualized on a web page. The visualizer consists of many replicated components, distributed across the wide area, so that an instance of the visualizer is close to any user: the design goal is to place an instance of the visualizer with an 20-ms latency of any user. It is the first exemplar of a new class of application enabled by the Distributed Cloud: real-time interaction with large data sets on arbitrarily thin devices, anywhere. The IDCVS features modular design, so it functions as a specialized Platform-as-a-Service: writing a new collaborative visualization application is as simple as designing a web page and distributing a data server. The system was demonstrated successfully on a significant worldwide air pollution data set, with values on 10, 25, 50, and 100 km worldwide grids, monthly over an 18-year period. It was demonstrated on a wide variety of clients, including laptop, tablet, and smartphone. The system itself has been deployed at over 20 sites worldwide. Distribution and deployment across the GENI Experiment Engine was accomplished in 15 min, and installing a new site is limited by ftp time for the data set.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    Source of Computation Costs: http://aiimpacts.org/wikipedia-history-of-gflops-costs/. Source of Communications Costs: http://drpeering.net/white-papers/Internet-Transit-Pricing-Historical-And-Projected.php.

References

  1. Ansible api documentation: http://docs.ansible.com/ (2016)

  2. Badam, S.K., Elmqvist, N.: Polychrome: a cross-device framework for collaborative web visualization. In: ACM ITS (2014)

    Book  Google Scholar 

  3. Baldine, I., Xin, Y., Mandal, A., Ruth, P., Heerman, C., Chase, J.: Exogeni: a multi-domain infrastructure-as-a-service testbed. In: Testbeds and Research Infrastructure, pp. 97–113. Springer, Berlin, Heidelberg (2012)

    Google Scholar 

  4. Bastin, N., Bavier, A., Blaine, J., Chen, J., Krishnan, N., Mambretti, J., McGeer, R., Ricci, R., Watts, N.: The Instageni initiative: an architecture for distributed systems and advanced programmable networks. Comput. Netw. 61 (0), 24–38 (2014). Special issue on Future Internet Testbeds—Part I

    Google Scholar 

  5. Bavier, A., McGeer, R.: The geni experiment engine. In: The GENI Book Prototype of the Next Internet. Springer, Berlin (2016)

    Book  Google Scholar 

  6. Bavier, A., Chen, J., Mambretti, J., McGeer, R., McGeer, S., Nelson, J., O’Connell, P., Ricart, G., Tredger, S., Coady, Y.: The geni experiment engine. In: Teletraffic Congress (ITC), 2014 26th International, September, pp. 1–6 (2014)

    Google Scholar 

  7. Bavier, A., Chen, J., Mambretti, J., McGeer, R., McGeer, S., Nelson, J., O’Connell, P., Tredger, S., Coady, Y.: The geni experiment engine. In: Tridentcom (2015)

    Book  Google Scholar 

  8. Berman, M., Chase, J.S., Landweber, L., Nakao, A., Ott, M., Raychaudhuri, D., Ricci, R., Seskar, I.: Geni: a federated testbed for innovative network experiments. Comput. Netw. 61 (0), 5–23 (2014). Special issue on Future Internet Testbeds—Part I

    Google Scholar 

  9. Bhojwani, S., Hemmings, M., Ingalls, D., Krahn, R., Lary, D., Lincke, J., McGeer, R., Ricart, G., Roder, M., Coady, Y., Stege, U.: The ignite distributed collaborative scientific visualization system. In: Distributed Cloud Computing Workshop (2015)

    Book  Google Scholar 

  10. Bhojwani, S., Hemmings, M., Ingalls, D., Krahn, R., Lary, D., Lincke, J., McGeer, R., Ricart, G., Roder, M., Coady, Y., Stege, U.: The ignite distributed collaborative scientific visualization system. In: IEEE CloudCom (2015)

    Book  Google Scholar 

  11. Choy, S., Wong, B., Simon, G., Rosenberg, C.: The brewing storm in cloud gaming: a measurement study on cloud to end-user latency. In: Proceedings of the 11th Annual Workshop on Network and Systems Support for Games, p. 2. IEEE Press, Piscataway (2012)

    Google Scholar 

  12. Cisco: Cisco global cloud index: forecast and methodology, Cisco, Inc., San Jose, CA, 2014–2019 (2015)

    Google Scholar 

  13. Cisco: Cisco visual networking index: forecast and methodology, Cisco, Inc., San Jose, CA, 2014–2019 (May 2015)

    Google Scholar 

  14. Cisco: The zettabyte era: trends and analysis, Cisco, Inc., San Jose, CA, (May 2015)

    Google Scholar 

  15. D3.js: Data-driven documents. http://d3js.org/ (2015)

  16. DeFanti, T.A., et al.: The optiportal, a scalable visualization, storage, and computing interface device for the optiputer. Futur. Gener. Comput. Syst. 25 (2), 114–123 (2009)

    Article  Google Scholar 

  17. DeFanti, T.A., Acevedo, D., Ainsworth, R.A., Brown, M.D., Cutchin, S., Dawe, G., Doerr, K.-U., Johnson, A., Knox, C., Kooima, R., et al.: The future of the cave. Cent. Eur. J. Eng. 1 (1), 16–37 (2011)

    Google Scholar 

  18. Docker: https://www.docker.com/whatisdocker/. (2016)

  19. Einstein, A.: On the electrodynamics of moving bodies. Ann. Phys. pp 891–921, (1905)

    Google Scholar 

  20. Ellis, C.A., Gibbs, S.J.: Concurrency control in groupware systems. In: Proceedings of the 1989 ACM SIGMOD International Conference on Management of Data, SIGMOD ’89, pp. 399–407. ACM, New York, NY (1989)

    Google Scholar 

  21. Fabric api documentation: http://docs.fabfile.org/en/1.8/ (2016)

  22. Dan Rayburn, Frost and Sullivan. Comparing cdn performance: Amazon cloudfront’s last mile testing results. https://media.amazonwebservices.com/FS_WP_AWS_CDN_CloudFront.pdf (2016)

  23. Goradia, I., Doshi, J., Kurup, L.: A review paper on http://store.steampowered.com/universe/vr rift & project morpheus. Int. J. Curr. Eng. Technol. 4 (5) pp. 3196–3200, (2014)

  24. Grochow, K., Howe, B., Stoermer, M., Barga, R., Lazowska, E.: Client + cloud: evaluating seamless architectures for visual data analytics in the ocean sciences. In: International Conference on Scientific and Statistical Database Management, SSDBM’10, pp. 114–131. Springer, Berlin, Heidelberg (2010)

    Google Scholar 

  25. Grochow, K., Stoermer, M., Fogarty, J., Lee, C., Howe, B., Lazowska, E.: Cove: a visual environment for multidisciplinary ocean science collaboration. In: IEEE Sixth International Conference on e-Science, ESCIENCE ’10, pp. 269–276. IEEE Computer Society, Washington, DC (2010)

    Google Scholar 

  26. Ingalls, D.: The smalltalk-76 programming system design implementation. In: ACM Conference on Principles of Programming Languages (1978)

    Book  Google Scholar 

  27. Ingalls, D., Kaehler, T., Maloney, J., Wallace, S., Kay, A.: Back to the future: the story of squeak, a practical smalltalk written in itself. In: OOPSLA, pp. 318–326. ACM Press, New York (1997)

    Google Scholar 

  28. IPCC: Climate Change 2013: The Physical Science Basis. Contribution of Working Group I to the Fifth Assessment Report of the Intergovernmental Panel on Climate Change. Cambridge University Press, Cambridge/New York, NY (2013)

    Google Scholar 

  29. Jagodic, R., Renambot, L., Johnson, A., Leigh, J., Deshpande, S.: Enabling multi-user interaction in large high-resolution distributed environments. Futur. Gener. Comput. Syst. 27 (7), 914–923 (2011)

    Article  Google Scholar 

  30. Johnson, G.P., Abram, G.D., Westing, B., Navr’til, P., Gaither, K.: Displaycluster: an interactive visualization environment for tiled displays. In: 2012 IEEE International Conference on Cluster Computing (CLUSTER), pp. 239–247. IEEE, New York (2012)

    Google Scholar 

  31. jQuery: jquery: write less, do more. https://jquery.com/ (2015)

  32. Kang, J., Lin, T., Bannazadeh, H., Leon-Garcia, A.: Software-defined infrastructure and the SAVI testbed. In: Testbeds and Research Infrastructure: Development of Networks and Communities - 9th International ICST Conference, TridentCom 2014, Guangzhou, 5–7 May 2014. Revised Selected Papers, pp. 3–13 (2014)

    Google Scholar 

  33. Krahn, R., Ingalls, D., Hirschfeld, R., Lincke, J., Palacz, K.: Lively Wiki a development environment for creating and sharing active web content. In: WikiSym ’09: Proceedings of the 5th International Symposium on Wikis and Open Collaboration, pp. 1–10. ACM, New York, NY (2009)

    Google Scholar 

  34. Lamport, L.: Time, clocks, and the ordering of events in a distributed system. Commun. ACM 21 (7), 558–565 (1978)

    Article  MATH  Google Scholar 

  35. Lamport, L: The part-time parliament. ACM Trans. Comput. Syst. 16 (2), 133–169 (1998)

    Article  Google Scholar 

  36. Lary, D.J., Faruque, F.S., Malakar, N., Moore, A., Roscoe, B., Adams, Z.L., Eggelston, Y.: Estimating the global abundance of ground level presence of particulate matter (pm2. 5). Geospat. Health 8 (3), 611–630 (2014)

    Google Scholar 

  37. Lary, D.J., Lary, T., Sattler, B.: Using machine learning to estimate global pm2. 5 for environmental health studies. Environ. Health Insights 2015 (1), 41–52 (2015). doi:10.4137/EHI.S15664

    Google Scholar 

  38. Leigh, J., Johnson, A., DeFanti, T.A.: Cavern: a distributed architecture for supporting scalable persistence and interoperability in collaborative virtual environments. Virtual Reality: Res. Dev. Appl. 2 (2), 217–237 (1997)

    Google Scholar 

  39. Leon-Garcia, A., Bannazadeh, H.: Savi testbed for applications on software-defined infrastructure. In: The GENI Book Prototype of the Next Internet. Springer, Berlin (2016)

    Book  Google Scholar 

  40. Leuf, B., Cunningham, W.: The Wiki Way: Collaboration and Sharing on the Internet. Addison-Wesley Professional, Reading (2001)

    Google Scholar 

  41. Lively: http://www.lively-web.org/ (2016)

  42. Maloney, J.H., Smith, R.B.: Directness and liveness in the morphic user interface construction environment. In: Proceedings of User Interface and Software Technology (UIST 95), pp. 21–28. ACM Press, New York (1995)

    Google Scholar 

  43. Maymounkov, P., Mazières, D.: Kademlia: a peer-to-peer information system based on the xor metric. In: Revised Papers from the First International Workshop on Peer-to-Peer Systems, IPTPS ’01, pp. 53–65. Springer, London (2001)

    Google Scholar 

  44. McGeer, R., Ricci, R.: The instageni project. In: The GENI Book Prototype of the Next Internet. Springer, Berlin (2016)

    Book  Google Scholar 

  45. Nakao, A.: Deeply programmable network through advanced network virtualization. In: IEICE International Symposium on Network Virtualization (2012)

    Google Scholar 

  46. Oculus: rift: Next generation virtual reality. https://www.oculus.com/en-us/rift/ (2015)

  47. Ongaro, D., Ousterhout, J.: In search of an understandable consensus algorithm. In: Proceedings of the 2014 USENIX Conference on USENIX Annual Technical Conference, USENIX ATC’14, pp. 305–320. USENIX Association, Berkeley, CA (2014)

    Google Scholar 

  48. Passarella, A.: A survey on content-centric technologies for the current internet: Cdn and p2p solutions. Comput. Commun. 35 (1), 1–32 (2012)

    Article  Google Scholar 

  49. Petersen, K., Spreitzer, M.J., Terry, D.B., Theimer, M.M., Demers, A.J.: Flexible update propagation for weakly consistent replication. In: Proceedings of the Sixteenth ACM Symposium on Operating Systems Principles, SOSP ’97, pp. 288–301. ACM, New York, NY (1997)

    Google Scholar 

  50. Polymaps: http://polymaps.org/ (2016)

  51. Ponto, K., Doerr, K., Kuester, F.: Giga-stack: a method for visualizing giga-pixel layered imagery on massively tiled displays. Futur. Gener. Comput. Syst. 26 (5), 693–700 (2010)

    Article  Google Scholar 

  52. Reed, D.P.: Designing croquet’s teatime: a real-time, temporal environment for active object cooperation. In: Companion to the 20th Annual ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications, OOPSLA ’05, p. 7. ACM, New York, NY (2005)

    Google Scholar 

  53. Resnick, M., Maloney, J., Monroy-Hernández, A., Rusk, N., Eastmond, E., Brennan, K., Millner, A., Rosenbaum, E., Silver, J., Silverman, B., Kafai, Y.: Scratch: programming for all. Commun. ACM 52 (11), 60–67 (2009)

    Article  Google Scholar 

  54. Smarr, L.L., Chien, A.A., DeFanti, T., Leigh, J., Papadopoulos, P.M.: The optiputer. Commun. ACM 46 (11), 58–67 (2003)

    Article  Google Scholar 

  55. Smith, D., Kay, A., Raab, A., Reed, D.P., et al.: Croquet-a collaboration system architecture. In: Proceedings. First Conference on Creating, Connecting and Collaborating Through Computing, 2003. C5 2003, pp. 2–9. IEEE, New York (2003)

    Google Scholar 

  56. Sun, C.: Undo as concurrent inverse in group editors. ACM Trans. Comput. Hum. Interact. 9 (4), 309–361 (2002)

    Article  Google Scholar 

  57. Taivalsaari, A., Mikkonen, T., Ingalls, D., Palacz, K.: Web browser as an application platform: the lively kernel experience, Technical Report. Sun Microsystems, Inc., Mountain View, CA, USA (2008)

    Book  Google Scholar 

  58. Tolia, N., Andersen, D.G., Satyanarayanan, M.: Quantifying interactive user experience on thin clients. IEEE Comput. 39 (3), 46–52 (2006)

    Article  Google Scholar 

  59. Ungar, D., Smith, R.B.: Self: the power of simplicity. In: Conference Proceedings on Object-Oriented Programming Systems, Languages and Applications, OOPSLA ’87, pp. 227–242. ACM, New York, NY (1987)

    Google Scholar 

  60. Valve: Steamvr. http://store.steampowered.com/universe/vr (2015)

  61. Vandenberghe, W., Vermeulen, B., Demeester, P., Willner, A., Papavassiliou, S., Gavras, A., Sioutis, M., Quereilhac, A., Al-Hazmi, Y., Lobillo, F., Schreiner, F., Velayos, C., Vico-Oton, A., Androulidakis, G., Papagianni, C.A., Ntofon, O., Boniface, M.: Architecture for the heterogeneous federation of future internet experimentation facilities. In: 2013 Future Network & Mobile Summit, Lisboa, 3–5 July 2013, pp. 1–11 (2013)

    Google Scholar 

  62. Vlachos, A.: Advanced vr rendering. In: Game Developers Conference (2015)

    Google Scholar 

  63. Vogels, W.: Eventually consistent. Commun. ACM 52 (1), 40–44 (2009)

    Article  Google Scholar 

  64. Zhao, B.Y., Kubiatowicz, J., Joseph, A.D.: Tapestry: an infrastructure for fault-tolerant wide-area location and routing. Technical Report, UC-Berkeley (2001)

    Google Scholar 

  65. 7 million premature deaths annually linked to air pollution. http://www.who.int/mediacentre/news/releases/2014/air-pollution/en/ (May 2014)

Download references

Acknowledgements

The authors thank our colleagues at US Ignite, particularly Joe Kochan and William Wallace. We were assisted in deployment by the GENI Project Office, with particular thanks to Niky Riga, Mark Berman, Marshall Brinn, and Sarah Edwards. Andy Bavier set us up on the GEE, as did Rob Ricci on InstaGENI and Ilya Baldin on ExoGENI. Andi Bergen, Hausi Muller and Hadi Bannazadeh of the SAVI Project in Canada, Aki Nakao of the VNode project in Japan, Brecht Vermuelen of Fed4Fire, Max Ott of NICTA and Robert Hirschfeld of HPI set up our international sites. Joe Mambretti iCAIR set up international networking connections. We thank our demo participants—Wim van der Meer of iMinds, Tobias Pape of HPI, and Pratama Putra of the University of Tokyo. We thank our collaborators at the Lively Web, Dan Ingalls and Jens Lincke who participated in an early version of this chapter [9, 10]. Shushil Bhojwani collaborated with us on an early version of the data server, and Ulrike Stege and Yvonne Coady supported us throughout. This work was partially supported by the GENI Project Office, by SAP and by MITACS.

Author information

Authors and Affiliations

  1. Computer Sciences Department, University of Victoria, Victoria, BC, Canada

    Matt Hemmings

  2. Department of Physics, University of Texas at Dallas, Dallas, TX, USA

    David Lary

  3. US Ignite, 1150, 18th St NW, Suite 900, Washington, DC, 20036, USA

    Rick McGeer (Chief Scientist) & Glenn Ricart (Chief Scientist)

  4. Y Combinator Research, San Francisco, CA, USA

    Robert Krahn & Marko Röder

Authors
  1. Matt Hemmings
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Robert Krahn
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. David Lary
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Rick McGeer
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Glenn Ricart
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Marko Röder
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Marko Röder .

Editor information

Editors and Affiliations

  1. US Ignite, Washington, DC, USA

    Rick McGeer

  2. GENI Project Office, Raytheon BBN Technologies, Cambridge, Massachusetts, USA

    Mark Berman

  3. GENI Project Office, Raytheon BBN Technologies, Cambridge, Massachusetts, USA

    Chip Elliott

  4. School of Computing, University of Utah, Salt Lake City, Utah, USA

    Robert Ricci

Rights and permissions

Reprints and permissions

Copyright information

© 2016 Springer International Publishing Switzerland

About this chapter

Cite this chapter

Hemmings, M., Krahn, R., Lary, D., McGeer, R., Ricart, G., Röder, M. (2016). The Ignite Distributed Collaborative Scientific Visualization System. In: McGeer, R., Berman, M., Elliott, C., Ricci, R. (eds) The GENI Book. Springer, Cham. https://doi.org/10.1007/978-3-319-33769-2_19

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-33769-2_19

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-33767-8

  • Online ISBN: 978-3-319-33769-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics