skip to main content
10.1145/2989293.2989300acmconferencesArticle/Chapter ViewAbstractPublication PagesmswimConference Proceedingsconference-collections
research-article

Importance of Repeatable Setups for Reproducible Experimental Results in IoT

Published: 13 November 2016 Publication History

Abstract

Performance analysis of newly designed solutions is essential for efficient Internet of Things and Wireless Sensor Network (WSN) deployments. Simulation and experimental evaluation practices are vital steps for the development process of protocols and applications for wireless technologies. Nowadays, the new solutions can be tested at a very large scale over both simulators and testbeds. In this paper, we first discuss the importance of repeatable experimental setups for reproducible performance evaluation results. To this aim, we present FIT IoT-LAB, a very large-scale and experimental testbed, i.e., consists of 2769 low-power wireless devices and 127 mobile robots. We then demonstrate through a number of experiments conducted on FIT IoT-LAB testbed, how to conduct meaningful experiments under real-world conditions. Finally, we discuss to what extent results obtained from experiments could be considered as scientific, i.e., reproducible by the community.

References

[1]
P. Thubert. An Architecture for IPv6 over the TSCH mode of IEEE 802.15.4. draft-ietf-6tisch-architecture-10, June 2016.
[2]
IEEE Standard for Low-Rate Wireless Personal Area Networks (LR-WPANs). IEEE Std 802.15.4--2015 (Revision of IEEE Std 802.15.4--2011), April 2016.
[3]
I. Stojmenovic. Simulations in Wireless Sensor and Ad Hoc Networks: Matching and Advancing Models, Metrics, and Solutions. IEEE Communications Magazine, 46(12):102--107, 2008.
[4]
Anne-Sophie Tonneau, Nathalie Mitton, and Julien Vandaele. How to choose an experimentation platform for wireless sensor networks? a survey on static and mobile wireless sensor network experimentation facilities. Ad Hoc Networks, 30:115 -- 127, 2015.
[5]
A. Gluhak, S. Krco, M. Nati, D. Pfisterer, N. Mitton, and T. Razafindralambo. A Survey on Facilities for Experimental Internet of Things Research. IEEE Communications Magazine, 49(11):58--67, 2011.
[6]
Georgios Z. Papadopoulos, Julien Beaudaux, Antoine Gallais, Thomas Noel, and Guillaume Schreiner. Adding value to WSN simulation using the IoT-LAB experimental platform. In Proceedings of the 9th IEEE International Conference on Wireless and Mobile Computing, Networking and Communications (WiMob), pages 485--490, 2013.
[7]
Georgios Z. Papadopoulos, Kosmas Kritsis, Antoine Gallais, Periklis Chatzimisios, and Thomas Noel. Performance Evaluation Methods in Ad Hoc and Wireless Sensor Networks: A Literature Study. IEEE Communications Magazine, 54(1):122--128, 2016.
[8]
Stuart Kurkowski, Tracy Camp, and Michael Colagrosso. Manet simulation studies: The incredibles. Mobile Computing and Communications Review, 9(4):50--61, 2005.
[9]
C. Adjih, E. Baccelli, E. Fleury, G. Harter, N. Mitton, T. Noel, R. Pissard-Gibollet, F. Saint-Marcel, G. Schreiner, J. Vandaele, and T. Watteyne. FIT IoT-LAB: A large scale open experimental IoT testbed. In Proceedings of the 2nd IEEE World Forum on Internet of Things (WF-IoT), pages 459--464, 2015.
[10]
Clement Burin des Roziers, Guillaume Chelius, Tony Ducrocq, Eric Fleury, Antoine Fraboulet, Antoine Gallais, Nathalie Mitton, Thomas Noel, and Julien Vandaele. Using SensLAB as a First Class Scientific Tool for Large Scale Wireless Sensor Network Experiments. In Proceedings of the 10th IFIP International Conference on Networking, pages 147--159, 2011.
[11]
Georgios Z. Papadopoulos. Improving Medium Access in Dynamic Wireless Sensor Networks. PhD thesis, University of Strasbourg, 2015.
[12]
Georgios Z. Papadopoulos, Julien Beaudaux, Antoine Gallais, and Thomas Noel. T-AAD: Lightweight Traffic Auto-ADaptations for Low-power MAC Protocols. In Proceedings of the 13th IEEE IFIP Annual Mediterranean Ad Hoc Networking Workshop (MED-HOC-NET), pages 79--86, 2014.
[13]
Stefan Lohs, Reinhardt Karnapke, and Jorg Nolte. Link Stability in a Wireless Sensor Network - an Experimental Study. 4th International ICST Conference on Sensor Systems and Software (S-CUBE), pages 146--161, 2013.
[14]
Reza S. Dilmaghani, Hossein Bobarshad, M. Ghavami, Sabrieh Choobkar, and Charles Wolfe. Wireless Sensor Networks for Monitoring Physiological Signals of Multiple Patients. IEEE Transactions on Biomedical Circuits and Systems, 5(4):347--356, 2011.
[15]
H. Kdouh, H. Farhat, G. Zaharia, C. Brousseau, G. Grunfelder, and G.E. Zein. Performance analysis of a hierarchical shipboard Wireless Sensor Network. In Proceedings of the 23rd IEEE International Symposium on Personal Indoor and Mobile Radio Communications (PIMRC), pages 765--770, 2012.
[16]
T. Watteyne, C. Adjih, and X. Vilajosana. Lessons Learned from Large-scale Dense IEEE802.15.4 Connectivity Traces. In Proceedings of the IEEE International Conference on Automation Science and Engineering (CASE), pages 145--150, 2015.
[17]
T. Watteyne, S. Lanzisera, A. Mehta, and K. S. J. Pister. Mitigating Multipath Fading through Channel Hopping in Wireless Sensor Networks. In Proceedings of the IEEE International Conference on Communications (ICC), pages 1--5, 2010.

Cited By

View all
  • (2023)Internet of Things (IoT) security dataset evolution: Challenges and future directionsInternet of Things10.1016/j.iot.2023.10078022(100780)Online publication date: Jul-2023
  • (2022)An Effective Channel Selection Solution for Reliable Scheduling in Industrial IoT NetworksJournal of Network and Systems Management10.1007/s10922-022-09679-z30:4Online publication date: 1-Oct-2022
  • (2021)An Efficient RPL-Based Mechanism for Node-to-Node Communications in IoTIEEE Internet of Things Journal10.1109/JIOT.2020.30386968:9(7152-7169)Online publication date: 1-May-2021
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
PE-WASUN '16: Proceedings of the 13th ACM Symposium on Performance Evaluation of Wireless Ad Hoc, Sensor, & Ubiquitous Networks
November 2016
108 pages
ISBN:9781450345057
DOI:10.1145/2989293
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 November 2016

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. experimentation testbeds
  2. internet of things
  3. large scale
  4. open platform
  5. performance evaluation
  6. repeatability
  7. reproducibility
  8. wireless sensor networks

Qualifiers

  • Research-article

Conference

MSWiM '16
Sponsor:

Acceptance Rates

Overall Acceptance Rate 70 of 240 submissions, 29%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)15
  • Downloads (Last 6 weeks)0
Reflects downloads up to 14 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2023)Internet of Things (IoT) security dataset evolution: Challenges and future directionsInternet of Things10.1016/j.iot.2023.10078022(100780)Online publication date: Jul-2023
  • (2022)An Effective Channel Selection Solution for Reliable Scheduling in Industrial IoT NetworksJournal of Network and Systems Management10.1007/s10922-022-09679-z30:4Online publication date: 1-Oct-2022
  • (2021)An Efficient RPL-Based Mechanism for Node-to-Node Communications in IoTIEEE Internet of Things Journal10.1109/JIOT.2020.30386968:9(7152-7169)Online publication date: 1-May-2021
  • (2020)IETF Reliable and Available Wireless (RAW): Use Cases and Problem StatementAd-Hoc, Mobile, and Wireless Networks10.1007/978-3-030-61746-2_23(303-314)Online publication date: 15-Oct-2020
  • (2019)JamLab-NG: Benchmarking Low-Power Wireless Protocols under Controllable and Repeatable Wi-Fi InterferenceProceedings of the 2019 International Conference on Embedded Wireless Systems and Networks10.5555/3324320.3324331(83-94)Online publication date: 25-Feb-2019
  • (2019)A novel approach to improve handover performanceInternational Journal of Communication Networks and Distributed Systems10.1504/ijcnds.2019.10064123:1(44-68)Online publication date: 1-Jan-2019
  • (2019)Repeatable and Reproducible Wireless Networking Experimentation through Trace-based Simulation2019 International Conference on Wireless and Mobile Computing, Networking and Communications (WiMob)10.1109/WiMOB.2019.8923120(53-58)Online publication date: Oct-2019
  • (2018)Reinforcement Learning Techniques for Optimized Channel Hopping in IEEE 802.15.4-TSCH NetworksProceedings of the 21st ACM International Conference on Modeling, Analysis and Simulation of Wireless and Mobile Systems10.1145/3242102.3242110(99-107)Online publication date: 25-Oct-2018
  • (2018)Moving Beyond Testbeds? Lessons (We) Learned About ConnectivityIEEE Pervasive Computing10.1109/MPRV.2018.287384717:4(15-27)Online publication date: 1-Oct-2018
  • (2018)Local or Global Radio Channel Blacklisting for IEEE 802.15.4-TSCH Networks?2018 IEEE International Conference on Communications (ICC)10.1109/ICC.2018.8423007(1-6)Online publication date: May-2018
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media