Skip to main content
SpringerLink
Log in

Benchmarking Functionalities of Domestic Service Robots Through Scientific Competitions

  • Technical Contribution
  • Published:
KI - Künstliche Intelligenz Aims and scope Submit manuscript

Abstract

Benchmarking via carefully designed competitions makes it possible to provide a common framework for the rigorous comparison of intelligent and autonomous systems; competitions may play the role of scientific experiments while being appealing both to researchers and to the general public thus promoting critical analysis of systems outside the labs. This paper describes our approach to benchmarking domestic service robots through organizing recurrent competitions under the European Robotics League. It details the tools and benchmarks designed to evaluate the performance of robots at task and functionality levels. In particular, the functionality benchmarks for object perception and navigation are described and an overview of the new benchmarks to appear in the league is presented.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Notes

  1. With replicable experiments we refer to experiments, and benchmarks, which are both reproducible and repeatable.

  2. List of certified testbeds: http://www.eu-robotics.net/robotics_league/erl-consumer/certified-test-beds.

  3. Example: http://sites.google.com/site/erlsrroman2017/.

  4. RSBB: http://github.com/rockin-robot-challenge/rsbb.

  5. RSBB robot communication package: http://github.com/rockin-robot-challenge/at_home_rsbb_comm_ros.

  6. Benchmark script tutorial: http://github.com/rockin-robot-challenge/rsbb/blob/master/rsbb_etc/doc/bmbox/benchmark_script_tutorial.md.

  7. Example: http://sites.google.com/site/erlsrroman2017/resources/FBM1inputb.

  8. A 3D model of the positioner used at the ERL is available at https://github.com/rockin-robot-challenge/ERL-Reports-Attachements.

  9. Dataset: https://doi.org/10.5281/zenodo.2547688.

References

  1. Amigoni F, Bastianelli E, Berghofer J, Bonarini A, Fontana G, Hochgeschwender N, Iocchi L, Kraetzschmar GK, Lima PU, Matteucci M, Miraldo M, Nardi D, Schiaffonati V (2015) Competitions for benchmarking: task and functionality scoring complete performance assessment. IEEE Robot Autom Mag 22(3):53–61

    Article  Google Scholar 

  2. Lima PU (2017) RoCKIn: benchmarking through robot competitions. InTechOpen, Rijeka

    Google Scholar 

  3. Dias J, Althoefer K, Lima PU (2016) Robot competitions: what did we learn? IEEE Robot Autom Mag 23(1):16–18

    Article  Google Scholar 

  4. Anderson M, Jenkins OC, Osentoski S (2011) Recasting robotics challenges as experiments. IEEE Robot Autom Mag 18(2):10–11

    Article  Google Scholar 

  5. Prassler E, Munich ME, Pirjanian P, Kosuge K (2016) Domestic robotics. Springer handbook of robotics. Springer, Cham, pp 1729–1758

    Chapter  Google Scholar 

  6. Young JE, Hawkins R, Sharlin E, Igarashi T (2009) Toward acceptable domestic robots: applying insights from social psychology. Int J Soc Robot 1(1):95

    Article  Google Scholar 

  7. Müller AC, Behnke S (2014) Learning depth-sensitive conditional random fields for semantic segmentation of rgb-d images. In: IEEE international conference on robotics and automation (ICRA), pp 6232-6237

  8. Messias J, Ventura R, Lima P, Sequeira J, Alvito P, Marques C, Carriço P (2014) A robotic platform for edutainment activities in a pediatric hospital. In: IEEE international conference on autonomous robot systems and competitions (ICARSC), pp 193-198

  9. Muszynski S, Stückler J, Behnke S (2012) Adjustable autonomy for mobile teleoperation of personal service robots. In: IEEE international symposium on robot and human interactive communication (Ro-MAN), pp 933–940

  10. Pineda LA, Salinas L, Meza IV, Rascon C, Fuentes G (2013) Sitlog: a programming language for service robot tasks. Int J Adv Rob Syst 10(10):358

    Article  Google Scholar 

  11. RoCKIn: Robot competitions kick innovation in cognitive systems. http://rockinrobotchallenge.eu. Accessed Dec 2018

  12. euRathlon: An outdoor robotics challenge for land, sea and air. https://www.eurathlon.eu. Accessed Dec 2018

  13. European Robotics League. https://www.eu-robotics.net/robotics_league/. Accessed Dec 2018

  14. Basiri M et al (2018) Rulebook of the European robotic league for consumer service robots. https://eu-robotics.net/robotics_league/erl-consumer

Download references

Acknowledgements

This research was supported by EU Horizon 2020 Programme for research, technological development and demonstration under Grants agreements n688441 and n780086.

Author information

Authors and Affiliations

  1. Institute For Systems and Robotics, Tecnico Lisboa, Lisbon, Portugal

    Meysam Basiri, Enrico Piazza & Pedro Lima

  2. Politecnico di Milano, Milan, Italy

    Matteo Matteucci

Authors
  1. Meysam Basiri
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Enrico Piazza
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Matteo Matteucci
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Pedro Lima
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Meysam Basiri.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Basiri, M., Piazza, E., Matteucci, M. et al. Benchmarking Functionalities of Domestic Service Robots Through Scientific Competitions. Künstl Intell 33, 357–367 (2019). https://doi.org/10.1007/s13218-019-00619-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13218-019-00619-9

Keywords

Search

Navigation