Abstract
For reaching deterministic self-stabilising distributed synchronisation with commercial-off-the-shelf (COTS) Ethernet, this paper explores the Wire-Adapted Linker-Decoupled EtherNet (WALDEN) architecture with the integration of several mass-produced COTS products. Upon this architecture, basic strategies of distributed synchronisation are discussed. Self-stabilising algorithms are presented and formally analysed. Besides, a prototype system is realised with COTS switches, PHY chips, and low-end ARM series under the WALDEN architecture with a minimum hardware modification. The experimental result shows that a sub-microsecond synchronisation precision can be achieved with this prototype implementation even in the presence of low-quality switch with non-queuing delay-jitters of several microseconds. Among existing solutions, this work features in considering the distributed synchronisation, backward-compatibility, and self-stabilisation problems as a whole.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
For simplicity, we assume the state-timer and the local clock in a node share the same hardware frequency source, which can be easily implemented by configuring the clock-trees of common embedded systems. And as COTS external crystal oscillators could support for \(\rho <10^{-4}\), the difference between \(1+\rho\) and \((1-\rho )^{-1}\) is ignored in computation and analysis.
SIGs which arrive at the same time can be ordered by unique node numbers.
Later arrived SIGs from the same MES node in a round should be ignored in the WS nodes to resist babbling-idiot ES nodes.
As shown later, the nonfaulty nodes would be synchronised at the end of the first round.
For CES nodes, they behave like MES nodes except for they never dispatching SIG.
The dispatching instants of a broadcasted SIG in a nonfaulty node are assumed identical.
In concrete computations, these static edge propagation delays on receiving and dispatching a SIG in an ES node should often be estimated respectively.
For dynamic local delays, this assumption can be supported by slightly adjusting the state-timer according to local delay measurements in actual implementation.
To reduce overall estimation errors, concrete algorithms should select a short path between the S-nodes of two SRs in computing region reference differences.
This can also be supported by slightly adjusting the state-timer according to local delay measurements in actual implementation.
To be heuristic, we prefer simpleness to tightness in the premises of the lemmata and theorems. Tighter bounds could be derived with more detailed models.
We do not use the dependent properties in proving this assumption.
References
Ashjaei M, Liu M, Behnam M, Mifdaoui A, Almeida L, Nolte T (2012) Worst-case delay analysis of master-slave switched ethernet networks. In: Proceeings of the 2nd international workshop on worst-case traversal time—WCTT ’12. ACM Press, New York. https://doi.org/10.1145/2428592.2428595
Ballesteros A, Gessner D, Proenza J, Barranco M, Pedreiras P (2013) Towards preventing error propagation in a real-time ethernet switch. In: International conference on emerging technologies and factory automation (ETFA). https://doi.org/10.1109/ETFA.2013.6648140
Bernus O, Vigmond E (2015) Asymptotic wave propagation in excitable media. Phys Rev E 92(1):010901. https://doi.org/10.1103/PhysRevE.92.010901
Caffarelli LA, Lee KA, Mellet A (2006) Homogenization and flame propagation in periodic excitable media: The asymptotic speed of propagation. Commun Pure Appl Math 59(4):501–525. https://doi.org/10.1002/cpa.20094
Clergue M, Verel S, Formenti E (2018) An iterated local search to find many solutions of the 6-states firing squad synchronization problem. Appl Soft Comput 66:449–461. https://doi.org/10.1016/j.asoc.2018.01.026
Cristian F, Aghili H, Strong R, Dolev D (1995) Atomic broadcast—from simple message diffusion to byzantine agreement. Inf Comput 118(1):158–179. https://doi.org/10.1006/inco.1995.1060
Daliot A, Dolev D (2005) Self-stabilization of byzantine protocols. In: Self-stabilizing systems, proceedings, vol 3764, pp 48–67. https://doi.org/10.1007/11577327_4
Daliot A, Dolev D (2006) Self-stabilizing byzantine pulse synchronization. arXiv preprint. arXiv:cs/0608092
Daliot A, Dolev D, Parnas H (2003) Self-stabilizing pulse synchronization inspired by biological pacemaker networks. In: Proceedings of the 6th international conference on self-stabilizing systems. Springer, Berlin, pp 32–48. https://doi.org/10.1007/3-540-45032-7_3
Dolev D (1982) The byzantine generals strike again. J Algorithms 3(1):14–30. https://doi.org/10.1016/0196-6774(82)90004-9
Dolev D, Hoch EN (2007) Byzantine self-stabilizing pulse in a bounded-delay model. In: SSS. Springer, Heidelberg, pp 234–252. https://doi.org/10.1007/978-3-540-76627-8_19
Dolev D, Lynch NA, Pinter SS, Stark EW, Weihl WE (1986) Reaching approximate agreement in the presence of faults. J ACM 33(3):499–516. https://doi.org/10.1145/5925.5931
Driscoll K, Hall B, Sivencrona H, Zumsteg P (2003) Byzantine fault tolerance, from theory to reality. In: Computer safety, reliability, and security, proceedings, vol 2788, pp 235–248. https://doi.org/10.1007/978-3-540-39878-3_19
Elshuber M, Obermaisser R (2013) Dependable and predictable time-triggered ethernet networks with COTS components. J Syst Archit 59(9):679–690. https://doi.org/10.1016/j.sysarc.2013.07.003
Ferrant J, Gilson M, Jobert S, Mayer M, Ouellette M, Montini L, Rodrigues S, Ruffini S (2008) Synchronous ethernet: a method to transport synchronization. IEEE Commun Mag 46(9):126–134. https://doi.org/10.1109/MCOM.2008.4623717
Galloway B, Hancke GP (2013) Introduction to industrial control networks. IEEE Commun Surv Tutor 15(2):860–880. https://doi.org/10.1109/Surv.2012.071812.00124
Garner ea Geoffrey M (2011) Timing and synchronization for time-sensitive applications in bridged local area networks. https://standards.ieee.org/standard/802_1AS-2011.html
Garner ea Geoffrey M (2019) Draft standard for local and metropolitan area networks - timing and synchronization for time-sensitive applications. IEEE P8021AS-Rev/D83. https://1.ieee802.org/tsn/802-1as-rev/
Gessner D, Proenza J, Barranco M (2014a) A proposal for managing the redundancy provided by the flexible time-triggered replicated star for Ethernet. In: Factory communication systems. https://doi.org/10.1109/WFCS.2014.6837600
Gessner D, Proenza J, Barranco M (2014b) A proposal for master replica control in the flexible time-triggered replicated star for ethernet. In: 2014 10th IEEE workshop on factory communication systems (WFCS 2014), pp 1–4. https://doi.org/10.1109/WFCS.2014.6837599
Gessner D, Furió I, Proenza J (2015) Towards a layered architecture for the flexible time-triggered replicated star for ethernet. In: 2015 IEEE 20th conference on emerging technologies & factory automation (ETFA), pp 1–4. https://doi.org/10.1109/ETFA.2015.7301636
Hall B, Driscoll K, Paulitsch M, Dajani-Brown S (2005) Ringing out fault tolerance. a new ring network for superior low-cost dependability. 2005 international conference on dependable systems and networks, proceedings, pp 298–307. https://doi.org/10.1109/Dsn.2005.84
Halpern JY, Simons B, Strong R, Dolev D (1984) Fault-tolerant clock synchronization. In: Proceedings of the third annual ACM symposium on Principles of distributed computing, pp 89–102. https://doi.org/10.1145/800222.806739
Hoch EN, Dolev D, Daliot A (2006) Self-stabilizing byzantine digital clock synchronization. In: Stabilization, safety, and security of distributed systems, Proceedings, pp 350–362. https://doi.org/10.1007/978-3-540-49823-0_25
Hodgkin AL, Huxley AF (1952) A quantitative description of membrane current and its application to conduction and excitation in nerve. J Physiol 117(4):500–544. https://doi.org/10.1016/S0092-8240(05)80004-7
IEEE (2019) Official website of the 802.1 time-sensitive networking task group. https://1.ieee802.org/tsn/#Published_TSN_Standards. Accessed 12 July 2019
IEEE1588 (2008) Standard for a precision clock synchronization protocol for networked measurement and control systems. IEEE Standard 1588–2008: https://doi.org/10.1109/IEEESTD.2008.4579760
Instruments T (2019) Sn74lvc1g08. www.ti.com/lit/ds/symlink/sn74lvc1g08.pdf. Accessed 12 July 2019
Kopetz H (2003) Fault containment and error detection in the time-triggered architecture. In: The sixth international symposium on autonomous decentralized systems, 2003 (ISADS 2003), pp 139–146. https://doi.org/10.1109/ISADS.2003.1193942
Kopetz H (2011) Real-time systems: design principles for distributed embedded applications. Springer, Incorporated. https://doi.org/10.1007/978-1-4419-8237-7
Kopetz H, Bauer G (2003) The time-triggered architecture. Proc IEEE 91(1):112–126. https://doi.org/10.1109/JPROC.2002.805821
Kopetz H, Grunsteidl G (1993) Ttp - a time-triggered protocol for fault-tolerant real-time systems. In: FTCS-23: The twenty-third international symposium on fault-tolerant computing, pp 524–533. https://doi.org/10.1109/FTCS.1993.627355
Kopetz H, Ochsenreiter W (1987) Clock synchronization in distributed real-time systems. IEEE Trans Comput 100(8):933–940. https://doi.org/10.1109/tc.1987.5009516
Kopetz H, Ademaj A, Grillinger P, Steinhammer K (2005) The time-triggered ethernet (TTE) design. In: Eighth IEEE international symposium on object-oriented real-time distributed computing (ISORC’05), pp 22–33. https://doi.org/10.1109/ISORC.2005.56
Lamport L, Melliarsmith PM (1985) Synchronizing clocks in the presence of faults. J ACM 32(1):52–78. https://doi.org/10.1145/2455.2457
Lamport L, Shostak R, Pease M (1982) The byzantine generals problem. ACM Trans Program Lang Syst 4(3):382–401. https://doi.org/10.1145/357172.357176
Le Boudec JY, Thiran P, Giordano S (2000) A short tutorial on network calculus ii: Min-plus system theory applied to communication networks. In: ISCAS 2000: IEEE international symposium on circuits and systems—proceedings, vol IV, pp 365–368. https://doi.org/10.1109/ISCAS.2000.858764
Lenzen C, Rybicki J (2019) Self-stabilising byzantine clock synchronisation is almost as easy as consensus. J ACM 66(5): Article 32. https://doi.org/10.1145/3339471
Lenzen C, Rybicki J, Suomela J (2015) Towards optimal synchronous counting. In: Proceedings of the 2015 ACM symposium on principles of distributed computing (PODC ’15). Association for Computing Machinery, New York, NY, USA, pp 441–450. https://doi.org/10.1145/2767386.2767423
Lisova E, Uhlemann E, Steiner W, Åkerberg J, Björkman M (2016a) Risk evaluation of an arp poisoning attack on clock synchronization for industrial applications. In: 2016 IEEE international conference on industrial technology (ICIT), pp 872–878. https://doi.org/10.1109/ICIT.2016.7474866
Lisova E, Gutierrez M, Steiner W, Uhlemann E, Akerberg J, Dobrin R, Bjorkman M (2016b) Protecting clock synchronization. In: JECE 2016. https://doi.org/10.1155/2016/6297476
Lisova E, Uhlemann E, Åkerberg J, Björkman M (2017) Monitoring of clock synchronization in cyber-physical systems: a sensitivity analysis. In: 2017 international conference on Internet of Things, embedded systems and communications (IINTEC), pp 134–139. https://doi.org/10.1109/IINTEC.2017.8325927
Loeser J, Haertig H (2004a) Low-latency hard real-time communication over switched ethernet. In: Proceedings. 16th Euromicro Conference on Real-Time Systems, 2004 (ECRTS 2004), pp 13–22. https://doi.org/10.1109/EMRTS.2004.1310992
Loeser J, Haertig H (2004b) Using switched ethernet for hard real-time communication. In: International conference on parallel computing in electrical engineering, pp 349–353. https://doi.org/10.1109/Pcee.2004.1335634
Marau R, Almeida L, Pedreiras P (2006) Enhancing real-time communication over cots Ethernet switches. In: 2006 IEEE international workshop on factory communication systems, pp 295–302. https://doi.org/10.1109/WFCS.2006.1704170
Mills DL (1991) Internet time synchronization: the network time protocol. IEEE Trans Commun 39(10):1482–1493. https://doi.org/10.1109/26.103043
Moore EF (1964) The firing squad synchronization problem. In: Sequential machines, selected papers. Addison-Wesley, Reading
Pedreiras P, Almeida L (2003) The flexible time-triggered (ftt) paradigm: an approach to qos management in distributed real-time systems. In: International symposium on parallel & distributed processing. https://doi.org/10.1109/IPDPS.2003.1213243
Rodriguez-Navas G, Proenza J (2013) A proposal for flexible, real-time and consistent multicast in ftt/hartes switched ethernet. In: Emerging technologies & factory automation. https://doi.org/10.1109/ETFA.2013.6648153
SAE (2011) Time-triggered ethernet. SAE International, Warrendale
Schneider FB (1987) Understanding protocols for byzantine clock synchronization. Tech. Report, USA. https://doi.org/10.5555/866262
Segall A (1983) Distributed network protocols. IEEE Trans Inf Theory 29(1):23–35. https://doi.org/10.1109/TIT.1983.1056620
Semiconductor N (2008) Dp83848c/i/vyb/yb phyter\(^{{\tt T\it M}}\) QFP single port 10/100 mb/s ethernet physical layer transceiver datasheet (rev edn). https://www.ti.com/product/DP83848C. Accessed 12 July 2019
Sivashinsky G (1977) Nonlinear analysis of hydrodynamic instability in laminar flames-i. derivation of basic equations. Acta Astronaut 4:1177–1206. https://doi.org/10.1016/0094-5765(77)90097-2
Steiner W, Peón PG, Gutiérrez M, Mehmed A, Rodriguez-Navas G, Lisova E, Pozo F (2016) Next generation real-time networks based on it technologies. In: 2016 IEEE 21st international conference on emerging technologies and factory automation (ETFA), pp 1–8. https://doi.org/10.1109/ETFA.2016.7733580
STMicroelectronics (2015) Rm0090 reference manual. https://www.st.com/. Accessed 12 July 2019
Ullmann M, Vögeler M (2009) Delay attacks—implication on ntp and ptp time synchronization. In: 2009 International symposium on precision clock synchronization for measurement, control and communication, IEEE, pp 1–6. https://doi.org/10.1109/ISPCS.2009.5340224
Uscinski BJ (1977) The elements of wave propagation in random media. McGraw-Hall, New York. https://doi.org/10.1088/0031-9112/29/4/032
Welch JL, Lynch N (1988) A new fault-tolerant algorithm for clock synchronization. Inf Comput 77(1):1–36. https://doi.org/10.1016/0890-5401(88)90043-0
Winfree AT (1980) The geometry of biological time. Springer, New York. https://doi.org/10.1007/978-1-4757-3484-3
Acknowledgements
The author would like to thank the reviewers for their valuable and constructive comments.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Yu, S., Zhu, J. & Yang, J. Reaching self-stabilising distributed synchronisation with COTS Ethernet components: the WALDEN approach. Real-Time Syst 57, 347–386 (2021). https://doi.org/10.1007/s11241-020-09356-x
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11241-020-09356-x