Tobias Herb
ABSTRACT
Many machine learning (ML) models are of a stochastic nature. We aim to combine the principles of weak consistency with large scale distributed machine learning. We see interesting opportunities in this domain in (1) perceiving parallel ML algorithms based on model replication as a "collaborative task" where local progress on models is instantaneously exchanged and by (2) making this exchange more efficient by exploiting the underlying stochastic nature. Based on this motivation, we extend the notion of consistency for replicated objects with intrinsic stochastic structure and introduce harmonization as the reconciliation principle to enable efficient consistency maintenance of these objects. We present as a concrete application the harmonization of replicated ML models.
- J. Duchi, E. Hazan, and Y. Singer. Adaptive subgradient methods for online learning and stochastic optimization. J. Mach. Learn. Res., 12:2121--2159, July 2011. Google Scholar
Digital Library
- G. Gibson and N. Zeldovich, editors. 2014 USENIX Annual Technical Conference, USENIX ATC '14, Philadelphia, PA, USA, June 19-20, 2014. USENIX Association, 2014. Google ScholarDigital Library
- R. McDonald, K. Hall, and G. Mann. Distributed training strategies for the structured perceptron. HLT '10, pages 456--464, Stroudsburg, PA, USA, 2010. Association for Computational Linguistics. Google ScholarDigital Library
- X. Meng, J. K. Bradley, B. Yavuz, E. R. Sparks, S. Venkataraman, D. Liu, J. Freeman, D. B. Tsai, M. Amde, S. Owen, D. Xin, R. Xin, M. J. Franklin, R. Zadeh, M. Zaharia, and A. Talwalkar. Mllib: Machine learning in apache spark. CoRR, abs/1505.06807, 2015.Google Scholar
- A. S. Tanenbaum and M. v. Steen. Distributed Systems: Principles and Paradigms (2Nd Edition). Prentice-Hall, Inc., Upper Saddle River, NJ, USA, 2006. Google ScholarDigital Library
- H. Yu and A. Vahdat. Design and evaluation of a conit-based continuous consistency model for replicated services. ACM Trans. Comput. Syst., 20(3):239--282, Aug. 2002. Google ScholarDigital Library
- M. Zinkevich, M. Weimer, L. Li, and A. J. Smola. Parallelized stochastic gradient descent. In J. D. Lafferty, C. K. I. Williams, J. Shawe-Taylor, R. S. Zemel, and A. Culotta, editors, Advances in Neural Information Processing Systems 23, pages 2595--2603. Curran Associates, Inc., 2010.Google ScholarDigital Library
- Wei Dai, Abhimanu Kumar, Jinliang Wei, Qirong Ho, Garth A. Gibson, and Eric P. Xing. High-performance distributed ML at scale through parameter server consistency models. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, January 25-30, 2015, Austin, Texas, USA., pages 79--87, 2015. Google ScholarDigital Library
- Léon Bottou. Large-scale machine learning with stochastic gradient descent. In Yves Lechevallier and Gilbert Saporta, editors, Proceedings of the 19th International Conference on Computational Statistics (COMPSTAT'2010), pages 177--187, Paris, France, August 2010. Springer.Google ScholarCross Ref
- Graham Cormode and S. Muthukrishnan. An improved data stream summary: The count-min sketch and its applications. J. Algorithms, 55(1):58--75, April 2005. Google ScholarDigital Library
Recommendations
Making weak consistency great again
PaPoC '16: Proceedings of the 2nd Workshop on the Principles and Practice of Consistency for Distributed DataThis paper focuses on the problem of implementing web applications on top of weakly consistent geo-replicated systems. Several techniques, such as CRDTs, have been proposed to achieve state convergence on a per-object and per-data type basis. However, ...
Collaborative Annotation of Videos Relying on Weak Consistency
This work discusses a distributed interactive video system that supports video annotation using simultaneous hyperlinking by multiple users. The users mark and annotate objects within the video with links to other media such as text, images, websites, ...
Distributed B-Tree with Weak Consistency
NETYS 2013: Revised Selected Papers of the First International Conference on Networked Systems - Volume 7853B-tree is a widely used data-structure indexing data for efficient Retrieval. We consider a decentralized B-tree, were parts of the structure are distributed among different processors and some parts are replicated, thus providing a decentralized ...
Comments