Online distributed stochastic learning algorithm for convex optimization in time-varying directed networks

Abstract

This paper investigates a online learning optimization problem in a distributed manner, where a set of agents aim to cooperatively minimize the sum of local time-varying cost functions while communications among agents depend on a sequence of time-varying directed graphs. For such a problem, we first propose a online distributed stochastic (sub)gradient push-sum algorithm by utilizing distributed optimization methods and push-sum protocols. Then we analyze the regret bounds for the proposed algorithm on the cases when the cost functions are convex and strongly convex, respectively. The bound on the expected regret for convex functions grows sub-linearly with order of $\mathcal{O}\left(\sqrt{T}\right),$where T is the time horizon. When the cost functions are strongly convex with Lipschitz gradients, the regret bound has an improved rate with order of $\mathcal{O}\left(\ln T\right)$. Numerical simulations on the localization in wireless sensor networks are used to show the effectiveness of the proposed algorithm.

Introduction

In recent years, distributed optimization has been receiving a lot of attention in information sciences and engineering fields, such as machine learning, control applications for wireless networks, signal processing, and power networks and sensor networks, e.g., see [1], [2], [3], [4], [5], [6], [7], [8], [9]. Compared to centralized optimization, the distributed optimization has an essential difference characterized by the lack of full knowledge about the overall problem structure. This means that all agents of a network collectively minimize an optimization problem by local information exchange and without any centralized coordination, and each agent is only allowed to exchange the information from its immediate neighbors over the network. The underlying communication among agents can be modeled as a undirected or directed time-varying graph. The main task of distributed optimization is to collaboratively minimize the sum of several local cost functions in general, where individual agent holds a private copy of one specific cost function, see [10], for more details. As well-known, there are many recent researches focusing on distributed optimization problems and their applications, see [11], [12], [13], [14], [15].

However, many practical scenarios in distributed optimization frequently encounter dynamically changing and uncertain environments. For examples, observations are time-changing due to noises in parameter estimation problems by using sensor networks, and uncertainties play an important role in the schedule of renewable energy in power systems. To address some of these issues, the online optimization is effective to deal with uncertainties arising in these problems. Different from distributed optimization, online optimization is to minimize a time-vary cost function, and design a online algorithm to reduce the so-called regret, which measures the gap between the accumulated collective cost and the cost obtained by the best single hindsight decision made by a hypothetical decision maker knowing the cost functions in advance. A online algorithm can be claimed “good” when the regret is sub-linear [16]. Online optimization problems have been studied extensively in literatures. For solving a online convex optimization, Zinkevich et al. [16] firstly proposed a gradient-based method and gave a regret bound with order of $\mathcal{O}\left(\sqrt{T}\right),$where T is the time horizon. Later, Hazan et al. [17] obtained an improved regret rate of $\mathcal{O}\left(\ln T\right)$for twice differentiable strongly convex functions. For more results on online optimization, please refer to the survey [18] and references therein.

However, the design for most of online algorithms stated above is based on centralized architectures. Until recently, motivated the interest in decentralized optimization and its many applications, distributed versions of online optimization are developed, see [19], [20]. In a online distributed optimization, the global cost function associated a multi-agent network is represented as the sum of local cost functions, where each local function is allocated to a agent varying possibly over time, and this change is seen by agents only in hindsight. The goal is to design a online distributed algorithm that cooperatively minimize the global cost function across a time horizon. By contrasts, online distributed optimization inherently differs from distributed or online optimization [21]. Based on consensus schemes, gradient descent methods for online distributed optimization were proposed in [22], [23], where all agents are able to collaboratively reduce their average regret. The authors in [22] proposed a consensus based dual averaging algorithm over undirected graphs. Motivated by the saddle-point dynamics in [24], subgradient-based online distributed algorithm was proposed on weight-balanced network topologies [23]. Based on the assumption that communication weight matrices are double stochastic, Tsianos et al. [25] introduced a gossip-based protocol for online distributed convex optimization. Later, Hosseini et al. [26] extended their previous results [22] to accommodate for time-varying weights, but on a fixed directed graph.

Nevertheless, most of the works cited above assumed that communications among agents are either fixed or undirected. Moreover, due to uncertainties in reality, there exist noises in the evaluation of (sub)gradients when designed algorithms. Recently, Akbari et al. [21] generalized the gradient push-sum algorithm in [27] to online setting, and proposed a discrete time online distributed algorithm over time-varying directed graphs. In [28], Lee et al. investigated a online distributed optimization problem with coupled inequality constraints and designed a primal-dual online distributed algorithm. When the calculation of gradients existed noises, a decentralized stochastic variant of dual averaging methods was proposed in [29].

Motivated by recent works [27], [29], we propose a online distributed stochastic (sub)gradient push-sum algorithm in this paper. The idea of our method is based push-sum protocol on imbalanced directed graphs [27], [30]. Each agent is required to know its out-degree at each time, without requiring knowledge of either the number of agents or the graph sequence. Meanwhile, the noisy (sub)gradients are also taken into consideration. For the proposed algorithm, we obtain the regret bounds on the cases of convex and strongly convex cost functions, respectively. By contrasts, we extend the methods in [27], [30] to online setting, and explicitly give regret estimates. Compared with the work in [21], we improve the bound of regret from $\mathcal{O}\left({\ln }^{2}T\right)$to $\mathcal{O}\left(\ln T\right)$for strongly convex objective functions while taking noisy gradients into consideration.

The remainder of this paper is organized as follows. In Section 2, we state the related problem, useful assumptions and algorithm. In Section 3, we give regret analysis and obtain main results. Numerical simulations are given in Section 4. Finally, Section 5 draws some conclusions.