ABSTRACT
Robots are deployed by a Web search engine for collecting information from different Web servers in order to maintain the currency of its data base of Web pages. In this paper, we investigate the number of robots to be used by a search engine so as to maximize the currency of the data base without putting an unnecessary load on the network. We adopt a finite-buffer queueing model to represent the system. The arrivals to the queueing system are Web pages brought by the robots; service corresponds to the indexing of these pages. Good performance requires that the number of robots, and thus the arrival rate of the queueing system, be chosen so that the indexing queue is rarely starved or saturated. Thus, we formulate a multi-criteria stochastic optimization problem with the loss rate and empty-buffer probability being the criteria. We take the common approach of reducing the problem to one with a single objective that is a linear function of the given criteria. Both static and dynamic policies can be considered. In the static setting the number of robots is held fixed; in the dynamic setting robots may be re-activated/de-activated as a function of the state. Under the assumption that arrivals form a Poisson process and that service times are independent and exponentially distributed random variables, we determine an optimal decision rule for the dynamic setting, i.e., a rule that varies the number of robots in such a way as to minimize a given linear function of the loss rate and empty-buffer probability. Our results are compared with known results for the static case. A numerical study indicates that substantial gains can be achieved by dynamically controlling the activity of the robots.
- 1.Bertsekas, D. P., Dynamic Programming. Deterministic and Stochastic Models, Prentice-Hall, Inc., Englewood Cliffs, 1987. Google ScholarDigital Library
- 2.Coffman Jr., E. G., Liu, Z. and Weber, R. R., "Optimal robot scheduling for Web search engines", J. Scheduling, 1, pp. 14-22, 1998.Google ScholarCross Ref
- 3.Kleinrock, L., Queueing Systems, Vol. I, Wiley & Sons, New York, 1975. Google ScholarDigital Library
- 4.Puterman, M. L., Markov Decision Processes, Wiley, New York, 1994.Google Scholar
- 5.Ross, S. M., Introduction to Stochastic Dynamic Programming, Academic Press, New York, 1983. Google ScholarDigital Library
- 6.J. Talim, Z. Liu, Ph. Nain, and E. G. Coffman, Jr. "Optimizing the number of robots for Web search engines", Telecommunication Systems, vol. 17, pp. 245-266, 2001.Google ScholarDigital Library
- 7.Wolff, R. L., "Poisson Arrivals See Time Averages," Oper. Res., vol. 30, pp. 223-231, 1982.Google ScholarDigital Library
Index Terms
- Controlling the robots of Web search engines
Recommendations
Optimizing the Number of Robots for Web Search Engines
Robots are deployed by a Web search engine for collecting information from different Web servers in order to maintain the currency of its data base of Web pages. In this paper, we investigate the number of robots to be used by a search engine so as to ...
Controlling the robots of Web search engines
Robots are deployed by a Web search engine for collecting information from different Web servers in order to maintain the currency of its data base of Web pages. In this paper, we investigate the number of robots to be used by a search engine so as to ...
Analysis of a queueing system with impatient customers and working vacations
QTNA '11: Proceedings of the 6th International Conference on Queueing Theory and Network ApplicationsIn this paper, we consider an M/M/1 queueing system with working vacations and impatient customers, where we examine the case that customers' impatience is due to a working vacation. During the working vacation, customers are served at a slower than ...
Comments