Online learning with expert advice plays an important role in various machine learning tasks. When multiple experts are available at hand, the learner can perform the learning task after collecting advice from the experts. In conventional learning with expert advice, the learner performs the learning task by either selecting the advice of one expert or combining all experts' advice. In the present paper, we consider the case where taking advice from experts incur costs, and the learner can only afford limited number of queries due to budget limitation. Expert advice are modeled as signals lying on a graph, and the learner selects and requests advice from only a subset of experts at each time based on the graph structure. Upon observing the incurred loss, the graph is refined actively ‘on the fly.' The proposed approach is proved to achieve sublinear regret. Numerical tests on real datasets are also presented to showcase the merits of our proposed approach.