Information Bottleneck with Local Consistency

Abstract

Given the joint distribution p(X,Y) of the original variable X and relevant variable Y, the Information Bottleneck (IB) method aims to extract an informative representation of the variable X by compressing it into a “bottleneck” variable T, while maximally preserving the relevant information about the variable Y. In practical applications, when the variable X is compressed into its representation T, however, this method does not take into account the local geometrical property hidden in data spaces, therefore, it is not appropriate to deal with non-linearly separable data. To solve this problem, in this study, we construct an information theoretic framework by integrating local geometrical structures into the IB methods, and propose Locally-Consistent Information Bottleneck (LCIB) method. The LCIB method uses k-nearest neighbor graph to model the local structure, and employs mutual information to measure and guarantee the local consistency of data representations. To find the optimal solution of LCIB algorithm, we adopt a sequential “draw-and-merge” procedure to achieve the converge of our proposed objective function. Experimental results on real data sets demonstrate the effectiveness of the proposed approach.