Title
Information Bottleneck Problem Revisited
Document Type
Conference Proceeding
Publication Date
9-1-2019
Abstract
In this paper, we revisit the information bottleneck problem whose formulation and solution are of great importance in both information theory and statistical learning applications. We go into details as to why the problem was first introduced and how the algorithm proposed using Lagrangian method to solve such problems fell short of an exact solution. We then revisit the limitations of such Lagrangian methods, and propose to adopt a more systematic method, namely, Alternate Direction Method of Multipliers (ADMM) to develop a more efficient ADMM algorithm with randomized permutation orders to solve such problems. More importantly, we mathematically demonstrate how our suggested method outperforms the original Information Bottleneck (IB) method. At the end, we provide numerical results to demonstrate the notable advantages our algorithm attains as compared with the well-known IB approach in terms of both attained objective function values and the resulting constraints. We further inspect the concepts of accuracy and convergence and the trade-off between them in our method.
Publication Source (Journal or Book title)
2019 57th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2019
First Page
40
Last Page
47
Recommended Citation
Bayat, F., & Wei, S. (2019). Information Bottleneck Problem Revisited. 2019 57th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2019, 40-47. https://doi.org/10.1109/ALLERTON.2019.8919752