Skip to main content
SpringerLink
Log in

An Effective Region Force for Some Variational Models for Learning and Clustering

  • Published:
Journal of Scientific Computing Aims and scope Submit manuscript

Abstract

In this paper we propose two variational models for semi-supervised clustering of high-dimensional data. The new models produce substantial improvements of the classification accuracy in comparison with the corresponding models without the regional force in cases that the sample rate is relatively low. For the proposed models, the data points are modeled as vertices of a weighted graph, and the labeling function defined on each vertex takes values from the unit simplex, which can be interpreted as the probability of belonging to each class. The algorithm is proposed as a minimization of a convex functional of the labeling function. The first model combines the Rayleigh quotient for the graph Laplacian and a region-force term, and the second one only replaces the Rayleigh quotient with the total variation of the labeling function. The region-force term is calculated by the affinity between each vertex and the training samples, characterizing the conditional probability of each vertex belonging to each class. The numerical methods for solving these two versions of the proposed algorithm are presented, and both are tested on several benchmark data sets such as handwritten digits (MNIST) and moons data. Experiments indicate that the classification accuracy and the computational speed are competitive with the state-of-the-art in multi-class semi-supervised clustering algorithms. Numerical experiments also confirm that the total variation model out performs the Laplacian counter part in most of the tests.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  1. Bonettini, S., Ruggiero, V.: On the convergence of primal dual hybrid gradient algorithms for total variation image restoration. J. Math. Imaging Vis. 44(3), 236–253 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  2. Boykov, Y., Funka-Lea, G.: Graph cuts and efficient N-D image segmentation. Int. J. Comput. Vis. 70(2), 109–131 (2006)

    Article  Google Scholar 

  3. Bresson, X., Laurent, T., Uminsky, D., von Brecht, J.: Multiclass total variation clustering. In: Advances in Neural Information Processing Systems, pp. 1421–1429 (2013)

  4. Bresson, X., Tai, X.-C., Chan, T.F., Szlam, A.: Multi-class transductive learning based on l1 relaxations of cheeger cut and Mumford-Shah–Potts model. J. Math. Imaging Vis. 49(1), 191–201 (2013)

    Article  MATH  Google Scholar 

  5. Bühler, T., Hein, M.: Spectral clustering based on the graph p-Laplacian. In: Proceedings of the 26th Annual International Conference on Machine Learning, pp. 81–88. ACM (2009)

  6. Chan, T.F., Vese, L.A.: Active contours without edges. IEEE Trans. Image Process. 10(2), 266–277 (2001)

    Article  MATH  Google Scholar 

  7. Chapelle, O., Schölkopf, B., Zien, A. (eds.): Semi-supervised Learning. MIT Press, Cambridge (2006)

    Google Scholar 

  8. Chen, Y., Ye, X.: Projection onto a simplex (2011). arXiv:1101.6081

  9. Chung, F: Spectral graph theory. In: CBMS Regional Conference Series in Mathematics, No. 92. American Mathematical Society, New York (1996)

  10. Coifman, R.R., Lafon, S., Lee, A.B., Maggioni, M., Nadler, B., Warner, F., Zucker, S.W.: Geometric diffusions as a tool for harmonic analysis and structure definition of data: diffusion maps. Proc. Natl. Acad. Sci. USA 102(21), 7426–7431 (2005)

    Article  Google Scholar 

  11. Cour, T., Benezit, F., Shi, J.: Spectral segmentation with multiscale graph decomposition. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2005 (CVPR 2005), vol. 2, pp. 1124–1131 (2005)

  12. Dai, Y.-H., Fletcher, R.: Projected Barzilai-Borwein methods for large-scale box-constrained quadratic programming. Numer. Math. 100(1), 21–47 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  13. Garcia-Cardona, C., Merkurjev, E., Bertozzi, A.L., Flenner, A., Percus, A.G.: Multiclass data segmentation using diffuse interface methods on graphs. IEEE Trans. Pattern Anal. Mach. Intell. 36(8), 1600–1613 (2014)

    Article  MATH  Google Scholar 

  14. Gibou, F., Fedkiw, R.: A fast hybrid k-means level set algorithm for segmentation. In: 4th Annual Hawaii International Conference on Statistics and Mathematics, Hawaii, USA, pp. 281–291 (2005)

  15. Gilboa, G., Osher, S.: Nonlocal operators with applications to image processing. Multiscale Model. Simul. 7(3), 1005–1028 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  16. Hein, M., Setzer, S.: Beyond spectral clustering—tight relaxations of balanced graph cuts. In: Shawe-Taylor, J., Zemel, R.S., Bartlett, P.L., Pereira, F., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems, vol. 24, pp. 2366–2374. Curran Associates, Inc. (2011)

  17. Hu, H., Sunu, J., Bertozzi, A.L.: Multi-class graph Mumford–Shah model for plume detection using the MBO scheme. In: Energy Minimization Methods in Computer Vision and Pattern Recognition, pp. 209–222. Springer (2015)

  18. Lézoray, O., Elmoataz, A., Ta, V.T.: Nonlocal PDEs on graphs for active contours models with applications to image segmentation and data clustering. In: 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 873–876 (2012)

  19. Mallat, S.: Group invariant scattering. Commun. Pure Appl. Math. 65(10), 1331–1398 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  20. Merkurjev, E., Bae, E., Bertozzi, A.L., Tai, X.-C.: Global binary optimization on graphs for classification of high-dimensional data. J. Math. Imaging Vis. 52(3), 414–435 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  21. Merkurjev, E., Kostic, T., Bertozzi, A.L.: An mbo scheme on graphs for classification and image processing. SIAM J. Imaging Sci. 6(4), 1903–1930 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  22. Merriman, B., Bence, J.K., Osher, S.: Diffusion Generated Motion by Mean Curvature. Department of Mathematics, University of California, Los Angeles (1992)

    Google Scholar 

  23. Muja, M., Lowe, D.G.: Fast approximate nearest neighbors with automatic algorithm configuration. VISAPP (1) 2, 331–340 (2009)

    Google Scholar 

  24. Ng, A.Y., Jordan, M.I., Weiss, Y., et al.: On spectral clustering: analysis and an algorithm. Adv. Neural Inf. Process. Syst. 2, 849–856 (2002)

    Google Scholar 

  25. Osting, B., White, C., Oudet, E.: Minimal Dirichlet energy partitions for graphs. SIAM J. Sci. Comput. 36(4), A1635–A1651 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  26. Schölkopf, B., Tsuda, K., Vert, J.-P.: Kernel Methods in Computational Biology. MIT Press, Cambridg (2004)

    Google Scholar 

  27. Shi, J., Malik, J.: Normalized cuts and image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 22(8), 888–905 (2000)

    Article  Google Scholar 

  28. Silpa-Anan, C., Hartley, R.: Optimised KD-trees for fast image descriptor matching. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8. IEEE (2008)

  29. Singhal, A.: Modern information retrieval: a brief overview. IEEE Data Eng. Bull. 24(4), 35–43 (2001)

    Google Scholar 

  30. Spielman, D., Srivastava, N.: Graph sparsification by effective resistances. SIAM J. Comput. 40(6), 1913–1926 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  31. Subramanya, A., Bilmes, J.: Semi-supervised learning with measure propagation. J. Mach. Learn. Res. 12, 3311–3370 (2011)

    MathSciNet  MATH  Google Scholar 

  32. Szlam, A.D., Maggioni, M., Coifman, R.R.: Regularization on graphs with function-adapted diffusion processes. J. Mach. Learn. Res. 9, 1711–1739 (2008)

    MathSciNet  MATH  Google Scholar 

  33. Tong, S., Koller, D.: Support vector machine active learning with applications to text classification. J. Mach. Learn. Res. 2, 45–66 (2002)

    MATH  Google Scholar 

  34. Vedaldi, A., Fulkerson, B.: VLFeat: an open and portable library of computer vision algorithms. http://www.vlfeat.org/ (2008)

  35. Wei, K., Tai, X.-C., Chan, T.F., Leung, S.: Primal-dual method for continuous max-flow approaches. In: Computational Vision and Medical Image Processing V: Proceedings of the 5th Eccomas Thematic Conference on Computational Vision and Medical Image Processing (VipIMAGE 2015), Tenerife, Spain, October 19–21, 2015, pp. 17. CRC Press (2015)

  36. Wu, Q., Ng, M.K., Ye, Y.: Markov-miml: a markov chain-based multi-instance multi-label learning algorithm. Knowl. Inf. Syst. 37(1), 83–104 (2013)

    Article  Google Scholar 

  37. Wu, Q., Ng, M.K., Ye, Y., Li, X., Shi, R., Li, Y.: Multi-label collective classification via markov chain based learning method. Knowl. Based Syst. 63, 1–14 (2014)

    Article  Google Scholar 

  38. Yang, Z., Hao, T., Dikmen, O., Chen, X., Oja, E.: Clustering by nonnegative matrix factorization using graph random walk. In: Advances in Neural Information Processing Systems, pp. 1079–1087 (2012)

  39. Yu, S.X., Shi, J.: Multiclass spectral clustering. In: Proceedings of the Ninth IEEE International Conference on Computer Vision, 2003, vol. 1, pp. 313–319 (2003)

  40. Yuan, J., Bae, E., Tai, X.-C.: A study on continuous max-flow and min-cut approaches. In: 2010 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2217–2224. IEEE (2010)

  41. Yuan, J., Bae, E., Tai, X.-C., Boykov, Y.: A continuous max-flow approach to potts model. In: Computer Vision–ECCV 2010, pp. 379–392. Springer (2010)

  42. Yuan, J., Bae, E., Tai, X.-C., Boykov, Y.: A spatially continuous max-flow and min-cut framework for binary labeling problems. Numer. Math. 126(3), 559–587 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  43. Zelnik-Manor, L., Perona, P.: Self-tuning spectral clustering. In: Advances in Neural Information Processing Systems, pp. 1601–1608 (2004)

  44. Zhu, M., Chan, T.: An efficient primal-dual hybrid gradient algorithm for total variation image restoration. UCLA CAM Report, pp. 08–34 (2008)

Download references

Acknowledgements

We would like to thank Prof. Stan Osher for insightful discussions, especially for pointing us to the seminal paper of Ronald Coifman on diffusion distance. XC Tai acknowledge the support from Norwegian Research Council through ISP-Matematikk (Project no. 239033/F20).

Author information

Authors and Affiliations

  1. Center for Mathematical Sciences, Huazhong Univeristy of Science and Technology, 10037 Luoyu Road, Wuhan, 430074, Hubei, China

    Ke Yin

  2. Department of Mathematics, University of Bergen, Postboks 7800, 5020, Mons, Norway

    Xue-Cheng Tai

Authors
  1. Ke Yin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xue-Cheng Tai
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ke Yin.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yin, K., Tai, XC. An Effective Region Force for Some Variational Models for Learning and Clustering. J Sci Comput 74, 175–196 (2018). https://doi.org/10.1007/s10915-017-0429-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10915-017-0429-4

Keywords

Search

Navigation