Skip to main content
SpringerLink
Log in

DiscoStyle: Multi-level Logistic Ranking for Personalized Image Style Preference Inference

  • Research Article
  • Published:
International Journal of Automation and Computing Aims and scope Submit manuscript

Abstract

Learning based on facial features for detection and recognition of people’s identities, emotions and image aesthetics has been widely explored in computer vision and biometrics. However, automatic discovery of users’ preferences to certain of faces (i.e., style), to the best of our knowledge, has never been studied, due to the subjective, implicative, and uncertain characteristic of psychological preference. Therefore, in this paper, we contribute to an answer to whether users’ psychological preference can be modeled and computed after observing several faces. To this end, we first propose an efficient approach for discovering the personality preference related facial features from only a very few anchors selected by each user, and make accurate predictions and recommendations for users. Specifically, we propose to discover the style of faces (DiscoStyle) for human’s psychological preference inference towards personalized face recommendation system/application. There are four merits of our DiscoStyle: 1) Transfer learning is exploited from identity related facial feature representation to personality preference related facial feature. 2) Appearance and geometric landmark feature are exploited for preference related feature augmentation. 3) A multi-level logistic ranking model with on-line negative sample selection is proposed for online modeling and score prediction, which reflects the users’ preference degree to gallery faces. 4) A large dataset with different facial styles for human’s psychological preference inference is developed for the first time. Experiments show that our proposed DiscoStyle can well achieve users’ preference reasoning and recommendation of preferred facial styles in different genders and races.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. F. Schroff, D. Kalenichenko, J. Philbin. FaceNet: A unified embedding for face recognition and clustering. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Boston, USA, pp. 815–823, 2015. DOI: https://doi.org/10.1109/CVPR.2015.7298682.

    Google Scholar 

  2. Y. Taigman, M. Yang, M. A. Ranzato, L. Wolf. Deepface: Closing the gap to human-level performance in face verification. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Columbus, USA, pp. 1701–1708, 2014. DOI: https://doi.org/10.1109/CVPR.2014.220.

    Google Scholar 

  3. Y. Sun, X. G. Wang, X. O. Tang. Deep learning face representation from predicting 10, 000 classes. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Columbus, USA, pp. 1891–1898, 2014. DOI: https://doi.org/10.1109/CVPR.2014.244.

    Google Scholar 

  4. J. Wright, A. Y. Yang, A. Ganesh, S. S. Sastry, Y. Ma. Robust face recognition via sparse representation. IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 31, no. 2, pp. 210–227, 2009. DOI: https://doi.org/10.1109/TPAMI.2008.79.

    Google Scholar 

  5. J. K. Chen, Z. H. Chen, Z. R. Chi, H. Fu. Facial expression recognition in video with multiple feature fusion. IEEE Transactions on Affective Computing, vol. 9, no. 1, pp. 38–50, 2018. DOI: https://doi.org/10.1109/TAFFC.2016.2593719.

    Google Scholar 

  6. L. Zhang, D. Zhang, M. M. Sun, F. M. Chen. Facial beauty analysis based on geometric feature: Toward attractiveness assessment application. Expert Systems with Applications, vol. 82, pp. 252–265, 2017. DOI: https://doi.org/10.1016/j.eswa.2017.04.021.

    Google Scholar 

  7. Y. Fu, G. D. Guo, T. S. Huang. Age synthesis and estimation via faces: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 32, no. 11, pp. 1955–1976, 2010. DOI: https://doi.org/10.1109/TPAMI.2010.36.

    Google Scholar 

  8. E. Eidinger, R. Enbar, T. Hassner. Age and gender estimation of unfiltered faces. IEEE Transactions on Information Forensics and Security, vol. 9, no. 12, pp. 2170–2179, 2014. DOI: https://doi.org/10.1109/TIFS.2014.2359646.

    Google Scholar 

  9. Z. Lian, Y. Li, J. H. Tao, J. Huang, M. Y. Niu. Expression analysis based on face regions in real-world conditions. International Journal of Automation and Computing, vol. 17, no. 1, pp. 96–107, 2020. DOI: https://doi.org/10.1007/s11633-019-1176-9.

    Google Scholar 

  10. H. S. Du, Q. P. Hu, D. F. Qiao, I. Pitas. Robust face recognition via low-rank sparse representation-based classification. International Journal of Automation and Computing, vol. 12, no. 6, pp. 579–587, 2015. DOI: https://doi.org/10.1007/s11633-015-0901-2.

    Google Scholar 

  11. H. Wu, Z. W. Chen, G. H. Tian, Q. Ma, M. L. Jiao. Item ownership relationship semantic learning strategy for personalized service robot. International Journal of Automation and Computing, vol. 17, no. 3, pp. 390–402, 2020. DOI: https://doi.org/10.1007/s11633-019-1206-7.

    Google Scholar 

  12. D. Zhang, Q. J. Zhao, F. M. Chen. Quantitative analysis of human facial beauty using geometric features. Pattern Recognition, vol. 44, no. 4, pp. 940–950, 2011. DOI: https://doi.org/10.1016/j.patcog.2010.10.013.

    Google Scholar 

  13. F. M. Chen, X. H. Xiao, D. Zhang. Data-driven facial beauty analysis: Prediction, retrieval and manipulation. IEEE Transactions on Affective Computing, vol. 9, no. 2, pp. 205–216, 2018. DOI: https://doi.org/10.1109/TAFFC.2016.2599534.

    Google Scholar 

  14. A. Krizhevsky, I. Sutskever, G. E. Hinton. ImageNet classification with deep convolutional neural networks. In Proceedings of the 25th International Conference on Neural Information Processing Systems, Lake Tahoe, USA, PP.1097–1105, 2012.

  15. K. Simonyan, A. Zisserman. Very deep convolutional networks for large-scale image recognition. In Proceedings of the 3rd International Conference on Learning Representations, San Diego, USA, 2015. https://arxiv.org/abs/1409.1556.

  16. W. Y. Liu, Y. D. Wen, Z. D. Yu, M. Yang. Large-margin softmax loss for convolutional neural networks. In Proceedings of the 33rd International Conference on Machine Learning, New York, USA, 2016.

  17. S. J. Pan, Q. Yang. A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering, vol. 22, no. 10, pp. 1345–1359, 2010. DOI: https://doi.org/10.1109/TKDE.2009.191.

    Google Scholar 

  18. K. Saenko, B. Kulis, M. Fritz, T. Darrell. Adapting visual category models to new domains. In Proceedings of the 11th European Conference on Computer Vision, Springer, Heraklion, Greece, 2010. DOI: https://doi.org/10.1007/978-3-642-15561-1_16.

    Google Scholar 

  19. M. S. Long, H. Zhu, J. M. Wang, M. I. Jordan. Unsupervised domain adaptation with residual transfer networks. In Proceedings of the 30th Conference on Neural Information Processing Systems, Barcelona, Spain, pp. 136–144, 2016.

  20. L. Zhang, W. M. Zuo, D. Zhang. LSDT: Latent sparse domain transfer learning for visual adaptation. IEEE Transactions on Image Processing, vol. 25, no. 3, pp. 1177–1191, 2016. DOI: https://doi.org/10.1109/TIP.2016.2516952.

    MathSciNet  MATH  Google Scholar 

  21. L. Zhang, S. S. Wang, G. B. Huang, W. M. Zuo, J. Yang, D. Zhang. Manifold criterion guided transfer learning via intermediate domain generation. IEEE Transactions on Neural Networks and Learning Systems, vol. 30, no. 12, pp. 3759–3773, 2019. DOI: https://doi.org/10.1109/TNNLS.2019.2899037.

    MathSciNet  Google Scholar 

  22. N. Tajbakhsh, J. Y. Shin, S. R. Gurudu, R. T. Hurst, C. B. Kendall, M. B. Gotway, J. M. Liang. Convolutional neural networks for medical image analysis: Full training or fine unning? IEEE Transactions on Medical Imaging, vol. 35, no. 5, pp. 1299–1312, 2016. DOI: https://doi.org/10.1109/TMI.2016.2535302.

    Google Scholar 

  23. H. C. Shin, H. R. Roth, M. C. Gao, L. Lu, Z. Y. Xu, I. Nogues, J. H. Yao, D. Mollura, R. M. Summers. Deep convolutional neural networks for computer-aided detection: CNN architectures, dataset characteristics and transfer learning. IEEE Transactions on Medical Imaging, vol. 35, no. 5, pp. 1285–1298, 2016. DOI: https://doi.org/10.1109/TMI.2016.2528162.

    Google Scholar 

  24. D. Marmanis, M. Datcu, T. Esch, U. Stilla. Deep learning earth observation classification using imagenet pretrained networks. IEEE Geoscience and Remote Sensing Letters, vol. 13, no. 1, pp. 105–109, 2016. DOI: https://doi.org/10.1109/LGRS.2015.2499239.

    Google Scholar 

  25. X. W. Yao, J. W. Han, G. Cheng, X. M. Qian, L. Guo. Semantic annotation of high-resolution satellite images via weakly supervised learning. IEEE Transactions on Geoscience and Remote Sensing, vol. 54, no. 6, pp. 3660–3671, 2016. DOI: https://doi.org/10.1109/TGRS.2016.2523563.

    Google Scholar 

  26. M. Xie, N. Jean, M. Burke, D. Lobell, S. Ermon. Transfer learning from deep features for remote sensing and poverty mapping. In Proceedings of the 30th AAAI Conference on Artificial Intelligence, AAAI, Phoenix, USA, 2015.

    Google Scholar 

  27. N. Jean, M. Burke, M. Xie, W. M. Davis, D. B. Lobell, S. Ermon. Combining satellite imagery and machine learning to predict poverty. Science, vol. 353, no. 6301, pp. 790–794, 2016. DOI: https://doi.org/10.1126/science.aaf7894.

    Google Scholar 

  28. Q. Y. Duan, L. Zhang, W. M. Zuo. From face recognition to kinship verification: An adaptation approach. In Proceedings of IEEE International Conference on Computer Vision Workshops, IEEE, Venice, Italy, pp. 1590–1598, 2017. DOI: https://doi.org/10.1109/ICCVW.2017.187.

    Google Scholar 

  29. L. Zhang, Q. Y. Duan, D. Zhang, W. Jia, X. Z. Wang. Advkin: Adversarial convolutional network for kinship verification. IEEE Transactions on Cybernetics, published online, 2020. DOI: https://doi.org/10.1109/TCYB.2019.2959403.

  30. C. Q. Hong, J. Yu, J. Zhang, X. N. Jin, K. H. Lee. Multimodal face-pose estimation with multitask manifold deep learning. IEEE Transactions on Industrial Informatics, vol. 15, no. 7, pp. 3952–3961, 2019. DOI: https://doi.org/10.1109/TII.2018.2884211.

    Google Scholar 

  31. Q. C. Zhu, Z. H. Chen, Y. C. Soh. A novel semisupervised deep learning method for human activity recognition. IEEE Transactions on Industrial Informatics, vol. 15, no. 7, pp. 3821–3830, 2019. DOI: https://doi.org/10.1109/TII.2018.2889315.

    Google Scholar 

  32. Y. D. Yang, W. Li, T. A. Gulliver, S. F. Li. Bayesian deep learning-based probabilistic load forecasting in smart grids. IEEE Transactions on Industrial Informatics, vol. 16, no. 7, pp. 4703–4713, 2020. DOI: https://doi.org/10.1109/TII.2019.2942353.

    Google Scholar 

  33. L. Zhang, D. Zhang. Efficient solutions for discreteness, drift, and disturbance (3D) in electronic olfaction. IEEE Transactions on Systems, Man, and Cybernetics: Systems, vol. 48, no. 2, pp. 242–254, 2018. DOI: https://doi.org/10.1109/TSMC.2016.2597800.

    Google Scholar 

  34. L. Zhang, P. L. Deng. Abnormal odor detection in electronic nose via self-expression inspired extreme learning machine. IEEE Transactions on Systems, Man, and Cybernetics: Systems, vol. 49, no. 10, pp. 1922–1932, 2019. DOI: https://doi.org/10.1109/TSMC.2017.2691909.

    Google Scholar 

  35. T. Serre, G. Kreiman, M. Kouh, C. Cadieu, U. Knoblich, T. Poggio. A quantitative theory of immediate visual recognition. Progress in Brain Research, vol. 165, pp. 33–56, 2007. DOI: https://doi.org/10.1016/S0079-6123(06)65004-8.

    Google Scholar 

  36. D. Cheng, Y. H. Gong, S. P. Zhou, J. J. Wang, N. N. Zheng. Person re-identification by multi-channel parts-based CNN with improved triplet loss function. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Las Vegas, USA, pp. 1335–1344, 2016. DOI: https://doi.org/10.1109/CVPR.2016.149.

    Google Scholar 

  37. E. Insafutdinov, L. Pishchulin, B. Andres, M. Andriluka, B. Schiele. DepprrCut: A depper, stronger, and faster multi-person pose estimation model. In Proceedings of the 14th European Conference on Computer Vision, Springer, Amsterdam, The Netherlands, 2016. DOI: https://doi.org/10.1007/978-3-319-46466-43.

    Google Scholar 

  38. Y. Li, H. Z. Qi, J. F. Dai, X. Y. Ji, Y. C. Wei. Fully convolutional instance-aware semantic segmentation. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Honolulu, USA, pp. 4438–4446, 2017. DOI: https://doi.org/10.1109/CVPR.2017.472.

    Google Scholar 

  39. C. Dong, C. C. Loy, K. M. He, X. O. Tang. Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 38, no. 2, pp. 295–307, 2016. DOI: https://doi.org/10.1109/TPAMI.2015.2439281

    Google Scholar 

  40. K. M. He, X. Y. Zhang, S. Q. Ren, J. Sun. Deep residual learning for image recognition. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Las Vegas, USA, pp.770–778, 2016. DOI: https://doi.org/10.1109/CVPR.2016.90.

    Google Scholar 

  41. G. Huang, Z. Liu, L. van der Maaten, K. Q. Weinberger. Densely connected convolutional networks, In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Honolulu, USA, pp.2261–2269, 2017. DOI: https://doi.org/10.1109/CVPR.2017.243.

    Google Scholar 

  42. S. Q. Ren, K. M. He, R Girshick, J. Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In Proceedings of Advances in Neural Information Processing Systems 28, Montreal, Canada, 2015.

  43. W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C. Y. Fu, A. C. Berg. SSD: Single shot multibox detector. In Proceedings of the 14th European Conference on Computer Vision, Springer, Amsterdam, The Netherland, pp. 21–37, 2016. DOI: https://doi.org/10.1007/978-3-319-46448-0_2.

    Google Scholar 

  44. J. Redmon, A. Farhadi. Yolo9000: Better, faster, stronger. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Honolulu, USA, pp. 6517–6525, 2017. DOI: https://doi.org/10.1109/CVPR.2017.690.

    Google Scholar 

  45. Z. Cao, T. Simon, S. E. Wei, Y. Sheikh. Realtime multi-person 2D pose estimation using part affinity fields. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Honolulu, USA, pp.1302–1310, 2017. DOI: https://doi.org/10.1109/CVPR.2017.143.

    Google Scholar 

  46. X. L. Wang, T. T. Xiao, Y. N. Jiang, S. Shao, J. Sun, C. H. Shen. Repulsion loss: Detecting pedestrians in a crowd. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Salt Lake City, USA, pp.7774–7783, 2018. DOI: https://doi.org/10.1109/CVPR.2018.00811.

    Google Scholar 

  47. Z. X. Feng, J. H. Lai, X. H. Xie. Learning view-specific deep networks for person re-identification. IEEE Transactions on Image Processing, vol. 27, no. 7, pp. 3472–3483, 2018. DOI: https://doi.org/10.1109/TIP.2018.2818438.

    MathSciNet  MATH  Google Scholar 

  48. L. Q. Liu, C. Xiong, H. W. Zhang, Z. H. Niu, M. Wang, S. C. Yan. Deep aging face verification with large gaps. IEEE Transactions on Multimedia, vol. 18, no. 1, pp. 64–75, 2016. DOI: https://doi.org/10.1109/TMM.2015.2500730.

    Google Scholar 

  49. Z. F. Li, D. H. Gong, X. L. Li, D. C. Tao. Aging face recognition: A hierarchical learning model based on local patterns selection. IEEE Transactions on Image Processing, vol. 25, no. 5, pp. 2146–2154, 2016. DOI: https://doi.org/10.1109/TIP.2016.2535284.

    MathSciNet  MATH  Google Scholar 

  50. U. Park, Y. Y. Tong, A. K. Jain. Age-invariant face recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 32, no. 5, pp. 947–954, 2010. DOI: https://doi.org/10.1109/TPAMI.2010.14.

    Google Scholar 

  51. H. Dibeklioglu, A. A. Salah, T. Gevers. Like father, like son: Facial expression dynamics for kinship verification. In Proceedings of IEEE International Conference on Computer Vision, IEEE, Sydney, Australia, pp.1497–1504, 2013. DOI: https://doi.org/10.1109/ICCV.2013.189.

    Google Scholar 

  52. R. G. Fang, K. D. Tang, N. Snavely, T. Chen. Towards computational models of kinship verification. In Proceedings of IEEE International Conference on Image Processing, IEEE, Hong Kong, China, pp. 1577–1580, 2010. DOI: https://doi.org/10.1109/ICIP.2010.5652590.

    Google Scholar 

  53. H. B. Yan, J. W. Lu, X. Z. Zhou. Prototype-based discriminative feature learning for kinship verification. IEEE Transactions on Cybernetics, vol. 45, no. 11, pp. 2535–2545, 2015. DOI: https://doi.org/10.1109/TCYB.2014.2376934.

    Google Scholar 

  54. D. I. Perrett, K. A. May, S. Yoshikawa. Facial shape and judgements of female attractiveness. Nature, vol. 368, no. 6468, pp. 239–242, 1994. DOI: https://doi.org/10.1038/368239a0.

    Google Scholar 

  55. K. P. Zhang, Z. P. Zhang, Z. F. Li, Y. Qiao. Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Processing Letters, vol. 23, no. 10, pp. 1499–1503, 2016. DOI: https://doi.org/10.1109/LSP.2016.2603342.

    Google Scholar 

  56. D. Yi, Z. Lei, S. C. Liao, S. Z. Li. Learning face representation from scratch. https://arxiv.org/abs/1411.7923, 2014.

  57. V. Kazemi, J. Sullivan. One millisecond face alignment with an ensemble of regression trees. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Columbus, USA, pp. 1867–1874, 2014. DOI: https://doi.org/10.1109/CVPR.2014.241.

    Google Scholar 

  58. G. B. Huang, M. Ramesh, T. Berg, E. Learned-Miller. Labeled Faces in the Wild: A Database for Studying Face Recognition in Unconstrained Environments, Technical Report, 07–49, Department of Computer Science, University of Massachusetts, USA, 2007.

    Google Scholar 

  59. N. Dalal, B. Triggs. Histograms of oriented gradients for human detection. In Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, IEEE, San Diego, USA, pp. 886–893, 2005. DOI: https://doi.org/10.1109/CVPR.2005.177.

    Google Scholar 

  60. B. Schölkopf, R Williamson, A. Smola, J. Shawe-Taylor, J. Platt. Support vector method for novelty detection. In Proceedings of the 12th International Conference on Neural Information Processing Systems, Denver, USA, pp. 582–588, 1999.

  61. C. C. Chang, C. J. Lin. Libsvm: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology, vol. 2, no. 3, Article number 27, 2011. DOI: https://doi.org/10.1145/1961189.1961199.

  62. G. B. Huang, H. M. Zhou, X. J. Ding, R. Zhang. Extreme learning machine for regression and multiclass classification. IEEE Transactions on Systems, Man, and Cybernetics — Part B: Cybernetics, vol. 42, no. 2, pp. 513–529, 2012. DOI: https://doi.org/10.1109/TSMCB.2011.2168604.

    Google Scholar 

  63. L. van der Maaten, G. Hinton. Visualizing data using T-SNE. Journal of Machine Learning Research, vol. 9, pp. 2579–2605, 2008.

    MATH  Google Scholar 

Download references

Acknowledgments

This work was supported by National Natural Science Fund of China (No. 61771079), Chongqing Natural Science Fund (No. cstc2018jcyjAX0250) and Chongqing Youth Talent Program. The authors would like to thank the volunteers for their contribution in labeling the StyleFace for preferences modeling.

Author information

Authors and Affiliations

  1. School of Microelectronics and Communication Engineering, Chongqing University, Chongqing, 400044, China

    Zhen-Wei He, Lei Zhang & Fang-Yi Liu

Authors
  1. Zhen-Wei He
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lei Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Fang-Yi Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lei Zhang.

Additional information

Recommended by Associate Editor Bin Luo

Zhen-Wei He received B. Eng. degree in information engineering from Tianjin University, China in 2014. From July 2014 to June 2016, he worked in Chongqing Cable Network Inc., China. Now, he is a Ph. D degree candidate in Chongqing University, China.

His research interests include deep learning and computer vision.

Lei Zhang received the Ph. D degree in circuits and systems from the College of Communication Engineering, Chongqing University, China in 2013. He worked as a Post-Doctoral Fellow with Hong Kong Polytechnic University, China from 2013 to 2015. He is currently a professor/distinguished research fellow with Chongqing University, China. He has authored more than 90 scientific papers in top journals and top conferences. He serves as associate editors for IEEE Transactions on Instrumentation and Measurement, Neural Networks, etc. He is a senior member of IEEE.

His research interests include machine learning, pattern recognition, computer vision and intelligent systems.

Fang-Yi Liu received the B. Eng. degree in communication engineering from Guangxi University, China in 2017. Since September 2017, he is a master student in information and communication engineering in Chongqing University, China.

His research interests include person re-identification and deep learning.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

He, ZW., Zhang, L. & Liu, FY. DiscoStyle: Multi-level Logistic Ranking for Personalized Image Style Preference Inference. Int. J. Autom. Comput. 17, 637–651 (2020). https://doi.org/10.1007/s11633-020-1244-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11633-020-1244-1

Keywords

Search

Navigation