Skip to main content
SpringerLink
Log in

Your clicks reveal your secrets: a novel user-device linking method through network and visual data

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Cameras for visual surveillance are extensively deployed to monitor people’s locations and activities. The law enforcement can analyze the surveillance videos (V-data) to track the whereabouts of the criminal suspects. On the other hand, with the popular use of the mobile phones and a wide coverage of wireless networks, people can easily access the Internet. The law enforcement also need to analyze the network traffic (N-data) to track the device so as to monitor the criminal suspects’ online behaviors. In order to match the suspects’ online and offline behaviors, the key problem is to link the device and its user. In this paper, we present a novel method to link the target with his mobile device by analyzing the N-V data. We use a camera and a wireless access point to monitor people operating their mobile devices in public places such as bars, shopping malls, or similar gathering places. Our user-device linking method is based on the premise that when a user is playing an app, his click activities can generate particular network traffic packets in a short time. Based on this premise, our research is carried out as follows. First, we design experiments to detect the particular packets and figure out the time gap distribution between the user’s clicks and these packets. Through statistical work, we find that for 97.4% of all instances, the time gap is less than 0.5 s. Then we choose five popular social networking apps to evaluate our method. We find that the main impact factors on the experimental results are the different user’s habits and the app’s category. Finally, by simulating two real-world scenarios in which people use different apps, we verify the effectiveness of the linking method. Both in scenario 1 and 2, the accuracy rate of experimental results reaches about 94% when the participants include 5 persons and exceeds 84% in experiments including 10 persons, with the fastest linking speed achieved in 20 s.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

Notes

  1. https://www.statista.com/statistics/272014/global-social-networks-ranked-by-number-of-users/

  2. http://www.androidtcpdump.com/

References

  1. Breiman L (2001) Random forests. Mach Learn 45(1):5–32

    Article  Google Scholar 

  2. Conti M, Mancini LV, Spolaor R, Verde NV (2016) Analyzing android encrypted network traffic to identify user actions. IEEE Trans Inf Forensics Secur 11(1):114–125

    Article  Google Scholar 

  3. Cunche M, Kaafar M-A, Boreli R (2014) Linking wireless devices using information contained in wi-fi probe requests. Pervasive Mob Comput 11:56–69

    Article  Google Scholar 

  4. Dai S, Tongaonkar A, Wang X, Nucci A, Song D (2013) Networkprofiler: towards automatic fingerprinting of android apps. In: Proceedings of the IEEE international conference on computer communications (INFOCOM). IEEE, pp 809–817

  5. Felzenszwalb P, McAllester D, Ramanan D (2008) A discriminatively trained, multiscale, deformable part model. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR). IEEE, pp 1–8

  6. Fischler MA, Bolles RC (1981) Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun ACM 24(6):381–395

    Article  MathSciNet  Google Scholar 

  7. Girshick R (2015) Fast r-cnn. In: Proceedings of the IEEE international conference on computer vision (ICCV), pp 1440–1448

  8. Girshick R, Donahue J, Darrell T, Malik J (2014) Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 580–587

  9. Herrmann D, Banse C, Federrath H (2013) Behavior-based tracking: exploiting characteristic patterns in dns traffic. Computers & Security 39:17–33

    Article  Google Scholar 

  10. Huang Y, Liu X, Jin L, Zhang X (2015) Deepfinger: a cascade convolutional neuron network approach to finger key point detection in egocentric vision with mobile camera. In: Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics (SMC). IEEE, pp 2944–2949

  11. Huang Y, Liu X, Zhang X, Jin L (2016) A pointing gesture based egocentric interaction system: dataset, approach and application. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 16–23

  12. Kalal Z, Mikolajczyk K, Matas J (2012) Tracking-learning-detection. IEEE Trans Pattern Anal Mach Intell 34(7):1409–1422

    Article  Google Scholar 

  13. Kumpošt M, Matyáš V (2009) User profiling and re-identification: case of university-wide network analysis. In: Proceedings of the international conference on trust, privacy and security in digital business. Springer, pp 1–10

  14. Li X, Teng J, Zhai Q, Zhu J, Xuan D, Zheng YF, Zhao W (2013) Ev-human: human localization via visual estimation of body electronic interference. In: Proceedings of the IEEE international conference on computer communications (INFOCOM). IEEE, pp 500–504

  15. Li G, Yang F, Chen G, Zhai Q, Li X, Teng J, Zhu J, Xuan D, Chen B, Zhao W (2017) Ev-matching: bridging large visual data and electronic data for efficient surveillance. In: Proceedings of the IEEE 37th international conference on distributed computing systems (ICDCS). IEEE, pp 689–698

  16. Matte C, Achara JP, Cunche M (2015) Device-to-identity linking attack using targeted wi-fi geolocation spoofing. In: Proceedings of the 8th ACM conference on security & privacy in wireless and mobile networks. ACM, p 20

  17. Mayer JR, Mitchell JC (2012) Third-party web tracking: policy and technology. In: Proceedings of the IEEE symposium on security and privacy (S&P). IEEE, pp 413–427

  18. Mills D (2006) Computer network time synchronization-the network time protocol. CRC Press, Boca Raton. 304pp

    Book  Google Scholar 

  19. Musa A, Eriksson J (2012) Tracking unmodified smartphones using wi-fi monitors. In: Proceedings of the 10th ACM conference on embedded network sensor systems. ACM, pp 281–294

  20. Myers CS, Rabiner LR (1981) A comparative study of several dynamic time-warping algorithms for connected-word recognition. Bell Syst Tech J 60 (7):1389–1409

    Article  Google Scholar 

  21. Nebehay G, Pflugfelder R (2014) Consensus-based tracking and matching of keypoints for object tracking. In: Proceedings of the IEEE Winter conference on applications of computer vision (WACV)

  22. Nebehay G, Pflugfelder R (2015) Clustering of static-adaptive correspondences for deformable object tracking. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 2784–2791

  23. Nguyen LT, Kim YS, Tague P, Zhang J (2014) Identitylink: user-device linking through visual and rf-signal cues. In: Proceedings of the 2014 ACM international joint conference on pervasive and ubiquitous computing. ACM, pp 529–539

  24. Olejnik L, Minh-Dung T, Castelluccia C (2013) Selling off privacy at auction. <hal-00915249>

  25. Peng P, Shou L, Chen K, Chen G, Wu S (2013) The knowing camera: recognizing places-of-interest in smartphone photos. In: Proceedings of the 36th international ACM SIGIR conference on research and development in information retrieval. ACM, pp 969–972

  26. Peng P, Shou L, Chen K, Chen G, Wu S (2014) The knowing camera 2: recognizing and annotating places-of-interest in smartphone photos. In: Proceedings of the 37th international ACM SIGIR conference on research and development in information retrieval. ACM, pp 707–716

  27. Ren S, He K, Girshick R, Sun J (2015) Faster r-cnn: towards real-time object detection with region proposal networks. In: Advances in neural information processing systems, pp 91–99

  28. Shukla D, Kumar R, Serwadda A, Phoha VV (2014) Beware, your hands reveal your secrets!. In: Proceedings of the 2014 ACM SIGSAC conference on computer and communications security. ACM, pp 904–917

  29. Stöber T, Frank M, Schmitt J, Martinovic I (2013) Who do you sync you are?: smartphone fingerprinting via application behaviour. In: Proceedings of the sixth ACM conference on security and privacy in wireless and mobile networks. ACM, pp 7–12

  30. Teng J, Zhu J, Zhang B, Xuan D, Zheng YF (2012) Ev: efficient visual surveillance with electronic footprints. In: Proceedings of the IEEE international conference on computer communications (INFOCOM). IEEE, pp 109–117

  31. Teng J, Zhang B, Zhu J, Li X, Xuan D, Zheng YF (2014) Ev-loc: integrating electronic and visual signals for accurate localization. IEEE/ACM Trans Netw (TON) 22(4):1285–1296

    Article  Google Scholar 

  32. Wang Y, Kankanhalli MS (2015) Tweeting cameras for event detection. In: Proceedings of the 24th international conference on world wide web. ACM, pp 1231–1241

  33. Wang Q, Yahyavi A, Kemme B, He W (2015) I know what you did on your smartphone: inferring app usage over encrypted data traffic. In: Proceedings of the IEEE conference on communications and network security (CNS). IEEE, pp 433–441

  34. Yoon S-H, Park J-S, Kim M-S (2015) Behavior signature for fine-grained traffic identification. Appl Math 9(2L):523–534

    Google Scholar 

  35. Yue Q, Ling Z, Fu X, Liu B, Ren K, Zhao W (2014) Blind recognition of touched keys on mobile devices. In: Proceedings of the 2014 ACM SIGSAC conference on computer and communications security. ACM, pp 1403–1414

Download references

Acknowledgements

We thank Yichao Huang at SCUT- HCII Laboratory for discussions and his valuable feedback.

This work was supported in part by National Key R&D Program of China 2017YFB1003000, and 2018YFB0803400, National Natural Science Foundation of China under grants 61502100, 61532013, 61572130, and 61632008, by Jiangsu Provincial Natural Science Foundation of China under grants BK20150637, Jiangsu Provincial Scientific and Technological Achievements Transfer Fund BA2016052, by Jiangsu Provincial Key Laboratory of Network and Information Security under grants BM2003201, by Key Laboratory of Computer Network and Information Integration of Ministry of Education of China under grants 93K-9 and by Collaborative Innovation Center of Novel Software Technology and Industrialization. Any opinions, findings, conclusions, and recommendations in this paper are those of the authors and do not necessarily reflect the views of the funding agencies.

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, Southeast University, Nanjing, People’s Republic of China

    Naixuan Guo, Junzhou Luo, Zhen Ling, Ming Yang & Wenjia Wu

  2. Department of Computer Science, University of Central Florida, Orlando, FL, USA

    Xinwen Fu

Authors
  1. Naixuan Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Junzhou Luo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zhen Ling
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ming Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Wenjia Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Xinwen Fu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Naixuan Guo.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Guo, N., Luo, J., Ling, Z. et al. Your clicks reveal your secrets: a novel user-device linking method through network and visual data. Multimed Tools Appl 78, 8337–8362 (2019). https://doi.org/10.1007/s11042-018-6815-6

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-018-6815-6

Keywords

Search

Navigation