research-article

Combining Smart Speaker and Smart Meter to Infer Your Residential Power Usage by Self-supervised Cross-modal Learning

Authors:

Zhengyuan Zhang,

Huadong MaAuthors Info & Claims

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, Volume 7, Issue 3

Article No.: 144, Pages 1 - 26

https://doi.org/10.1145/3610905

Published: 27 September 2023 Publication History

Abstract

Energy disaggregation is a key enabling technology for residential power usage monitoring, which benefits various applications such as carbon emission monitoring and human activity recognition. However, existing methods are difficult to balance the accuracy and usage burden (device costs, data labeling and prior knowledge). As the high penetration of smart speakers offers a low-cost way for sound-assisted residential power usage monitoring, this work aims to combine a smart speaker and a smart meter in a house to liberate the system from a high usage burden. However, it is still challenging to extract and leverage the consistent/complementary information (two types of relationships between acoustic and power features) from acoustic and power data without data labeling or prior knowledge. To this end, we design COMFORT, a cross-modality system for self-supervised power usage monitoring, including (i) a cross-modality learning component to automatically learn the consistent and complementary information, and (ii) a cross-modality inference component to utilize the consistent and complementary information. We implement and evaluate COMFORT with a self-collected dataset from six houses in 14 days, demonstrating that COMFORT finds the most appliances (98%), improves the appliance recognition performance in F-measure by at least 41.1%, and reduces the Mean Absolute Error (MAE) of energy disaggregation by at least 30.4% over other alternative solutions.

References

[1]

2022. WattTime. https://www.watttime.org/ Accessed February 6, 2023.

[2]

Ryan Prescott Adams and David JC MacKay. 2007. Bayesian online changepoint detection. arXiv preprint arXiv:0710.3742 (2007).

[3]

Humam Alwassel, Dhruv Mahajan, Bruno Korbar, Lorenzo Torresani, Bernard Ghanem, and Du Tran. 2020. Self-supervised learning by cross-modal audio-video clustering. Advances in Neural Information Processing Systems 33 (2020), 9758--9770.

[4]

Nipun Batra, Yiling Jia, Hongning Wang, and Kamin Whitehouse. 2018. Transferring decomposed tensors for scalable energy breakdown across regions. In Proceedings of the AAAI, Vol. 32.

[5]

Gissella Bejarano, David DeFazio, and Arti Ramesh. 2019. Deep latent generative models for energy disaggregation. In Proceedings of the AAAI, Vol. 33. 850--857.

Digital Library

[6]

Corinne Belley, Sebastien Gaboury, Bruno Bouchard, and Abdenour Bouzouane. 2014. An efficient and inexpensive method for activity recognition within a smart home based on load signatures of appliances. Pervasive and Mobile Computing 12 (2014), 58--78.

[7]

Sejal Bhalla, Mayank Goel, and Rushil Khurana. 2021. IMU2Doppler: Cross-Modal Domain Adaptation for Doppler-based Activity Recognition Using IMU Data. Proceedings of the ACM IMWUT 5, 4 (2021), 1--20.

[8]

Sarnab Bhattacharya, Rebecca Adaimi, and Edison Thomaz. 2022. Leveraging sound and wrist motion to detect activities of daily living with commodity smartwatches. Proceedings of the ACM IMWUT 6, 2 (2022), 1--28.

[9]

Brian Chen, Andrew Rouditchenko, Kevin Duarte, Hilde Kuehne, Samuel Thomas, Angie Boggust, Rameswar Panda, Brian Kingsbury, Rogerio Feris, David Harwath, et al. 2021. Multimodal clustering networks for self-supervised learning from unlabeled videos. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 8012--8021.

[10]

Yanjiao Chen, Yijie Bai, Richard Mitev, Kaibo Wang, Ahmad-Reza Sadeghi, and Wenyuan Xu. 2021. FakeWake: Understanding and mitigating fake wake-up words of voice assistants. In Proceedings of the ACM SIGSAC Conference on Computer and Communications Security. 1861--1883.

Digital Library

[11]

Kaikai Deng, Dong Zhao, Qiaoyue Han, Zihan Zhang, Shuyue Wang, Anfu Zhou, and Huadong Ma. 2023. Midas: Generating mmWave Radar Data from Videos for Training Pervasive and Privacy-preserving Human Sensing Tasks. Proceedings of the ACM IMWUT 7, 1 (2023), 1--26.

Digital Library

[12]

Drohoo. 2022. Smart Meter ZHDB-002. http://www.drohoo.com/znkgcz/ Accessed April 6, 2022.

[13]

Drohoo. 2022. Smart Plugin ZHCZ-003. http://www.drohoo.com/znkgcz/ Accessed April 6, 2022.

[14]

Julien Eberle, Tri Kurniawan Wijaya, and Karl Aberer. 2015. Online unsupervised state recognition in sensor data. In IEEE PerCom. 29--36.

[15]

Frank Englert, Irina Diaconita, Andreas Reinhardt, Alaa Alhamoud, Richard Meister, Lucas Backert, and Ralf Steinmetz. 2013. Reduce the number of sensors: Sensing acoustic emissions to estimate appliance energy usage. In Proceedings of the 5th ACM Workshop on Embedded Systems For Energy-Efficient Buildings. 1--8.

Digital Library

[16]

Yixin Gao, Yuriy Mishchenko, Anish Shah, Spyros Matsoukas, and Shiv Vitaladevuni. 2020. Towards data-efficient modeling for wake word spotting. In IEEE ICASSP. 7479--7483.

[17]

Dilan Görür and Carl Edward Rasmussen. 2010. Dirichlet process gaussian mixture models: Choice of the base distribution. Journal of Computer Science and Technology 25, 4 (2010), 653--664.

Digital Library

[18]

Shawn Hershey, Sourish Chaudhuri, Daniel PW Ellis, Jort F Gemmeke, Aren Jansen, R Channing Moore, Manoj Plakal, Devin Platt, Rif A Saurous, Bryan Seybold, et al. 2017. CNN architectures for large-scale audio classification. In IEEE ICASSP. 131--135.

[19]

Peter Hevesi, Sebastian Wille, Gerald Pirkl, Norbert Wehn, and Paul Lukowicz. 2014. Monitoring household activities and user location with a cheap, unobtrusive thermal sensor array. In Proceedings of the 2014 ACM international joint conference on pervasive and ubiquitous computing. 141--145.

Digital Library

[20]

Sarra Houidi, François Auger, Houda Ben Attia Sethom, Dominique Fourer, and Laurence Miègeville. 2017. Relevant feature selection for home appliances recognition. In Electrimacs 2017.

[21]

Chen-Yu Hsu, Abbas Zeitoun, Guang-He Lee, Dina Katabi, and Tommi Jaakkola. 2019. Self-supervised learning of appliance usage. In International Conference on Learning Representations.

[22]

Di Hu, Feiping Nie, and Xuelong Li. 2019. Deep multimodal clustering for unsupervised audiovisual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 9248--9257.

[23]

Younghun Kim, Thomas Schmid, Zainul M Charbiwala, and Mani B Srivastava. 2009. ViridiScope: design and implementation of a fine grained power monitoring system for homes. In Proceedings of the ACM UBICOMP. 245--254.

Digital Library

[24]

J Zico Kolter and Tommi Jaakkola. 2012. Approximate inference in additive factorial hmms with application to energy disaggregation. In Artificial intelligence and statistics. PMLR, 1472--1482.

[25]

J Zico Kolter and Matthew J Johnson. 2011. REDD: A public data set for energy disaggregation research. In SIGKDD Workshop, San Diego, CA, Vol. 25. Citeseer, 59--62.

[26]

Quan Kong and Takuya Maekawa. 2014. Identifying outlets at which electrical appliances are used by electrical wire sensing to gain positional information about appliance use. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing. 349--360.

Digital Library

[27]

Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. 2016. Pruning filters for efficient convnets. arXiv preprint arXiv:1608.08710 (2016).

[28]

Xiang Li, Daqing Zhang, Qin Lv, Jie Xiong, Shengjie Li, Yue Zhang, and Hong Mei. 2017. IndoTrack: Device-free indoor human tracking with commodity Wi-Fi. Proceedings of the ACM IMWUT 1, 3 (2017), 1--22.

Digital Library

[29]

Zhenning Li, Kyle Gluesenkamp, Bo Shen, Jeffrey Munk, Helia Zandi, Praveen Cheekatamarla, and Steve Kowalski. 2022. Seamlessly Fuel Flexible Heat Pump with Optimal Model-based Control Strategies to Reduce Peak Demand, Utility Cost and CO2 Emission. Technical Report. Oak Ridge National Lab (ORNL), Oak Ridge, TN (United States).

[30]

Dawei Liang and Edison Thomaz. 2019. Audio-based activities of daily living (adl) recognition with large-scale acoustic embeddings from online videos. Proceedings of the ACM IMWUT 3, 1 (2019), 1--18.

[31]

Ji Lin, Ligeng Zhu, Wei-ming Chen, Wei-chen Wang, Chuang Gan, and Song Han. 2022. On-Device Training Under 256KB Memory. In Annual Conference on Neural Information Processing Systems.

[32]

Saeed Mian Qaisar and Futoon Alsharif. 2020. Signal piloted processing of the smart meter data for effective appliances recognition. Journal of Electrical Engineering & Technology 15, 5 (2020), 2279--2285.

[33]

Xiaomin Ouyang, Xian Shuai, Jiayu Zhou, Ivy Wang Shi, Zhiyuan Xie, Guoliang Xing, and Jianwei Huang. 2022. Cosmo: contrastive fusion learning with small data for multimodal human activity recognition. In Proceedings of the 28th Annual International Conference on Mobile Computing And Networking. 324--337.

Digital Library

[34]

Francesca Paradiso, Federica Paganelli, Antonio Luchetta, Dino Giuli, and Pino Castrogiovanni. 2013. ANN-based appliance recognition from low-frequency energy monitoring data. In 2013 IEEE 14th International Symposium on" A World of Wireless, Mobile and Multimedia Networks"(WoWMoM). IEEE, 1--6.

[35]

Shwetak N Patel, Thomas Robertson, Julie A Kientz, Matthew S Reynolds, and Gregory D Abowd. 2007. At the flick of a switch: detecting and classifying unique electrical events on the residential power line. In Proceedings of the ACM UBICOMP. 271--288.

[36]

Nilavra Pathak, Md Abdullah Al Hafiz Khan, and Nirmalya Roy. 2015. Acoustic based appliance state identifications for fine-grained energy analytics. In IEEE PerCom. 63--70.

[37]

Dennis E Phillips, Rui Tan, Mohammad-Mahdi Moazzami, Guoliang Xing, Jinzhu Chen, and David KY Yau. 2013. Supero: A sensor system for unsupervised residential power usage monitoring. In IEEE PerCom. 66--75.

[38]

Alex Rodriguez and Alessandro Laio. 2014. Clustering by fast search and find of density peaks. science 344, 6191 (2014), 1492--1496.

[39]

Nirmalya Roy, Nilavra Pathak, and Archan Misra. 2016. Fine-grained appliance usage and energy monitoring through mobile and power-line sensing. Pervasive and Mobile Computing 30 (2016), 132--150.

Digital Library

[40]

Aaqib Saeed, Flora D Salim, Tanir Ozcelebi, and Johan Lukkien. 2020. Federated self-supervised learning of multisensor representations for embedded intelligence. IEEE Internet of Things Journal 8, 2 (2020), 1030--1040.

[41]

Manaswi Saha, Shailja Thakur, Amarjeet Singh, and Yuvraj Agarwal. 2014. Energylens: combining smartphones with electricity meter for accurate activity detection and user annotation. In Proceedings of the 5th international conference on Future energy systems. 289--300.

Digital Library

[42]

Seeed. 2022. ReSpeaker 6-Mic Circular Array kit for Raspberry Pi. https://wiki.seeedstudio.com/ReSpeaker_6-Mic_Circular_Array_kit_for_Raspberry_Pi/ Accessed November 3, 2022.

[43]

Huijuan Shao, Manish Marwah, and Naren Ramakrishnan. 2013. A temporal motif mining approach to unsupervised energy disaggregation: Applications to residential and commercial buildings. In Proceedings of the AAAI, Vol. 27. 1327--1333.

[44]

Imran Sheikh and Charles Barnhart. 2022. Carbon Reduction Potential through Optimal Control of Grid Interactive Heat Pump Water Heaters across the United States. Available at SSRN 4371455 (2022).

[45]

Changho Shin, Sunghwan Joo, Jaeryun Yim, Hyoseop Lee, Taesup Moon, and Wonjong Rhee. 2019. Subtask gated networks for non-intrusive load monitoring. In Proceedings of the AAAI, Vol. 33. 1150--1157.

Digital Library

[46]

Wei Sun, Tuochao Chen, Jiayi Zheng, Zhenyu Lei, Lucy Wang, Benjamin Steeper, Peng He, Matthew Dressa, Feng Tian, and Cheng Zhang. 2020. Vibrosense: Recognizing home activities by deep learning subtle vibrations on an interior surface of a house from a single point using laser doppler vibrometry. Proceedings of the ACM IMWUT 4, 3 (2020), 1--28.

Digital Library

[47]

Z Cihan Taysi, M Amac Guvensan, and Tommaso Melodia. 2010. Tinyears: spying on house appliances with audio sensor nodes. In Proceedings of the 2nd ACM Workshop on Embedded Sensing Systems for Energy-Efficiency in Building. 31--36.

Digital Library

[48]

Yonglong Tian, Dilip Krishnan, and Phillip Isola. 2020. Contrastive multiview coding. In European conference on computer vision. Springer, 776--794.

Digital Library

[49]

Catherine Tong, Jinchen Ge, and Nicholas D Lane. 2021. Zero-shot learning for imu-based activity recognition using video embeddings. Proceedings of the ACM IMWUT 5, 4 (2021), 1--23.

Digital Library

[50]

Mostafa Uddin and Tamer Nadeem. 2012. EnergySniffer: Home energy monitoring system using smart phones. In IEEE IWCMC. 159--164.

[51]

Anastasios Vafeiadis, Thanasis Vafeiadis, Stelios Zikos, Stelios Krinidis, Konstantinos Votis, Dimitrios Giakoumis, Dimosthenis Ioannidis, Dimitrios Tzovaras, Liming Chen, and Raouf Hamzaoui. 2018. Energy-based decision engine for household human activity recognition. In IEEE PerCom Workshops. 704--709.

[52]

Mark Valovage, Akshay Shekhawat, and Maria Gini. 2018. Model-free iterative temporal appliance discovery for unsupervised electricity disaggregation. In Proceedings of the AAAI, Vol. 32.

[53]

Joe H Ward Jr. 1963. Hierarchical grouping to optimize an objective function. Journal of the American statistical association 58, 301 (1963), 236--244.

[54]

Jason Wu, Chris Harrison, Jeffrey P Bigham, and Gierad Laput. 2020. Automated Class Discovery and One-Shot Interactions for Acoustic Activity Recognition. In Proceedings of the CHI Conference on Human Factors in Computing Systems. 1--14.

Digital Library

[55]

Da Yan, Yuan Jin, Hongsan Sun, Bing Dong, Zi Ye, Zhaoxuan Li, and Yanping Yuan. 2019. Household appliance recognition through a Bayes classification model. Sustainable Cities and Society 46 (2019), 101393.

[56]

Chaoyun Zhang, Mingjun Zhong, Zongzuo Wang, Nigel Goddard, and Charles Sutton. 2018. Sequence-to-point learning with neural networks for non-intrusive load monitoring. In Proceedings of the AAAI, Vol. 32.

[57]

Ye Zhang, Longguang Wang, Huiling Chen, Aosheng Tian, Shilin Zhou, and Yulan Guo. 2022. IF-ConvTransformer: A framework for human activity recognition using IMU fusion and ConvTransformer. Proceedings of the ACM IMWUT 6, 2 (2022), 1--26.

Digital Library

[58]

Yu Zhibin and Chen Hong. 2019. Research on household appliances recognition method based on data screening of deep learning. IFAC-PapersOnLine 52, 24 (2019), 140--144.

Cited By

Chen YHu CKimura TLi QLiu SWu FChen G(2024)SemiCMT: Contrastive Cross-Modal Knowledge Transfer for IoT Sensing with Semi-Paired Multi-Modal SignalsProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36997798:4(1-30)Online publication date: 21-Nov-2024
https://dl.acm.org/doi/10.1145/3699779
Xia PZhou HYang TZhou WLiu ZWang XLi X(2024)ESATED: Leveraging Extra-weak Supervision with Auxiliary Task for Enhanced Non-intrusiveness in Energy DisaggregationProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36997298:4(1-32)Online publication date: 21-Nov-2024
https://dl.acm.org/doi/10.1145/3699729
Wang ZZhang XWang YPeng XXu ZShu YLiu JTan RHe YChen J(2024)Hawk: An Efficient NALM System for Accurate Low-Power Appliance RecognitionProceedings of the 22nd ACM Conference on Embedded Networked Sensor Systems10.1145/3666025.3699359(578-591)Online publication date: 4-Nov-2024
https://dl.acm.org/doi/10.1145/3666025.3699359

Index Terms

Combining Smart Speaker and Smart Meter to Infer Your Residential Power Usage by Self-supervised Cross-modal Learning
1. Human-centered computing
  1. Ubiquitous and mobile computing
    1. Ubiquitous and mobile computing design and evaluation methods
    2. Ubiquitous and mobile computing systems and tools

Recommendations

Utilizing Self-Supervised Learning for Recognizing Human Activity in Older Adults through Labeling Applications in Real-World Smart Homes
GoodIT '24: Proceedings of the 2024 International Conference on Information Technology for Social Good

Deep learning models have significantly contributed to recognizing older adults’ daily activities for telemonitoring and assistance. However, recognizing human activities in real-world smart homes over the long term presents substantial challenges. ...
Domain Adaptation for Speaker Verification Based on Self-supervised Learning with Adversarial Training
MultiMedia Modeling
Abstract
Speaker verification models trained on a single domain have difficulty keeping performance on new domain data. Adversarial training maps different domain data to the same subspace to handle this problem. However, adversarial training only uses ...
Improving Few-Shot Image Classification with Self-supervised Learning
Cloud Computing – CLOUD 2022
Abstract
Few-Shot Image Classification (FSIC) aims to learn an image classifier with only a few training samples. The key challenge of few-shot image classification is to learn this classifier with scarce labeled data. To tackle the issue, we leverage the ...

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies Volume 7, Issue 3

September 2023

1734 pages

EISSN:2474-9567

DOI:10.1145/3626192

Issue’s Table of Contents

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 September 2023

Published in IMWUT Volume 7, Issue 3

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

Innovation Research Group Project of NSFC
National Natural Science Foundation of China

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
300
Total Downloads

Downloads (Last 12 months)165
Downloads (Last 6 weeks)28

Reflects downloads up to 03 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Chen YHu CKimura TLi QLiu SWu FChen G(2024)SemiCMT: Contrastive Cross-Modal Knowledge Transfer for IoT Sensing with Semi-Paired Multi-Modal SignalsProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36997798:4(1-30)Online publication date: 21-Nov-2024
https://dl.acm.org/doi/10.1145/3699779
Xia PZhou HYang TZhou WLiu ZWang XLi X(2024)ESATED: Leveraging Extra-weak Supervision with Auxiliary Task for Enhanced Non-intrusiveness in Energy DisaggregationProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36997298:4(1-32)Online publication date: 21-Nov-2024
https://dl.acm.org/doi/10.1145/3699729
Wang ZZhang XWang YPeng XXu ZShu YLiu JTan RHe YChen J(2024)Hawk: An Efficient NALM System for Accurate Low-Power Appliance RecognitionProceedings of the 22nd ACM Conference on Embedded Networked Sensor Systems10.1145/3666025.3699359(578-591)Online publication date: 4-Nov-2024
https://dl.acm.org/doi/10.1145/3666025.3699359

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents