research-article

BiliScreen: Smartphone-Based Scleral Jaundice Monitoring for Liver and Pancreatic Disorders

Authors:

Alex Mariakakis,

Megan A. Banks,

Lauren Phillipi,

Shwetak N. PatelAuthors Info & Claims

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, Volume 1, Issue 2

Article No.: 20, Pages 1 - 26

https://doi.org/10.1145/3090085

Published: 30 June 2017 Publication History

Abstract

Pancreatic cancer has one of the worst survival rates amongst all forms of cancer because its symptoms manifest later into the progression of the disease. One of those symptoms is jaundice, the yellow discoloration of the skin and sclera due to the buildup of bilirubin in the blood. Jaundice is only recognizable to the naked eye in severe stages, but a ubiquitous test using computer vision and machine learning can detect milder forms of jaundice. We propose BiliScreen, a smartphone app that captures pictures of the eye and produces an estimate of a person's bilirubin level, even at levels normally undetectable by the human eye. We test two low-cost accessories that reduce the effects of external lighting: (1) a 3D-printed box that controls the eyes' exposure to light and (2) paper glasses with colored squares for calibration. In a 70-person clinical study, we found that BiliScreen with the box achieves a Pearson correlation coefficient of 0.89 and a mean error of -0.09 ± 2.76 mg/dl in predicting a person's bilirubin level. As a screening tool, BiliScreen identifies cases of concern with a sensitivity of 89.7% and a specificity of 96.8% with the box accessory.

Supplementary Material

mariakakis (mariakakis.zip)

Supplemental movie, appendix, image and software files for, BiliScreen: Smartphone-Based Scleral Jaundice Monitoring for Liver and Pancreatic Disorders

Download
61.20 KB

References

[1]

Alireza Abdolvahabi, Brandon W. Taylor, Rebecca L. Holden, Elizabeth V Shaw, Alex Kentsis, Carlos Rodriguez-Galindo, Shizuo Mukai, and Bryan F. Shaw. Colorimetric and longitudinal analysis of leukocoria in recreational photographs of children with retinoblastoma. PloS one, 8(10):e76677, oct 2013.

[2]

American Cancer Society. Cancer Facts 8 Figures 2016. Technical report, American Cancer Society, Atlanta, GA, 2016.

[3]

Andrew Bastawrous, Hillary K. Rono, Iain A. T. Livingstone, Helen A. Weiss, Stewart Jordan, Hannah Kuper, and Matthew J. Burton. Development and Validation of a Smartphone-Based Visual Acuity Test (Peek Acuity) for Clinical Practice and Community-Based Fieldwork. JAMA Ophthalmology, 133(8):930, aug 2015.

[4]

Vinod K Bhutani, Lois Johnson, and Emidio M Sivieri. Predictive ability of a predischarge hour-specific serum bilirubin for subsequent significant hyperbilirubinemia in healthy term and near-term newborns. Pediatrics, 103(1):6--14, 1999.

[5]

Giles Bond-Smith, Neal Banga, Toby M Hammond, and Charles J Imber. Pancreatic adenocarcinoma. BMJ, 344, 2012.

[6]

Yuri Y Boykov and Marie-Pierre Jolly. Interactive graph cuts for optimal boundary 8 region segmentation of objects in ND images. In Proc. ICCV ’01, volume 1, pages 105--112. IEEE, 2001.

[7]

Simona Crihalmeanu and Arun Ross. Multispectral scleral patterns for ocular biometric recognition. Pattern Recognition Letters, 33(14):1860--1869, 2012.

Digital Library

[8]

Abhijit Das, Umapada Pal, Miguel Angel Ferrer Ballester, and Michael Blumenstein. A new efficient and adaptive sclera recognition system. In 2014 IEEE Symposium on Computational Intelligence in Biometrics and Identity Management (CIBIM), pages 1--8. IEEE, dec 2014.

[9]

Abhijit Das, Umapada Pal, Miguel Angel Ferrer Ballester, and Michael Blumenstein. Sclera recognition using dense-SIFT. In International Conference on Intelligent Systems Design and Applications, ISDA, pages 74--79. IEEE, dec 2014.

[10]

Lilian de Greef, Mayan Goel, Min Joon Seo, Eric C Larson, James W Stout, James A Taylor, and Shwetak N Patel. Bilicam: using mobile phones to monitor newborn jaundice. In Proc. UbiComp ’14, pages 331--342, 2014.

Digital Library

[11]

Maria Syl D De La Cruz, Alisa P Young, and Mack T Ruffin. Diagnosis and management of pancreatic cancer. American family physician, 89(8):626--32, apr 2014.

[12]

Mario E Giardini, Iain A T Livingstone, Stewart Jordan, Nigel M Bolster, Tunde Peto, Matthew Burton, and Andrew Bastawrous. A smartphone based ophthalmoscope. Proc. EMBC ’14, 2014:2177--2180, 2014.

[13]

Dorothy M Greig, Bruce T Porteous, and Allan H Seheult. Exact maximum a posteriori estimation for binary images. Series of the Royal Statistical Society, 51(2):271--279, 1989.

[14]

G. Heath. The episclera, sclera and conjunctiva. An overview of relevant ocular anatomy. Differential Diagnosis of Ocular Disease, 9(2):36--42, 2006.

[15]

Gudrun J Klinker, Steven A Shafer, and Takeo Kanade. A physical approach to color image understanding. International Journal of Computer Vision, 4(1):7--38, 1990.

Digital Library

[16]

LF Kozachenko and NN Leonenko. Sample estimate of the entropy of a random vector. Problemy Peredachi Informatsii, 23(2):9--16, 1987.

[17]

Everett Lawson, Jason Boggess, Siddharth Khullar, Alex Olwal, Gordon Wetzstein, and Ramesh Raskar. Computational retinal imaging via binocular coupling and indirect illumination. In Proc. SIGGRAPH ’12, page 51, 2012.

Digital Library

[18]

Somsak Leartveravat. Transcutaneous bilirubin measurement in full term neonate by digital camera. Medical Journal of Srisaket Surin Buriram Hospitals, 24(1):105--118, 2009.

[19]

Terence S Leung, Karan Kapur, Ashley Guilliam, Jade Okell, Bee Lim, Lindsay W MacDonald, and Judith Meek. Screening neonatal jaundice based on the sclera color of the eye using digital photography. Biomedical optics express, 6(11):4529--4538, 2015.

[20]

Rainer Lienhart and Jochen Maydt. An extended set of Haar-like features for rapid object detection. In Proceedings. International Conference on Image Processing, volume 1, pages 0--3, 2002.

[21]

Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, volume 07-12-June, pages 3431--3440, mar 2015.

[22]

Dan L Longo, Anthony S Fauci, Dennis L Kasper, Stephen L Hauser, J Larry Jameson, and Joseph Loscalzo. Harrison’s Principles of Internal Medicine. 18th edition, 2006.

[23]

Marco Marcon, Eliana Frigerio, and Stefano Tubaro. Sclera segmentation for gaze estimation and iris localization in unconstrained images. In CompIMAGE, pages 25--29, 2012.

[24]

Nobuyuki Otsu. A threshold selection method from gray-level histograms. IEEE Transactions on Systems, Man, and Cybernetics, 9(1):62--66, jan 1979.

[25]

Christopher G Owen, Richard SB Newsom, Alicja R Rudnicka, Sarah A Barman, and E Goeffrey Woodward. Diabetes and the tortuosity of vessels of the bulbar conjunctiva. Ophthalmology, 115(6):e27--e32, 2008.

[26]

Christopher G Owen, Richard SB Newsom, Alicja R Rudnicka, and Tim J Ellis. Vascular response of the bulbar conjunctiva to diabetes and elevated blood pressure. Ophthalmology, 112(10):1801--1808, 2005.

[27]

Vitor F Pamplona, Ankit Mohan, Manuel M Oliveira, and Ramesh Raskar. NETRA: interactive display for estimating refractive errors and focal range. ACM transactions on graphics (TOG), 29(4):77, 2010.

Digital Library

[28]

Vitor F Pamplona, Erick B Passos, Jan Zizka, Manuel M Oliveira, Everett Lawson, Esteban Clua, and Ramesh Raskar. Catra: cataract probe with a lightfield display and a snap-on eyepiece for mobile phones. In Proc. SIGGRAPH ’11, pages 7--11, 2011.

[29]

Danny Pascale. RGB coordinates of the Macbeth ColorChecker. Technical report, 2006.

[30]

Carsten Rother, Vladimir Kolmogorov, and Andrew Blake. Grabcut: Interactive foreground extraction using iterated graph cuts. ACM Transactions on Graphics (TOG) ’04, 23(3):309--314, 2004.

Digital Library

[31]

Mario A Ruiz, Sammy Saab, and Leland S Rickman. The clinical detection of scleral icterus: observations of multiple examiners. Military medicine, 162(8):560--563, 1997.

[32]

Gaurav Sharma, Wencheng Wu, and Edul N. Dalal. The CIEDE2000 color-difference formula: Implementation notes, supplementary test data, and mathematical observations. Color Research and Application, 30(1):21--30, feb 2005.

[33]

DO Sillence, Alison Senn, and DM Danks. Genetic heterogeneity in osteogenesis imperfecta. Journal of medical genetics, 16(2):101--116, 1979.

[34]

Audrey Vincent, Joseph Herman, Rich Schulick, Ralph H Hruban, and Michael Goggins. Pancreatic cancer. The Lancet, 378(9791):607--620, aug 2011.

[35]

Paul Viola and Michael J Jones. Rapid object detection using a boosted cascade of simple features. In Computer Vision and Pattern Recognition (CVPR), volume 1, pages 511--518, 2001.

[36]

Russell Wiesner, Erick Edwards, Richard Freeman, Ann Harper, Ray Kim, Patrick Kamath, Walter Kremers, John Lake, Todd Howard, Robert M. Merion, Robert A. Wolfe, Ruud Krom, and United Network for Organ Sharing Liver Disease Severity Score Committee. Model for end-stage liver disease (MELD) and allocation of donor livers. Gastroenterology, 124(1):91--96, jan 2003.

[37]

Stephen Wolf. Color correction matrix for digital still and video imaging systems. pages 1--40, 2003.

[38]

Zhi Zhou, Eliza Yingzi Du, N. Luke Thomas, and Edward J. Delp. A New Human Identification Method: Sclera Recognition. Systems, Man and Cybernetics, Part A: Systems and Humans, IEEE Transactions on, 42(3):571--583, may 2012.

Digital Library

Cited By

Ma YMeng YLi XFu YXu YLu YWeng F(2025)JaunENet: An effective non-invasive detection of multi-class jaundice deep learning method with limited labeled dataApplied Soft Computing10.1016/j.asoc.2025.112878172(112878)Online publication date: Mar-2025
https://doi.org/10.1016/j.asoc.2025.112878
Mendoza-Mora DVilchis-González AMartínez-Méndez RMuñoz-Jiménez VFrancisco-Valencia I(2024)Sclera Segmentation in Images for Bilirubin Level Measurement Using the U-Net NetworkSclera Segmentation in Images for Bilirubin Level Measurement Using the U-Net NetworkAvances en Interacción Humano-Computadora10.47756/aihc.y9i1.1649:1(179-184)Online publication date: 30-Nov-2024
https://doi.org/10.47756/aihc.y9i1.164
Adhikary RSadeh MBatra NGoel M(2024)JoulesEyeProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36314227:4(1-29)Online publication date: 12-Jan-2024
https://dl.acm.org/doi/10.1145/3631422
Show More Cited By

Index Terms

BiliScreen: Smartphone-Based Scleral Jaundice Monitoring for Liver and Pancreatic Disorders
1. Applied computing
  1. Life and medical sciences
    1. Consumer health
2. Human-centered computing
  1. Ubiquitous and mobile computing
    1. Ubiquitous and mobile devices
      1. Smartphones

Recommendations

Bilicam: using mobile phones to monitor newborn jaundice
UbiComp '14: Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing

Health sensing through smartphones has received considerable attention in recent years because of the devices' ubiquity and promise to lower the barrier for tracking medical conditions. In this paper, we focus on using smartphones to monitor newborn ...
Research and Design of Neonatal Jaundice Detector Based on Color Sensor
DMIP '21: Proceedings of the 2021 4th International Conference on Digital Medicine and Image Processing

Objective: To solve the problem of invasive detection of neonatal jaundice and prone to infection. Methods: A neonatal jaundice detector based on color sensing was proposed. The microprocessor controls the color sensor to read the RGB color components of ...
BiliCam: using mobile phones to monitor newborn jaundice
UbiComp '14 Adjunct: Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication

Health sensing through smartphones has received considerable attention in recent years because of its promise to lower costs and provide more continuous data for tracking medical conditions. In this poster, we focus on using smartphones to sense newborn ...

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies Volume 1, Issue 2

June 2017

665 pages

EISSN:2474-9567

DOI:10.1145/3120957

Issue’s Table of Contents

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 30 June 2017

Accepted: 01 April 2017

Received: 01 February 2017

Published in IMWUT Volume 1, Issue 2

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

National Science Foundation Graduate Research Fellowship Program and the Coulter Foundation

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

32
Total Citations
View Citations
1,140
Total Downloads

Downloads (Last 12 months)67
Downloads (Last 6 weeks)4

Reflects downloads up to 03 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Ma YMeng YLi XFu YXu YLu YWeng F(2025)JaunENet: An effective non-invasive detection of multi-class jaundice deep learning method with limited labeled dataApplied Soft Computing10.1016/j.asoc.2025.112878172(112878)Online publication date: Mar-2025
https://doi.org/10.1016/j.asoc.2025.112878
Mendoza-Mora DVilchis-González AMartínez-Méndez RMuñoz-Jiménez VFrancisco-Valencia I(2024)Sclera Segmentation in Images for Bilirubin Level Measurement Using the U-Net NetworkSclera Segmentation in Images for Bilirubin Level Measurement Using the U-Net NetworkAvances en Interacción Humano-Computadora10.47756/aihc.y9i1.1649:1(179-184)Online publication date: 30-Nov-2024
https://doi.org/10.47756/aihc.y9i1.164
Adhikary RSadeh MBatra NGoel M(2024)JoulesEyeProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36314227:4(1-29)Online publication date: 12-Jan-2024
https://dl.acm.org/doi/10.1145/3631422
Karim RZaman MYong W(2023)A Non-invasive Methods for Neonatal Jaundice Detection and Monitoring to Assess Bilirubin Level: A ReviewAnnals of Emerging Technologies in Computing10.33166/AETiC.2023.01.0027:1(15-29)Online publication date: 1-Jan-2023
https://doi.org/10.33166/AETiC.2023.01.002
Wemyss TNixon-Hill MOutlaw FKarsa AMeek JEnweronu-Laryea CLeung T(2023)Feasibility of smartphone colorimetry of the face as an anaemia screening tool for infants and young children in GhanaPLOS ONE10.1371/journal.pone.028173618:3(e0281736)Online publication date: 3-Mar-2023
https://doi.org/10.1371/journal.pone.0281736
Nixon-Hill MMookerjee RLeung T(2023)Assessment of bilirubin levels in patients with cirrhosis via forehead, sclera and lower eyelid smartphone imagesPLOS Digital Health10.1371/journal.pdig.00003572:10(e0000357)Online publication date: 6-Oct-2023
https://doi.org/10.1371/journal.pdig.0000357
He QLi WShi YYu YGeng WSun ZWang R(2023)SpeCamX: mobile app that turns unmodified smartphones into multispectral imagersBiomedical Optics Express10.1364/BOE.49760214:9(4929)Online publication date: 25-Aug-2023
https://doi.org/10.1364/BOE.497602
Kazankov KNixon‐Hill MKumar RAmin AAlabsawy EChikhlia ALeung TMookerjee R(2023)A novel smartphone scleral‐image based tool for assessing jaundice in decompensated cirrhosis patientsJournal of Gastroenterology and Hepatology10.1111/jgh.1609338:2(330-336)Online publication date: 4-Jan-2023
https://doi.org/10.1111/jgh.16093
Prajapati JDas DUdutalapally VMahapatra RWasnik P(2023)jScan: Smartphone-Assisted Bilirubin Quantification and Jaundice ScreeningIEEE Sensors Journal10.1109/JSEN.2023.331545223:21(26654-26661)Online publication date: 1-Nov-2023
https://doi.org/10.1109/JSEN.2023.3315452
Prasher SNelson LAnnam S(2023)Jaundice Recognition in Newborn Face, Chest and Abdomen using Spatial and Spectral Domain Graph Neural Network2023 International Conference on Advancement in Computation & Computer Technologies (InCACCT)10.1109/InCACCT57535.2023.10141723(171-175)Online publication date: 5-May-2023
https://doi.org/10.1109/InCACCT57535.2023.10141723
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents