Abstract
The design of collaborative robotics, such as driver-assisted operations, engineer a potential automation of decision-making predicated on unobtrusive data gathering of human users. This form of ‘somatic surveillance’ (Hayles, Unthought: the power of the cognitive nonconscious. University of Chicago Press, Chicago, 2017, p. 11) increasingly relies on behavioural biometrics and sensory algorithms to verify the physiology of bodies in cabin interiors. Such processes secure cyber-physical space, but also register user capabilities for control that yield data as insured risk. In this technical re-formation of human–machine interactions for control and communication ‘a dissonance of attribution’ (Hancock et al., Proc Natl Acad Sci 116(16):7684, 2019. https://doi.org/10.1073/pnas.1805770115) is created between perceptions of phenomena, materials and decision-making. This reconfigures relations not only between humans and machines, objects and subjects, but possibly disrupts attributive functions in the social system of Law. What it requires is shifting a legal accountability for action from a sovereignty of the human to a new materialist account based on a ‘cognitive assemblage’ between physiological data, computation and algorithmic sensing. This paper investigates the function of law as a guidance system to acknowledge this account of sensory and algorithmic computation as autonomous ‘sensing agents’ (Hansen, Feed-forward: on the future of twenty-first-century media. University of Chicago Press, Chicago, 2015) that may be accountable in situations of risk. This assemblage of robotic computation and sensory determination requires a clearer legal differentiation across the current static terminologies of person, property, liability and rights that maintain strict separations of object from subject. To neglect this, we argue, law will solely impute attributions of error to humans despite evidence of operation via mutual control.
Similar content being viewed by others
Notes
“Engineers and regulators were left in the dark about a fundamental overhaul to an automated system that would ultimately play a role in two crashes […] while the original version relied on data from at least two types of sensors, the final version used just one, leaving the system without a critical safeguard.” (Nicas et al. 2019).
“Chief Technical Pilot Mark Forkner described in leaked messages how MCAS cockpit software, which has since been linked to crashes in Indonesia in 2018 and in Ethiopia in March this year, was “running rampant” during a flight simulator session” (Johnson 2019).
Sources interviewed stated: “final control column inputs from the first officer were weaker than ones made earlier by the captain” (Silviana et al. 2019). This construed a fatal outcome was perceived by the pilot (ibid) and led to less resistance in tactile control.
“As part of the fix, Boeing has reworked MCAS to more closely resemble the first version. It will be less aggressive, and it will rely on two sensors.” Jack Nicas, Natalie Kitroeff, David Gelles and James Glanz. "Boeing Built Deadly Assumptions into 737 Max, Blind to a Late Design Change." The New York Times (Nicas et al. 2019).
The four core principles used in bioethics: beneficence, non-maleficence, autonomy, and justice. Of all areas of applied ethics, bioethics most closely resembles digital ethics in dealing ecologically with new forms of agents, patients, and environments (Floridi 2013).
General Motors submission for exemption from US Federal Regulations includes description of human autonomous vehicle trainers (AVTs) in testing procedures “who can take‐over driving from the ADS (automated driving system) to prevent potential crashes. GM analyzes the take‐over events to determine whether actual crashes were avoided and whether the ADS control software needs to be updated. […] These establish human driving behavior indicators and crash surrogates as a basis for understanding driver performance.” (General Motors 2018, p. 90 Appendix III).
European Civil Law Rules in Robotics. Study for the JURI Committee (2016).
Ironically the Open Letter could only be signed if the signatories ticket the box “I am not a robot” to avoid robotic hacking of the open letter on robot automation! http://www.robotics-openletter.eu/.
The Open Letter gives three main reasons against an EU e-personhood: a. “Legal status for a robot can’t derive from the Natural Person model, since the robot would then hold human rights, such as the right to dignity, the right to its integrity, the right to remuneration or the right to citizenship, thus directly confronting the Human rights. This would be in contradiction with the Charter of Fundamental Rights of the European Union and the Convention for the Protection of Human Rights and Fundamental Freedoms. b. The legal status for a robot can’t derive from the Legal Entity model, since it implies the existence of human persons behind the legal person to represent and direct it. And this is not the case for a robot. c. The legal status for a robot can’t derive from the Anglo-Saxon Trust model also called Fiducie or Treuhand in Germany. Indeed, this regime is extremely complex, requires very specialized competences and would not solve the liability issue. More importantly, it would still imply the existence of a human being as a last resort – the trustee or fiduciary – responsible for managing the robot granted with a Trust or a Fiducie.” Open Letter: http://www.robotics-openletter.eu/.
“Fatal 2016 highway crash involving a Tesla Model S and a tractor-semitrailer truck near Williston, Florida. System performance data downloaded from the Tesla revealed that the driver was operating the car using automated vehicle control systems: Traffic-Aware Cruise Control and Autosteer lane keeping systems.” https://www.ntsb.gov/news/press-releases/Pages/PR20170619.aspx.
US National Transport Safety Board found “Boeing had underestimated the effect that a malfunction of new automated software in the aircraft could have on the environment in the cockpit.” (Kitroeff 2019).
Google’s self-driving pod-cars don’t even have a steering wheel or pedals. (The National Highway Traffic Safety Administration defines this as “Level 4” autonomy—the agency’s highest level.) https://www.scientificamerican.com/article/driverless-cars-must-have-steering-wheels-brake-pedals-feds-say/?redirect=1.
The technical language we do employ has a larger social imperative. In an interview with Hayles she describes how this worked for academics and activists in their influence on the medical field during the HIV epidemic in the 1980’s specifically “to change medical practices, activists needed to learn the concepts and the vocabulary at issue.” (Amoore and Piotukh 2019, p. 150).
In a collaboration between the New South Wales Police Force in Australia and technology company Fujitsu biometric applications are being introduced into NSW police vehicles: “Fujitsu’s biometric authentication technology Palm-Secure secures sensitive information […] enhancing the safety of officers who are no longer required to take their eyes off the road to operate a complex control pad.” (McLennan 2019) https://www.governmentnews.com.au/meet-the-new-digital-cop-car-of-the-future/.
Google’s self-driving pod-cars don’t even have a steering wheel or pedals. (The National Highway Traffic Safety Administration defines this as “Level 4” autonomy—the agency’s highest level.) https://www.scientificamerican.com/article/driverless-cars-must-have-steering-wheels-brake-pedals-feds-say/?redirect=1.
Sensors are the crucial sensory organs of vehicle safety systems “by sensing the air quality in cabins, not only is the risk of drowsiness reduced, but detecting ethanol on the driver's breath, drunk driving can be prevented. https://senseair.com/applications/automotive/.
The National Highway Traffic Safety Administration (NHTSA) published the report and requested comment. As of March 2019, only one comment was received, stating: “It is true that machines can sense things that humans cannot see and can react faster than humans can. This does not mean that machines make good drivers […] At minimum NHTSA should forbid driverless cars until the majority of the nation's freight trains have been operating without any human input (even in emergencies) for several years.” https://www.regulations.gov/document?D=NHTSA-2019-0016-0002.
Austria’s AMS sensor specialist “made a 4.3 billion euro ($4.7 billion) bid for Osram earlier this month.” https://www.reuters.com/article/us-osram-licht-m-a-ams-explainer/explainer-battle-over-osram-takes-further-twist-idUSKBN1WA1NL Update 2020: “The Osram acquisition is fully on track, AMS said.” https://www.reuters.com/article/us-ams-results/shares-in-sensor-maker-ams-soar-after-upbeat-outlook-idUSKCN22B0HM.
Patent Claim: “A system, comprising a computer in a vehicle, the computer comprising a processor and a memory, wherein the computer is configured to detect a condition of an operator of the vehicle; determine that the driver condition is an impaired condition; and perform at least one autonomous operation based on the impaired condition.” https://patents.google.com/patent/US20150066284A1/en.
“Technical experts found serious programming mistakes in the machines’ software.” https://www.nytimes.com/2019/11/03/business/drunk-driving-breathalyzer.html.
References
Abo-Zahhad M, Ahmed SM, Abbas SN (2015) A novel biometric approach for human identification and verification using eye blinking signal. IEEE Signal Process Lett 22(7):876–880. https://doi.org/10.1109/LSP.2014.2374338
Akhtar Z, Micheloni C, Foresti GL (2015) Biometric liveness detection: challenges and research opportunities. IEEE Secur Priv 13(5):63–72
Allianz (2017) A brave new world: Vehicle biometrics. Motor. https://www.allianzebroker.co.uk/content/allianzebroker/en_gb/application/content/documents/news-and-insight/industry/a-brave-new-world-vehicle-biometrics/_jcr_content/documentProperties/currentDocument.res/vehicle-biometrics_ama052.pdf. Accessed 11 Mar 2019
Almada M (2019) Human intervention in automated decision-making: toward the construction of contestable systems. Paper presented at the Forthcoming, 17th International Conference on Artificial Intelligence and Law (ICAIL 2019)
Amoore L (2020) Cloud ethics: algorithms and the attributes of ourselves and others. Duke University Press, Durham
Amoore L, Hall A (2009) Taking people apart: digitised dissection and the body at the border. Environ Plann D: Soc Space 27(3):444–464. https://doi.org/10.1068/d1208
Amoore L, Piotukh V (2019) Interview with N. Katherine Hayles. Theory Cult Soc 36(2):145–155. https://doi.org/10.1177/0263276419829539
Amoore L, Raley R (2016) Securing with algorithms: knowledge, decision, sovereignty. Secur Dialog 48(1):3–10. https://doi.org/10.1177/0967010616680753
Balka E, Leigh Star S (2016) Mapping the body across diverse information systems: shadow bodies and how they make us human. In: Geoffrey ST, Bowker C, Clarke AE, Balka E (eds) Boundary objects and beyond: working with Leigh Starr. MIT Press, Cambridge, pp 417–434
Barbaro M (2019) The Daily. Podcast audio. Two Crashes, A Single Jet: The Story of Boeing 737 Max 24 minutes 48 seconds. https://www.nytimes.com/2019/03/19/podcasts/the-daily/boeing-737-max-ethiopia-crash.html?showTranscript=1. Accessed 11 June 2019
Bateson G (1972) Steps to an ecology of mind. Ballantine Books, New York
Bryson JJ, Diamantis ME, Grant TD (2017) Of, for, and by the people: the legal lacuna of synthetic persons. Artif Intell Law 25(3):273–291
Chattopadhyay A, Lam KY (2018) Autonomous vehicle: Security by design. arXiv preprint arXiv:1810.00545
Chesterman S (2019) Artificial intelligence and The Problem of Autonomy. NUS Law Working Paper Series, 016. http://law.nus.edu.sg/wps/pdfs/016_2019_Simon.pdf. Accessed 30 Nov 2019
Coeckelbergh M (2010) Robot rights? Towards a social-relational justification of moral consideration. Ethics Inf Technol 12(3):209–221. https://doi.org/10.1007/s10676-010-9235-5
Cowley S, Silver-Greenberg J (2019) These machines can put you in jail. Don’t trust them. New York Times. https://www.nytimes.com/2019/11/03/business/drunk-driving-breathalyzer.html. Accessed 30 Nov 2019
Davies M (2020) Distributed cognition, distributed being, and the foundations of law. In: de Leeuw M, van Wichelen S (eds) Personhood in the age of biolegality: brave new law. Springer International Publishing, Cham, pp 205–223
Day S, Lury C (2016) Biosensing: tracking persons. In: Nafus D (ed) Biosensing technologies in everyday life. MIT Press, California, pp 43–66
Dreier T, Spiecker Genannt Döhmann I (2012) Legal aspects of service robotics. Poiesis Prax 9(3):201–217. https://doi.org/10.1007/s10202-012-0115-4
Driver Alcohol Detection System for Safety (2018) Breath-Based Technology. In D. A. D. S. S. (Ed.). https://www.dadss.org/breath-based-technology/:DADSS.org. Accessed 8 Oct 2019
Floridi L (2013) The ethics of information. Oxford University Press, Oxford
Floridi L, Cowls J, Beltrametti M, Chatila R, Chazerand P, Dignum V, Vayena E (2018) AI people—an ethical framework for a good AI society: opportunities, risks, principles, and recommendations. Mind Mach 28(4):689–707. https://doi.org/10.1007/s11023-018-9482-5
Follett KJ, Hess TM (2002) Aging, cognitive complexity, and the fundamental attribution error. J Gerontol: Ser B 57(4):P312–P323. https://doi.org/10.1093/geronb/57.4.P312
Gabrys J (2016) Program earth: environmental sensing technology and the making of a computational planet. University of Minnesota Press, Minneapolis
Gácsi M, Szakadát S, Miklósi A (2013) Assistance dogs provide a useful behavioral model to enrich communicative skills of assistance robots. Front Psychol 4:971. https://doi.org/10.3389/fpsyg.2013.00971
General Motors (2018) Safety petition: to advance safety and zero-emission vehicles through technology that achives the safety purpose of the FMVSS (USG 4708). https://www.regulations.gov/document?D=NHTSA-2019-0016-0002. Accessed 17 Mar 2019
Georgia Institute of Technology (2018) Real-time Captcha technique improves biometric authentication. ScienceDaily. Retrieved from https://www.sciencedaily.com/releases/2018/02/180219141219.htm Accessed 20 Aug 2018
Gillespie T (2014) The relevance of algorithms. Media Technol: Essays Commun Mater Soc 167:167–194. https://doi.org/10.7551/mitpress/9780262525374.003.0009
Gless S, Silverman E, Weigend T (2016) If robots cause harm, who is to blame? Self-driving cars and criminal liability. New Crim Law Rev: Int Interdiscip J 19:412–436. https://doi.org/10.1525/nclr.2016.19.3.412
Halper M (2017) Osram opens $440 M Malaysian plant amid world’s widening clamor for LED chips. LEDs Mag. https://www.ledsmagazine.com/manufacturing-services-testing/article/16700601/osram-opens-440m-malaysian-plant-amid-worlds-widening-clamor-for-led-chips-updated. Accessed 7 June 2019
Hancock PA (2017) Mind, machine and morality: toward a philosophy of human-technology symbiosis. CRC Press, London
Hancock PA, Nourbakhsh I, Stewart J (2019) On the future of transportation in an era of automated and autonomous vehicles. Proc Natl Acad Sci 116(16):7684. https://doi.org/10.1073/pnas.1805770115
Hansen M (2013) Ubiquitous sensation: toward an atmospheric, collective, and microtemporal model of media. Art Cult Emerg Ubiquitous Comput, Throughout, pp 63–88
Hansen M (2015) Feed-forward: on the future of twenty-first-century media. University of Chicago Press, Chicago
Harper CD, Hendrickson CT, Mangones S, Samaras C (2016) Estimating potential increases in travel with autonomous vehicles for the non-driving, elderly and people with travel-restrictive medical conditions. Transp Res Part C: Emerg Technol 72:1–9
Harries-Jones P (1995) A recursive vision: ecological understanding and Gregory Bateson. University of Toronto Press, Toronto
Harrison S, Tatar D, Sengers P (2007) The three paradigms of HCI. Paper presented at the Alt. Chi. Session at the SIGCHI Conference on human factors in computing systems San Jose, California, USA
Hayles NK (2017) Unthought: the power of the cognitive nonconscious. University of Chicago Press, Chicago
Huhtamo E (2004) An archaeology of mobile media. Keynote address, 12th International Symposium of Electronic Art ISEA, Helsinki. Available at http://www.mrexhibition.net/cours/wp-content/uploads/2012/09/Underlined_Errki_Fuhtamo_An_Archaeology_of_Mobile_Media_.pdf, Accessed 12 Dec 2018
Igarashi K, Miyajima C, Itou K, Takeda K, Itakura F, Abut H (2004) Biometric identification using driving behavioral signals. Paper presented at the 2004 IEEE International Conference on Multimedia and Expo (ICME) (IEEE Cat. No. 04TH8763)
Jobin A, Ienca M, Vayena E (2019) Artificial Intelligence: the global landscape of ethics guidelines. arXiv preprint arXiv:1906.11668
Johnson E (2019) Boeing texts reveal flawed simulator, not smoking gun: ex-colleagues. Reuters. https://www.reuters.com/article/uk-ethiopia-airplane-forkner/boeing-texts-reveal-flawed-simulator-not-smoking-gun-ex-colleagues-idUSKBN1X1268. Accessed 10 Nov 2019
Johnston M (2019) Canberra expands semi-autonomous car trail: potential to help ageing population. ITNEWS. https://www.itnews.com.au/news/canberra-expands-semi-autonomous-car-trial-530333. Accessed 15 Sept 2019
Kelsen H (1991) “Causality and Imputation.” In General Theory of Norms, translated by Michael Hartney. Oxford University Press. Oxford Scholarship Online. https://www.oxfordscholarship.com/view/10.1093/acprof:oso/9780198252177.001.0001/acprof-9780198252177-chapter-7. Accessed 22 Oct 2019
Khaitan SK, McCalley JD (2014) Design techniques and applications of cyberphysical systems: a survey. IEEE Syst J 9(2):350–365
Kitroeff N (2019) Boeing underestimated cockpit Chaos on 737 Max, N.T.S.B. Says. New York Times. https://www.nytimes.com/2019/09/26/business/boeing-737-max-ntsb-mcas.html?module=inline. Accessed 6 Dec 2019
Marzano G (2018) The turing test and android science. J Robot Automat 2(1):64–68
Mattern S (2017) Mapping’s intelligent agents. Places. https://doi.org/10.22269/170926
McLennan L (2019) “Meet the Digital Cop Car of the Future.” Government News. Published electronically 9 May. https://www.governmentnews.com.au/meet-the-new-digital-cop-car-of-the-future/. Accessed 10 June 2019
Menon V, Jayaraman B, Govindaraju V (2011) The three RS of cyberphysical spaces. Computer 44(9):73–79. https://doi.org/10.1109/MC.2011.59
Müller VC, Bertolini A, Rademakers E (2017) Ethical, legal and socio-economic issues in robotics. euRobotics topics group on ‘ethical, legal and socio- economic issues’ (ELS). http://www.sophia.de/pdf/pdf_others/2017_TG-ELS_position_paper.pdf. Accessed 20 Aug 2018
Murashov V, Hearl F, Howard J (2016) Working safely with robot workers: recommendations for the new workplace. J Occup Environ Hyg 13(3):D61–D71. https://doi.org/10.1080/15459624.2015.1116700
Nicas J, Kitroeff N, Gelles D, Glanz J (2019) Boeing built deadly assumptions into 737 max, blind to a late design change. New York Times. https://www.nytimes.com/2019/06/01/business/boeing-737-max-crash.html?action=click&module=Top%20Stories&pgtype=Homepage. Accessed 15 June 2019
Okereafor K, Onime C, Osuagwu O (2017) Enhancing biometric liveness detection using trait randomization technique. Paper presented at the 2017 UKSim-AMSS 19th international conference on computer modelling & simulation (UKSim)
Parisi L (2013) Contagious architecture: computation, aesthetics, and space. MIT Press, Cambridge
Perno J, Probst C (2017) Behavioural profiling in cyber-social systems. In: Tryfonas T (ed) Human aspects of information security, privacy and trust. 5th international conference, pp 507–517
Perrow C (1999) Normal accidents: living with high-risk technologies. Princeton University Press, Princeton
Pottage A (2007) The socio-legal implications of the new biotechnologies. Annu Rev Law Soc Sci 3(1):321–344. https://doi.org/10.1146/annurev.lawsocsci.3.081806.112856
Ross A, Jain A (2003) Information fusion in biometrics. Pattern Recogn Lett 24(13):2115–2125. https://doi.org/10.1016/s0167-8655(03)00079-5
Saeed K, Nagashima T (2012) Biometrics and Kansei Engineering. Springer, New York
Santoni de Sio F, Van den Hoven J (2018) Meaningful human control over autonomous systems: a philosophical account. Front Robot AI 5:15
Shirouzu N, Tajitsu N (2019) Toyota’s not alone in the slow lane to self-driving cars. Reuters. Retrieved from https://www.reuters.com/article/us-autoshow-tokyo-toyota-technology/toyotas-not-alone-in-the-slow-lane-to-self-driving-cars-idUSKBN1X41XF. Accessed 27 Oct 2019
Silviana C, Freed J, Hepher T (2019) Exclusive: lion air pilots scoured handbook in minutes before crash. Reuters. https://www.reuters.com/article/us-indonesia-crash-exclusive/exclusive-cockpit-voice-recorder-of-doomed-lion-air-jet-depicts-pilots-frantic-search-for-fix-sources-idUSKCN1R10FB?il=0. Accessed 19 May 2019
Solaz J, De Rosario H, Gameiro P, Bande D (2014) Drowsiness and fatigue sensing system based on driver’s physiological signals. Paper presented at the Transport Research Arena (TRA) 5th Conference
Teubner G (2006) Rights of non-humans? Electronic agents and animals as new actors in politics and law. J Law Soc 33(4):497–521. https://doi.org/10.1111/j.1467-6478.2006.00368.x
Teubner G (2018) Digital personhood? The status of autonomous software agents in private law. Ancilla Iuris. https://doi.org/10.2139/ssrn.3177096
Thrun S (2010) Toward robotic cars. Commun ACM 53(4):99–106. https://doi.org/10.1145/1721654.1721679
Tsonev D, Videv S, Haas H (2014) Light fidelity (Li-Fi): towards all-optical networking. Paper presented at the Broadband Access Communication Technologies VIII
Umbrello S, Yampolskiy R (2019) Designing AI for explainability and verifiability: a value sensitive design approach to avoid artificial stupidity in autonomous vehicles. https://philpapers.org/rec/UMBDAF-2. Accessed 10 Dec 2019
Wilcox L (2016) Embodying algorithmic war: gender, race, and the posthuman in drone warfare. Secur Dialog 48(1):11–28. https://doi.org/10.1177/0967010616657947
Wilcox L (2017) Embodying algorithmic war: gender, race, and the posthuman in drone warfare. Secur Dialog 48(1):11–28. https://doi.org/10.1177/0967010616657947
Yampolskiy R (2016) Taxonomy of pathways to dangerous artificial intelligence. Paper presented at the Workshops of the Thirtieth AAAI Conference on Artificial Intelligence AI, Ethics, and Society, Technical Report WS-16-02, Phoenix, Arizona. https://www.aaai.org/ocs/index.php/WS/AAAIW16/paper/view/12566. Accessed 5 Oct 2018
Yampolskiy RV (2019) Unexplainability and incomprehensibility of artificial intelligence. arXiv preprint. Retrieved from https://arxiv.org/pdf/1907.03869v1.pdf. Accessed 14 Sept 2019
Yampolskiy R, Govindaraju V (2008) Behavioural biometrics: a survey and classification. Int J Biom 1(1):81–113. https://doi.org/10.1504/ijbm.2008.018665
Acknowledgement
This work is supported by UNSW Allens Hub for Law, Technology, and Innovation.
Author information
Authors and Affiliations
Contributions
Both authors contributed to study design. Initial research analysis and first draft written by Simon M. Taylor. Dr. Marc De Leeuw contributed substantially to following versions—particular focus on legal aspects. Both authors read/approved the final manuscript.
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Taylor, S.M., De Leeuw, M. Guidance systems: from autonomous directives to legal sensor-bilities. AI & Soc 36, 521–534 (2021). https://doi.org/10.1007/s00146-020-01012-z
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00146-020-01012-z