Skip to main content

Advertisement

Log in

Guidance systems: from autonomous directives to legal sensor-bilities

  • Open Forum
  • Published:
AI & SOCIETY Aims and scope Submit manuscript

Abstract

The design of collaborative robotics, such as driver-assisted operations, engineer a potential automation of decision-making predicated on unobtrusive data gathering of human users. This form of ‘somatic surveillance’ (Hayles, Unthought: the power of the cognitive nonconscious. University of Chicago Press, Chicago, 2017, p. 11) increasingly relies on behavioural biometrics and sensory algorithms to verify the physiology of bodies in cabin interiors. Such processes secure cyber-physical space, but also register user capabilities for control that yield data as insured risk. In this technical re-formation of human–machine interactions for control and communication ‘a dissonance of attribution’ (Hancock et al., Proc Natl Acad Sci 116(16):7684, 2019. https://doi.org/10.1073/pnas.1805770115) is created between perceptions of phenomena, materials and decision-making. This reconfigures relations not only between humans and machines, objects and subjects, but possibly disrupts attributive functions in the social system of Law. What it requires is shifting a legal accountability for action from a sovereignty of the human to a new materialist account based on a ‘cognitive assemblage’ between physiological data, computation and algorithmic sensing. This paper investigates the function of law as a guidance system to acknowledge this account of sensory and algorithmic computation as autonomous ‘sensing agents’ (Hansen, Feed-forward: on the future of twenty-first-century media. University of Chicago Press, Chicago, 2015) that may be accountable in situations of risk. This assemblage of robotic computation and sensory determination requires a clearer legal differentiation across the current static terminologies of person, property, liability and rights that maintain strict separations of object from subject. To neglect this, we argue, law will solely impute attributions of error to humans despite evidence of operation via mutual control.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Notes

  1. “Engineers and regulators were left in the dark about a fundamental overhaul to an automated system that would ultimately play a role in two crashes […] while the original version relied on data from at least two types of sensors, the final version used just one, leaving the system without a critical safeguard.” (Nicas et al. 2019).

  2. “Chief Technical Pilot Mark Forkner described in leaked messages how MCAS cockpit software, which has since been linked to crashes in Indonesia in 2018 and in Ethiopia in March this year, was “running rampant” during a flight simulator session” (Johnson 2019).

  3. Sources interviewed stated: “final control column inputs from the first officer were weaker than ones made earlier by the captain” (Silviana et al. 2019). This construed a fatal outcome was perceived by the pilot (ibid) and led to less resistance in tactile control.

  4. “As part of the fix, Boeing has reworked MCAS to more closely resemble the first version. It will be less aggressive, and it will rely on two sensors.” Jack Nicas, Natalie Kitroeff, David Gelles and James Glanz. "Boeing Built Deadly Assumptions into 737 Max, Blind to a Late Design Change." The New York Times (Nicas et al. 2019).

  5. The four core principles used in bioethics: beneficence, non-maleficence, autonomy, and justice. Of all areas of applied ethics, bioethics most closely resembles digital ethics in dealing ecologically with new forms of agents, patients, and environments (Floridi 2013).

  6. General Motors submission for exemption from US Federal Regulations includes description of human autonomous vehicle trainers (AVTs) in testing procedures “who can take‐over driving from the ADS (automated driving system) to prevent potential crashes. GM analyzes the take‐over events to determine whether actual crashes were avoided and whether the ADS control software needs to be updated. […] These establish human driving behavior indicators and crash surrogates as a basis for understanding driver performance.” (General Motors 2018, p. 90 Appendix III).

  7. European Civil Law Rules in Robotics. Study for the JURI Committee (2016).

  8. Ironically the Open Letter could only be signed if the signatories ticket the box “I am not a robot” to avoid robotic hacking of the open letter on robot automation! http://www.robotics-openletter.eu/.

  9. The Open Letter gives three main reasons against an EU e-personhood: a. “Legal status for a robot can’t derive from the Natural Person model, since the robot would then hold human rights, such as the right to dignity, the right to its integrity, the right to remuneration or the right to citizenship, thus directly confronting the Human rights. This would be in contradiction with the Charter of Fundamental Rights of the European Union and the Convention for the Protection of Human Rights and Fundamental Freedoms. b. The legal status for a robot can’t derive from the Legal Entity model, since it implies the existence of human persons behind the legal person to represent and direct it. And this is not the case for a robot. c. The legal status for a robot can’t derive from the Anglo-Saxon Trust model also called Fiducie or Treuhand in Germany. Indeed, this regime is extremely complex, requires very specialized competences and would not solve the liability issue. More importantly, it would still imply the existence of a human being as a last resort – the trustee or fiduciary – responsible for managing the robot granted with a Trust or a Fiducie.” Open Letter: http://www.robotics-openletter.eu/.

  10. “Fatal 2016 highway crash involving a Tesla Model S and a tractor-semitrailer truck near Williston, Florida. System performance data downloaded from the Tesla revealed that the driver was operating the car using automated vehicle control systems: Traffic-Aware Cruise Control and Autosteer lane keeping systems.” https://www.ntsb.gov/news/press-releases/Pages/PR20170619.aspx.

  11. US National Transport Safety Board found “Boeing had underestimated the effect that a malfunction of new automated software in the aircraft could have on the environment in the cockpit.” (Kitroeff 2019).

  12. Google’s self-driving pod-cars don’t even have a steering wheel or pedals. (The National Highway Traffic Safety Administration defines this as “Level 4” autonomy—the agency’s highest level.) https://www.scientificamerican.com/article/driverless-cars-must-have-steering-wheels-brake-pedals-feds-say/?redirect=1.

  13. The technical language we do employ has a larger social imperative. In an interview with Hayles she describes how this worked for academics and activists in their influence on the medical field during the HIV epidemic in the 1980’s specifically “to change medical practices, activists needed to learn the concepts and the vocabulary at issue.” (Amoore and Piotukh 2019, p. 150).

  14. In a collaboration between the New South Wales Police Force in Australia and technology company Fujitsu biometric applications are being introduced into NSW police vehicles: “Fujitsu’s biometric authentication technology Palm-Secure secures sensitive information […] enhancing the safety of officers who are no longer required to take their eyes off the road to operate a complex control pad.” (McLennan 2019) https://www.governmentnews.com.au/meet-the-new-digital-cop-car-of-the-future/.

  15. Google’s self-driving pod-cars don’t even have a steering wheel or pedals. (The National Highway Traffic Safety Administration defines this as “Level 4” autonomy—the agency’s highest level.) https://www.scientificamerican.com/article/driverless-cars-must-have-steering-wheels-brake-pedals-feds-say/?redirect=1.

  16. https://www.dmv.ca.gov/portal/dmv/detail/vr/autonomous/definitions.

  17. https://www.mercedes-benz.com/en/next/connectivity/cars-as-the-central-health-hub/.

  18. https://www.mercedes-benz.com/en/next/connectivity/cars-as-the-central-health-hub/.

  19. Sensors are the crucial sensory organs of vehicle safety systems “by sensing the air quality in cabins, not only is the risk of drowsiness reduced, but detecting ethanol on the driver's breath, drunk driving can be prevented. https://senseair.com/applications/automotive/.

  20. The National Highway Traffic Safety Administration (NHTSA) published the report and requested comment. As of March 2019, only one comment was received, stating: “It is true that machines can sense things that humans cannot see and can react faster than humans can. This does not mean that machines make good drivers […] At minimum NHTSA should forbid driverless cars until the majority of the nation's freight trains have been operating without any human input (even in emergencies) for several years.” https://www.regulations.gov/document?D=NHTSA-2019-0016-0002.

  21. https://blogs.cdc.gov/niosh-science-blog/2015/11/20/working-with-robots/.

  22. Austria’s AMS sensor specialist “made a 4.3 billion euro ($4.7 billion) bid for Osram earlier this month.” https://www.reuters.com/article/us-osram-licht-m-a-ams-explainer/explainer-battle-over-osram-takes-further-twist-idUSKBN1WA1NL Update 2020: “The Osram acquisition is fully on track, AMS said.” https://www.reuters.com/article/us-ams-results/shares-in-sensor-maker-ams-soar-after-upbeat-outlook-idUSKCN22B0HM.

  23. Patent Claim: “A system, comprising a computer in a vehicle, the computer comprising a processor and a memory, wherein the computer is configured to detect a condition of an operator of the vehicle; determine that the driver condition is an impaired condition; and perform at least one autonomous operation based on the impaired condition.” https://patents.google.com/patent/US20150066284A1/en.

  24. “Technical experts found serious programming mistakes in the machines’ software.” https://www.nytimes.com/2019/11/03/business/drunk-driving-breathalyzer.html.

  25. https://senseair.com/applications/automotive/.

References

Download references

Acknowledgement

This work is supported by UNSW Allens Hub for Law, Technology, and Innovation.

Author information

Authors and Affiliations

Authors

Contributions

Both authors contributed to study design. Initial research analysis and first draft written by Simon M. Taylor. Dr. Marc De Leeuw contributed substantially to following versions—particular focus on legal aspects. Both authors read/approved the final manuscript.

Corresponding author

Correspondence to Simon M. Taylor.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Taylor, S.M., De Leeuw, M. Guidance systems: from autonomous directives to legal sensor-bilities. AI & Soc 36, 521–534 (2021). https://doi.org/10.1007/s00146-020-01012-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00146-020-01012-z

Keywords

Navigation