Keywords

1 Introduction

Performance requirements for US Air Force (USAF) weapon systems increasingly drive needs for higher precision and quality manufacturing on the first build. For example, complex composite aircraft assemblies require meticulous, exact work to meet performance requirements. Emerging complex materials, manufacturing methods, and assembly processes bring increased opportunities for errors on the production floor. These errors necessitate rework, repairs, or sometimes even scrap of the non-conforming materiel. The costs and time involved in correcting errors consume resources that the Air Force cannot afford, particularly as the culture of “doing more with less” continues to grow across the US Department of Defense (DoD) [1].

The Air Force is interested in moving toward production of weapon systems in smaller lot sizes and tailoring each lot to address specific mission requirements, which increases production variability and the need for getting things right on the first try. When building small lot sizes, the production workforce does not have the same opportunity to learn on the job and improve quality with repetition as they do in current production settings where lot sizes include hundreds of the same item. The Air Force and its supporting contractors need to flatten the learning curve so that the quality of the first aircraft is the same as the twentieth. Efficient training and interactive, in-process instructions can help the workforce adapt more quickly to new tasks. As older workers retire, a way is needed to transfer their expertise and guidance more efficiently to a new generation. This is even more important considering that the DoD does not have twenty years to train new workers to the same skill and knowledge levels existing in the current manufacturing workforce. Younger members of the workforce in particular are accustomed to using electronic devices in everyday life, but when they reach a manufacturing floor, particularly within the defense industrial base, they must rely on antiquated systems—many still paper-based—to obtain necessary training, work instructions, and quality requirements. Even when electronic instructions are available, they are often text-based and include only two-dimensional representations of the parts.

The Air Force Research Lab (AFRL) Manufacturing and Industrial Technologies (AF ManTech) Division seeks to address these and other factors to modernize the Air Force’s manufacturing capabilities and processes. Envisioned is a “Factory of the Future” integrating humans and technology on the production floor to enable agile manufacturing and to improve quality, efficiency, and safety [2]. To achieve this vision, AF ManTech strives to invest in emerging and developing state-of-the-art technologies and to incorporate those technologies into USAF operations, both those performed directly by the Air Force and those contracted to supporting companies.

Technologies that show promise for improving manufacturing and maintenance workforce effectiveness include digital work instructions and, in particular, presentation thereof through augmented reality. Major aerospace OEMs such as Airbus and Boeing are investing heavily in augmented/mixed reality (A/MR) solutions, and they are developing rapidly [3, 4].

To gain the maximum advantages of digital work instructions, interactive, context-specific instructions should walk workers through tasks step by step, and feedback should let workers know whether tasks have been accomplished correctly. Task completion information should be collected and returned automatically to factory management and aircraft data repositories as needed to provide actionable, real-time factory awareness and as-built information for individual aircraft. These technologies should be unobtrusive and simple to navigate. An industrial setting for providing digital work instructions provides a particular challenge—but also a ripe opportunity space—for A/MR. The Air Force is beginning to explore this space and seeks to determine what applications of wearable technologies and augmented reality may be best suited for USAF-related manufacturing environments.

2 Augmented Reality for Industrial Settings & the DoD

To clarify further discussions in this paper, a few definitions will be given regarding terms that are often heard in discussions of augmented reality, particularly for industrial environments.

Digital work instructions are simply work instructions which are presented electronically to the worker at their point of use (POU) on the production or depot floor. Even a simple move from paper work instructions to a digital representation of the same instructions—same text and images, perhaps with related information appropriately hyperlinked to each other—provides benefits on the shop floor. The paper trail of printed instructions can be greatly reduced, and foot traffic to obtain the instructions may be minimized if the instructions are presented on a portable device. Using electronic work instructions also helps keep the instructions up to date with engineering changes in real time, reducing rework by ensuring that workers are always utilizing the most recent version.

When 3D image capabilities replace 2D images in electronic work instructions, even greater benefits are seen. Directions may include much clearer visual information, making task completion or component placement more intuitive. Interactivity with the 3D images is particularly helpful, since workers may manipulate the images to gain better understanding of the task. Errors due to complexity of the parts being assembled are thus decreased, and work accomplished shows greater fidelity to the product design [5]. If as-manufactured, as-built data is incorporated, work instructions may even be tailored to be specific to the tail or serial number being produced.

Digital, 3D display of instructions can be improved further by presenting information and instructions to workers through augmented reality. Augmented reality (AR, and used interchangeably with the term “mixed reality” in this writing) overlays digital information on the real world. This may be achieved through a number of technologies, and a few examples are as follows:

  • A tablet that simultaneously uses its camera function to show the user an assembly in front of them while displaying visual and/or audible information about and 3D models of the parts

  • A projection system which displays step-by-step work instructions directly onto a fuselage while skin sections are being attached, with color coding for different types of fasteners

  • Glasses (or a helmet with a transparent visor) that allow the user to look through them and see both the real world and information that is placed in the user’s field of view and related to the objects which the user is seeing

Augmented/mixed reality (A/MR) should not be confused with virtual reality. Virtual reality is a fully immersive experience, often achieved by means of a headset which encapsulates the wearer’s field of vision [6]. Virtual reality is not seen as a viable option for manufacturing settings. Workers cannot walk around a factory floor while wearing something which impedes their entire field of vision! Needless to say, this would cause an unacceptable safety concern.

Whatever technologies are used, AR combines the physical and digital worlds, endeavoring to use information from the latter to help the user make better sense of the former.

2.1 Initial AFRL Study

To understand the current state of AR for manufacturing, AF ManTech sponsored an industry study [5] which performed the steps described below and was completed in April 2015:

  • Reviewed recent literature on augmented reality, especially for industrial applications

  • Assessed current and developing state-of-the-art POU technologies by visiting major technology providers (Google, Sony, Epson, etc.)

  • Defined the DoD’s baseline of needs and capabilities by tours and discussions at multiple DoD locations and industrial base partners (such as Boeing Defense Systems, Lockheed Martin, Oklahoma City Air Logistics Center, Portsmouth Naval Shipyard, and others)

Four major technologies were studied—tablets, projection, glasses, and VR visors. Because of the previously-mentioned safety issues found for VR visors, they will not be discussed further in this paper. The goals of the study were to identify and prioritize manufacturing and sustainment user requirements, common enabling technologies, and technology gaps for production and depot maintenance environments. The study team was comprised of AF ManTech, Universal Technologies Corporation, and Schaefer Marketing Solutions. Their findings formed a discussion (summarized in the following sections) of the advantages of each POU technology studied, challenges observed for industrial A/MR overall, and recommendations for further AF ManTech investments.

2.1.1 Tablets

Tablets have been found to greatly reduce or eliminate paperwork on the shop floor. They deliver more accurate instructions to the worker, since they are kept up-to-date with engineering changes in real time. Additional time savings are seen when workers do not have to wait for lengthy instructions to print on paper. Users gain a better perspective with the ability to manipulate and zoom in and out of images to understand the part or assembly in question. New ideas or best practices are very easy to capture, as well, using a tablet’s photography and videography tools; these can then be incorporated into training or instructions as appropriate to transfer skills and knowledge to younger workers. Other advantages noticed through use of tablets during production were (1) immediate documentation of non-conformances, which saved workers’ time since they did not have to document it on paper, then wait till the end of their shift to enter it into a computer, and (2) reduced miscommunication that was simply due to poor handwriting!

During the visit to Portsmouth Naval Shipyard, a Navy project was discussed which had investigated tablet usage in ship production at Portsmouth and three other Naval shipyards and maintenance sites. Parallel work was performed with one group using the old paper-based system and the other group utilizing a pilot POU system. The demonstration’s extrapolated cost savings were $61M a year by 2018—for a total project cost of $27M. A similar case study by Iowa State University and a major DoD industry partner found an 800% improvement in quality and a 30% productivity improvement. The industry partner was “blown away by the quality improvement,” which they considered to be a bigger business case than the cost improvement [5].

Tablet applications for use in industrial settings are fairly mature and were the most widely available of the four technologies studied. Users are often already familiar with tablet usage, as many have their own tablets at home, so that training is easier than with some other technologies.

2.1.2 Projection

Projection has major advantages in that it induces the least burden on IT infrastructure and has gained worker acceptance due to its ease of use. It is scalable from a relatively small area to much larger areas. Since projectors are typically anchored well above the working area, they do not impede movement of people, materiel, and equipment into the work area. Projectors are fairly easy to install, after which they are also low-maintenance; other than the occasional bulb needing to be replaced, they do not require much care. Projection is the most mature technology of the three for industrial settings, but it has limited adoption throughout the DoD thus far, primarily because of lack of awareness.

Previous AF ManTech investment successfully developed a projection application which is currently in use for several different USAF weapon systems [7]. It was developed to assist with composite skin fastener installation. Because of variations in composite skin thicknesses, each fastener hole is measured and the correct fastener grip length for that hole is recorded. Previously, this process was entirely manual; holes were measured with a handheld tool, while the appropriate fastener grip lengths were recorded on sticky notes and masking tape near the holes. The fasteners were then manually selected, prepared, placed in the corresponding holes, and installed. As one might imagine, this was very time-intensive and resulted in numerous incorrectly located fasteners (which then led to rework, repair, or scrap of the affected assemblies). With the new projection system, a wireless grip gun is used to identify and digitally record the fastener sizes needed for the different holes. This information is automatically sent to supply, which creates a fastener kit accordingly. Fastener sizes and locations are projected directly onto the assembly, with color-coding for different fastener sizes. Installation instructions are projected onto the assembly to guide workers step by step through assembly of the different components. One of the Air Force Original Equipment Manufacturers (OEMs) has found several hundred hours of labor reduction per aircraft from using this projection system, as well as an unquantified amount of error reduction and rework, for an estimated total savings of $111M on one weapon system alone (after total development and installation costs around $8M for several weapon system production lines). Further developments of this capability could include job progress tracking and feedback to the user throughout the process.

2.1.3 Glasses

Glasses are seen as the ultimate ideal for manufacturing settings since they provide truly hands-free capability. Because they overlay instructions and 3D models over real objects directly in the user’s field of view, the worker does not have to move their line of sight from work in progress to view needed information. They provide the ability to record as well as to transmit, which may be particularly helpful when paired with image recognition to provide job tracking and feedback to the workers on the successful completion of their tasks.

Numerous technology companies are working to develop augmented reality glasses, and this momentum has only increased since AFRL’s baseline study. While many of them focus mostly on gaming and home life applications, some are delving into AR glasses for industrial use, as well. Some safety glass and helmet versions [8, 9] are in development, which is especially positive because many manufacturing workers are already required to wear safety glasses and hard hats. Rather than introducing yet another thing for the workers to wear and keep track of during their shift, AR safety glasses would simply provide additional utility for something to which workers are already accustomed.

2.1.4 Challenges

As with most things, these technologies are not perfect, and each has its own unique challenges to overcome for optimal effectiveness on the production line. Tablets are not hands-free, so workers may have difficulty performing their tasks while using one. Many of the electronic work instructions used with tablets are still focused on 2D, text-based instructions, rather than moving to more updated presentations which provide additional benefits. Projection requires a fairly fixed, open space with multiple projectors for each location. It also suffers from line of sight issues, so it does not work well for a dynamic environment or for enclosed spaces. Glasses experience issues with correctly adjusting to the depth of the user’s field of vision and fidelity of the data presented. Major concerns are the power requirements and data bandwidth required, as well.

Other challenges are common to all of these digital work instruction technologies. Augmented reality is only as good as the data that drives it. To be of use on a factory floor, good technical data packages and translation to useful information for the worker will be crucial. A healthy connection with the Digital Thread must be arranged so that Point of Use technologies both provide input to and feed from the information contained therein. Digital Thread refers to the DoD’s goal of modern data management, linking, and analysis to enable informed decision-making with as-manufactured, as-built, as-flown, and as-maintained information for each unique item or aircraft [10]. This is especially difficult for legacy aircraft, which often lack complete technical data packages (deficient or absent process and property definitions, as well as design intent), 3D data, and other important digital information. The DoD struggles to balance modern IT infrastructure with security needs, usually to the detriment of their internet and network capabilities. Between DoD network policies and cybersecurity concerns inherent to live wireless internet connections, security of the data and data feeds will be a large obstacle to work through before implementation of any A/MR solutions in DoD facilities. Ownership of the data being recorded must stay with the DoD and/or its partners, as appropriate, rather than being owned by the technology provider of any augmented reality solution. One final DoD-specific consideration: different processes, needs, and policies amongst the Services (or even at different locations within a single Service) may impede the fullest extent of advancement available through joint problem-solving and technology improvements.

2.1.5 Findings

The initial case study improved AF ManTech’s understanding of user requirements and technology gaps across multiple domains. A formal analysis of technological alternatives provided a prioritized list of available technologies and potential solutions. Estimated returns on investment were compared, and ideas were provided for implementation in both production and sustainment environments. An important thing to note is that the breadth of user requirements and operational environments will likely negate a “one size fits all” solution. While technology development will reap greater benefits from multi-organization and -Service cooperation, it is expected that some locations will benefit more from some technologies than others. Also, using different technologies for different personnel or functions at any given location may prove more beneficial than using the same solution for every employee there. For example, inspectors and quality assurance personnel might find tablets to be most useful, while glasses may be ideal for the technicians assembling components.

An interesting finding of the study highlighted when it makes economic sense to implement an augmented reality or POU solution. As can be seen in Fig. 1, the best value of AR seems to be found for tasks of relatively high complexity and high repetition—but not quite so much repetition as to justify automating the task.

Fig. 1.
figure 1

Economics for POU Technology [5]

Full size image

Several benefits were found from POU implementation. Work instructions and 3D models were provided to the user at the location of work accomplishment. Users could more easily track their own job progress and identify and fix errors very quickly after they occurred. The paper trail and foot traffic to obtain work instructions was eliminated. Training efforts were eased greatly, and corporate knowledge could be more easily transferred. In addition, as-built, as-manufactured data could then be used to make work instructions individualized to each job or aircraft tail number. An example of when this would be beneficial is during depot maintenance on legacy aircraft; cyclical loading on the aircraft over time may mean its components no longer exactly match the nominal dimensions of their original drawings, or repairs may exist on part of the skin or structure that affect how a worker must access inspection areas. Tailoring a task’s instructions to accommodate the actual dimensions of the aircraft components or the repair in place could save time and mistakes due to conflicts between the as-is and as-designed states of the aircraft.

2.1.6 Baseline Study’s Recommendations

Several recommendations were made in the final report for this study based on the information learned. The first recommendation was to combine wearable image recognition technology with POU work instructions to improve quality through automated real-time error-proofing. For instance, this would enable continuous, automatic aircraft panel seam inspection throughout assembly and repair. All seams are currently inspected 100 % more than once (between steps and after each assembly). If the seams are found to be out of specification, rework must be accomplished to remedy the issue. With the recommended improvement, however, the computer could scan and record the seams while workers assemble the parts. The system could then alert the workers if pieces are being misassembled. The error could be resolved immediately, rather than disassembling the component to redo several steps upon finding the error later. This would save significant time on both inspection and rework.

The second recommendation involved projection systems. As stated earlier, it is a fairly mature technology for industrial applications, and its IT infrastructure burden is much lower than those of the other technologies studied. This means it might be the best option for DoD depot maintenance applications, given their restricted network capabilities. They often have large, fairly fixed workspaces, which lend themselves well to projection solutions.

Finally, the DoD needs to invest in the necessary infrastructure and data procurement standards which will be essential to enable AR technologies. Current data procurement standards result in substandard technical data packages (TDPs) available for government use, even when considering applications much more mature than AR. Mixed reality use in DoD facilities will be of limited benefit at best if TDP completeness and availability are not addressed. Limited 3D models and data are used in the DoD now; the lack of ability to handle 3D data on the Government side will restrict or preclude the usability of tablets and glasses in DoD facilities. Even without taking AR into account, the DoD’s limitations regarding 3D data and IT capabilities must still be addressed. Industry is increasing its reliance on 3D work instructions. Although these are often presented on a computer workstation and driven by nominal CAD model data, rather than being presented at the worker’s POU with as-manufactured information, they are still a step in the right direction. Innovation in the DoD will be limited at best if the DoD does not move toward more modern standards regarding TDPs, 3D data, and IT infrastructure and capabilities.

3 Next Steps for Air Force ManTech

After the initial AFRL study, AF ManTech has continued to monitor industrial A/MR technology developments. There is much work being done to improve AR applications for manufacturing and maintenance environments, as well [3, 4, 8, 9, 11–13]. AF ManTech is in the process of defining appropriate AR technology topics on which to focus, selecting specific application areas, and determining demonstration and use case opportunities. Potential interest areas include the following:

  • Real-time training during actual task completion

    • On-the-job training—work done during training is part of the production line

    • Increased production with more experienced workers continuing to provide value-added work on production rather than training less experienced personnel

    • Reduction in quality differences from personal learning styles—worker capabilities depend on directly following visual instructions, rather than being dependent on correctly interpreting text instructions and 2D drawings

  • Work instructions

    • Quantify benefits of work instruction presentation through POU applications

    • Compare POU benefits to the costs to implement—including installation, training to use the POU technology, preparation of work instruction packages for POU vs. conventional methods

    • Understand maintenance needs for POU tools and the methods and personnel needed to keep them up to date

  • Improved tracking and verification of task completion

  • Instant communication with SMEs who are not on the production floor or maintenance line

    • Description of non-conformances with actual images (video, photographs, and/or 3D model annotation)

    • Explanation and clarification of instructions

    • Coordination of repair design and installation, even if the user and SME are on opposite sides of the globe

  • Business intelligence for factory awareness—more accurate, real-time information for plant management, as well as providing information for value stream analyses and process improvements

  • Quality improvement through in-situ quality inspections and immediate notification and resolution of errors

A number of considerations must be addressed during these AF ManTech efforts. For starters, mixed reality development should coincide and have a natural symbiosis with Digital Thread efforts. POU technologies should help populate the Digital Thread with data gathered during task completion, such as job tracking and quality assessment. Conversely, AR technologies will be useless for manufacturing settings without input from a well-constructed, healthy Digital Thread.

Another consideration will be to balance the standardization of AR usage across the DoD with location- and Service-unique requirements. More standardization than is currently in place would likely be beneficial, but a single solution will not address all the needs of every location—at least not without becoming so unwieldy that it ceases to be a solution and turns into a another, more technologically advanced problem. A case in point would be the infamous Expeditionary Combat Support System (ECSS) in which the Air Force spent over a billion dollars attempting to develop an over-arching enterprise resource planning program which ultimately failed [14]. In a related vein, one POU technology will likely not solve every problem on the shop floor; combinations of technologies are expected to provide the eventual solution. To illustrate the advantage of a smartly connected federated system as opposed to having one large system which addresses all information needs, consider the sharing of information amongst smart devices. One might have a smart watch, a cell phone, and a tablet, each of which contains its own version of a particular fitness tracking app. One would not expect the smart watch to perform all the same functions as the tablet—but it does not need to, either. The smart watch unobtrusively tracks the specific fitness items for which it has been programmed, and the information is stored in the cloud. The cloud maintains the authoritative records of the fitness information, but it also disseminates the data to each device with the fitness tracking program and that user’s account so that the data on each device is identical. One device may serve better than another for recording that data, while a difference device may serve better for displaying information derived from that data. Finding the combination of AR technologies that works best for all involved parties will present its own challenge, and developers will need to make sure that information is shared easily between different locations and types of technology with minimal human involvement.

While many electronic work instructions are still focused on 2D images and text-based instructions, AR may provide more intuitive ways to communicate directions. The process to assemble components may be made much clearer with 3D animations and step-by-step guides. For instance, IKEA® and LEGO® directions contain very few, if any, words and are on paper rather than being animated, but they are still able to effectively communicate to their users what needs to be done. Of course, any solution(s) must be user-friendly, reduce human error, and maintain end-user safety. Developing these technologies is pointless unless they provide faster, easier production and maintenance and lead to higher asset availability, reliability, and quality.

Finally, while initial demonstrations and use cases are expected to take place in production environments, AF ManTech would like to structure these demonstrations so that they will also provide as much useful information as possible for implementation in sustainment environments. Guiding developments and demonstrations to be scalable from one environment to another may meet with limited success, though, due to the differences in requirements and capabilities already in place.

4 Conclusions

The economic arguments for advancing and adopting Point of Use technologies are clearly favorable. There is still work to be done to improve them, especially for manufacturing settings, but it is being done and will become increasingly more widespread in the next several years. This is a vital time for the United States Department of Defense to be involved and to take a leadership position to guide development of these industrial AR solutions. Otherwise, they will not be positioned to take greatest advantage of the benefits these advancements have to offer. Providing safe, working equipment to our warfighters in a timely manner is a worthy goal. Indeed, it is and must be the end goal of any technological improvements brought about through DoD ManTech efforts. Augmented reality can help us achieve it.