1 Introduction
Medical making is emerging alongside maker efforts (e.g., hobbyists, engineers, designers, digital fabrication enthusiasts) to apply crafting and digital fabrication to invent, manufacture, and repair medical devices. Prior research provides rich insights into the social norms of makers [
14,
30,
55] and their collaborative practices in shared repositories [
1,
4,
7]. Unlike many other domains of making, medical making raises vital concerns about safety and efficacy because medical devices can pose significant risks to life and limb. The demand for
Personal Protective Equipment (PPE) in the COVID-19 pandemic overwhelmed global supply chains. In response, US governmental institutions, including the
Food and Drug Administration (FDA) and
National Institute of Health (NIH) adapted an existing open source platform for bio-scientific models to allow sourcing and reviewing of alternative, open-source PPE designs. To ensure that designs are safe to use, the
NIH, in partnership with the
Veteran’s Health Administration (VHA),
FDA, and
Center for Disease Control (CDC), began reviewing these submissions. By analysing this collection, we contribute a better understanding of the effect of a critical, safety-review process on how makers create, reuse, and share designs during a health care crises.
Our analysis of this extraordinary occurrence helps us learn how to better utilize online-repositories to support safety-critical open-source design efforts. We present a mixed-methods analysis of the NIH 3D Print Exchange’s COVID-19 Collection. We use a combination of qualitative data from a thematic analysis and quantitative data from web scraped details of the 623 submissions posted between March 20th, 2020 (its start date) and January 1st, 2021. Our aim was to understand how makers design, produce, and share PPE on a safety-focused open source repository.
Our results characterize what was submitted to the NIH 3D Print Exchange, who contributed, and how designs were remixed and shared. Open maker repositories with no formal review process tend to include a wide range of unique designs [
4,
27]. Contrary to these observations, on the NIH 3D Print Exchange, we observe only a few design archetypes and numerous derivatives with manufacturing-focused changes. We discuss possible factors that may have caused this unusual design convergence. We further discuss the characteristics of the design teams. We find that there were two main groups of makers: those
affiliated with professional organizations (e.g., engineering companies, universities, clinics) and
unaffiliated makers. These groups differed in their review outcomes, detail of documentation, and the types of designs they created (e.g., novel archetypes versus remixes of existing popular designs). We found that the affiliation of makers impacted the amount of design documentation provided. Ultimately, we believe this contributed to discrepancies between the reviews of unaffiliated and affiliated makers.
Based on our findings, we recommend three different ways that open-source design repositories, particularly those with review processes, can yield greater design diversity while maintaining safety. First, make the reviewing process more transparent and effective by (1) ensuring that all key information required for a review is marked mandatory, and (2) providing feedback about why designs received their review rating. Second, introduce a required field to explain updates made to a design in remixes. Small updates can be more rapidly reviewed than more involved changes. Finally, pose clear requests to the community. Clearer communication can help ensure that designs diverge rather than converge on what is already positively reviewed. These features together save reviewers time and position innovation and creativity as values complimentary to a community’s value of safety.
In summary, medical making is a burgeoning field. Being able to share designs via repositories is critical, as is the ability to tailor these designs to meet individual needs. Via a case study into the outcomes of PPE design, we sought to understand how unaffiliated and affiliated makers contributed to an open-source medical making repository that imposes a review process. Our work contributes: (1) an analysis of the types of submissions to the forum, including how designs were made and documented; (2) insights into how a safety-focused review process impacted the scope of what makers design; (3) details around how designs were shared and remixed among makers from various backgrounds; (4) data about who participated in the forum and how we can broaden participation in safety-critical making; and (5) insights into how we can support safe, streamlined making in regulated domains.
2 Related Work
2.1 Digital Fabrication and Medical Making
Medical applications for digital fabrication are on the rise with advances in 3D printing [
24,
37,
52]. Most studies track clinician experiments with the novel use of fabrication technologies in bio-printing [
53], surgical guides [
34], dentistry [
11], implants [
10], prosthetics [
18], and orthotics [
9]. This trend correlates to a history of crafting practices and device improvisations [
15] and open source infrastructures. However, recent HCI studies indicate a wider variety of
“medical makers” [
29] who engage in medical device development and deployment in care delivery roles. They adapt the fabrication process to suit specialized practice [
22] and generalized care norms [
29].
While maker culture is often framed around individual’s interest in applying new technologies (e.g., hedonistic preferences [
51]), maker communities exist in a complex ecosystem of sociopolitical actors [
30], community structures [
14]), and entrepreneurial opportunities [
31]. Particularly at the intersection of making and health, communities are highly influenced by the socio-technical constraints on healthcare systems. In work focused on intermediaries between maker communities and healthcare institutions, Lakshmi et al. [
28] found that medical making is a form of infrastructure repair, rather than a strictly innovation oriented practice. This aligns with Hofmann et al.’s and Slegers et al.’s findings that occupational therapists limit material iterations to integrate digital fabrication into their standard practices, packed schedules, and to keep costs to a minimum [
22,
49]. Lakshmi et al. discuss how clinician-makers hesitate to distribute prototype designs without due regulatory approval or licensing extending from an ethos of safety and a risk aversion [
29]. In contrast, Hofmann et al. find that medical maker communities with little participation by clinicians tend to be action and innovation oriented, at times at the cost of safety [
21]. While 3D printing advocates in medicine proved prescient in the COVID-19 PPE crisis [
21,
28,
44], regulatory and policy infrastructure in the medical space is underdeveloped. Such confusion inhibits the potential of medical makers. Careful mediation between clinical, professional, and hobbyist makers is required within these communities to mitigate this confusion [
20]. Further similar to the open source software development communities [
56], flexible and ad hoc coordination is key for efficient medical maker response. Medical makers already defer to their professional norms to uphold safety and reliability with risk-averse approaches. In uncertain times, these factors may conflict with expectations of novelty and variety with 3D printing overtly recognized as a tool for innovation within the medical research community (e.g., VHA’s Innovators Network [
46]). It is unclear how these social and material constraints influence how medical makers engaged in digital fabrication.
2.2 Barriers to Reuse and Remix in Digital Fabrication
Reuse and adaptation of shared designs is a perceived benefit of online repositories [
7]. Makers can learn collectively by remixing each others work in these repositories [
5,
27]. They expect to adapt designs and re-share them as part of their articulation work for future reuse. Schmidt defines articulation work as
“cooperative work to make cooperative work work” undertaken by members inside a community [
48]. However, these expectations are constrained by factors specific to the digital-physical material process affecting both adaptation and articulation work, and adding barriers to collaboration [
40].
Unlike physical artifacts, novelty of an adapted digital artifact can be attributed to the extent of variation from the original as Cheliotis et al. note in their study on a musician community [
7]. On Thingiverse, Kim et al. describe how popular contributions of preferred digital file types rely on real-world constraints around printer filaments and reliable outcomes [
26]. To support collaboration between users, it is better to share the source files generated on modeling tools to retain the original geometry of the model and make editing easier [
23]. However, Alcock et al. report an overwhelming preference for STLs
1 (84%) over design files (3.7%) on Thingiverse [
1]. This signals a preference of makers to conveniently download and print models, rather than modify them. Regardless of their popularity, STLs are inflexible file types lacking metadata, forcing makers to rebuild designs from scratch to perform modifications [
19]. Consequently, Stemasov et al. suggest building tools that abstract challenging design work and focus on how to best support remixing behaviors [
50].
When makers share editable models, they fail to articulate key details such as the design’s purpose and manufacturing details. Even experienced users can struggle with inferring the details of a print when there is insufficient documentation on materials, print settings, and/or assembly instructions [
32,
45]. One part of this challenge is that makers rarely document these aspects of their designs as they go, and are unlikely to add documentation before sharing models online [
2]. Furthermore, many novices in 3D-modeling struggle to understand the intricacies of models such as dealing with print uncertainties [
26] and learning how 3D-models interact with real-world geometries [
2,
8,
19], making it difficult for them to explicitly document how their design works. One solution may be to integrate the documentation process directly into 3D modeling processes [
12,
19], however no widely adopted standard tools support this workflow. In the rare case that all relevant information is included, variations in printer and filament can still cause prints to fail [
26]. On sharing platforms, insufficient documentation is partly addressed on user forums by the community’s discussion on specific 3D-models. This reactive process is not sustainable over time as users continue to remix the model. Documentation can be lost with each iteration [
1,
13]. Moreover, the process increases the burden on the author of articulating their designs.
Looking beyond maker communities to other open-source volunteer efforts, we see that there is a tradeoff between encouraging articulation work and encouraging participation. For example, on Wikipedia, Kriplean et al.’s case study analyses how moderators’ contribute “
meta-work activities” (e.g., participation, support, and quality of outcome) rather than specific and measurable editing-tasks. Morgan et al. in their analysis of alternate WikiProjects found open collaborations persist when they maintain low barriers for participation and community-adapted social structures [
36]. Most maker communities favor a flexible, informal structure [
25] to maximize participation especially from volunteers [
56]. Erring on the side of supporting participation may have safety consequences when documentation is incomplete for medical device designs.
It is not surprising that time constrained medical makers avoid adopting open source designs [
29]. Working within institutional infrastructure, their efforts to make medical devices are further subject to available technical expertise, uncertainties about the materials they are working with, and licensing or regulatory mandates to ensure safe use. Yet, adoption of repositories like the NIH 3D Print Exchange indicates that medical makers publish their designs for use, reuse, and distribution.
We add to existing literature about making in the pandemic by examining the emergent practices around the recent push to make and design PPE [
6,
21,
28] on the NIH 3D Print Exchange. Novak and Loy undertake a wider analysis of COVID-19 response efforts in early 2020 [
44], while Buford et al. examine the practices around making cloth masks during the pandemic [
6]. The latter study found that remixing of cloth mask patterns was common to meet personal objectives (e.g., sustainability), but could contribute to misinformation around safe mask making; they noted a lack of a common technological infrastructure to aid with the manufacturing or material acquisition process. Our study takes a deep dive into the medical maker community on a single platform, the NIH 3D Print Exchange.
3 Background: the NIH 3d Print Exchange and COVID-19
The NIH 3D Print Exchange is a 3D model repository hosted by the US government with an exclusive focus on collecting “bioscientific” models. Prior to the COVID-19 pandemic, the NIH 3D Print Exchange served as an “
open, comprehensive, and interactive website for searching, browsing, downloading, and sharing biomedical 3D print files, modeling tutorials, and educational material” [
41]—an open repository for medical making. Submissions included biomedical models (e.g., molecules, organs), a small collection of open source prosthetic-like devices from e-NABLE, and simple 3D printable lab equipment (e.g., test tube holders). The goal of the project was to be an authoritative source for medical makers’ designs. It included the extensive documentation needed to reliably make such models wherever physical equipment was available nationwide. By early 2020, there were efforts to add an expert review process to the exchange that would enable makers to receive feedback from VHA and FDA experts. The program roll out was hastened to completion in response to the surging COVID-19 pandemic in March of 2020.
In January 2020, the COVID-19 virus spread across the world, and United States saw a surge in hospitalization rates. To prevent the spread of the virus, people were recommended to wear cloth face masks and socially distance from each other. Essential workers in healthcare, retail, and other areas required protective gear on a routine basis. The mass hospitalization rates combined with greater non-medical use of face masks placed unmanageable demands on traditional supply chains. Around the same time (March–April 2020), local and state governments were issuing stay-at-home orders, and many people were left at home without work. Those who were makers turned to their fabrication devices and began designing and creating stop-gap PPE designs. Additionally, because of global shortages, the NIH 3D Print Exchange released its new review process early through the new COVID-collection, which was meant to be a place for “Open Design” [
17]. The collections’ goal was specifically: “
to inform decision-making on PPE and medical device production, without stifling innovation...by filtering designs through a systematic review process” [42]. The collection was intended to connect innovative makers and manufacturers to produce products to fill in supply gaps. Anyone could submit their designs to the collection, queuing them for review by medical and engineering experts within the NIH and other government affiliates.
Makers submit their designs through an extensive, publicly available form (Figure
1) [
43]. They can provide: a textual description, manufacturing details (e.g., 3D printer model, materials, design files, pre-processing, assembly, cleaning, and user instructions), licensing information, and documentation (e.g., images, testing procedures, and data). Few of these fields were mandatory. All submissions are marked as “prototypes” before they are reviewed. Submissions are reviewed based on a priority determined by (1) demand (i.e., the design meets an unmet need), (2) feasibility (i.e., it seems reasonable that the design works as described), and (3) detail (i.e., the submission includes enough information to make review possible) [
42]. Reviewed designs are independently produced and tested with actual materials by reviewers to determine what classification, if any, is appropriate. More detailed criteria for review are listed on the collection’s FAQ and in a document detailing the different types of masks (general use face masks, surgical face masks, and N95 respirators). Specific criteria for other types of PPE are not present at this time.
Besides the default prototype status, submissions’ review status can be: “reviewed for clinical use”, “reviewed for community use”, and “warning”. Note that none of these terms include the word “approved”; this is a purposeful decision to remove confusion between the exchange’s review process and FDA approval processes. Positively reviewed designs on the NIH 3D Print Exchange still do not have FDA approval. Submissions reviewed for clinical use are deemed to be the safest and most effective submissions. These are appropriate to use in a high-risk clinical environment. Community use denotes a lower standard where the device itself is expected to be safe but its efficacy cannot be guaranteed. That is, it will not hurt the user, but it might not protect them either. The warning category is used in rare cases where the design itself is not safe. Usually this was received for high risk designs like ventilator parts. If a design did not meet the clinical or community standards but was not so risky to merit a warning, the reviewers would privately provide feedback and leave the design marked as a prototype. Occasionally, reviewers left public comments before deciding the design’s final status. We cannot determine if reviewers left comments in all cases or only in the absence of a private email response.
This review process was quickly overloaded with a surge of new designs: “due to the growing number of submissions, the VA and its collaborating reviewers cannot guarantee that all designs will be reviewed, and there is no process to ‘fast-track’ any reviews”. On July 24th, the exchange stopped considering common face shield and ear-saver designs for review “due to the volume of submissions, unless the face shield is a novel design adapted for a specific use”. They turned their review efforts exclusively to nasal swabs for COVID-19 tests which make up only 7 of the 623 designs submitted during the study period.
The NIH 3D Print Exchange presents a unique opportunity for researchers to study what medical makers in the United States do when collectively tasked to address a healthcare crisis. Unlike open maker repositories, the NIH 3D Print Exchange includes an explicit review process and heightened community safety standards that are in line with the standards clinicians strive to uphold. However, similar to other traditional repositories, makers contributing to the exchange likely still faced challenges in learning how to make safely and document their designs so that others can reproduce and build on them.
4 Methods
To analyze the NIH 3D Print Exchange PPE submissions, we collected fields from each of the 623 submissions (i.e., one PPE design) for quantitative analysis. We further qualitatively coded 520 submissions made before January 1st, 2021. We coded for three types of PPE which made up the majority of the submissions (83.5%): face shields, masks, and ear-savers. Each submission was reviewed manually to determine if it was a face shield, mask, ear-saver, or another device. Our final sample of masks, face shields, and ear-savers was 520 of the 623 total submissions made prior to January 1st, 2021.
Based on the submission form structure, for each submission we scraped, where applicable, the:
—
Remixing attribution and the original design
—
Manufacturing method (e.g., 3D printing, laser cutting)
—
3D printer model, if applicable
—
External documentation (e.g., images, videos, PDFs, website links)
—
Pre- and post-processing instructions
Each of these pieces of information was either scraped from a well-formatted field on the design submission page (see Figure
1) or found by searching the text associated with each entry for relevant keywords, since few of the fields were required for submission.
We performed an additional layer of processing on this scraped data to gain insights into makers’ reuse of other submissions in their designs (i.e., remixing). The form did not require makers to declare changes made in remixes, though some makers noted it in text. To capture differences across remixes, we compared fields between original and derivative designs and logged differences in key fields (e.g., manufacturing method, materials, modeling software, printer-used). Additionally, we searched all text associated with a model for a list of qualifying words that we saw repeatedly in our qualitative coding (e.g., more, less, faster, slower, thicker, thinner, safer).
In addition to this automatically collected data, the authors deductively coded each submission. Two authors investigated the codes by inductively examining 50 submissions selected through stratified random sampling across the three design categories; these codes were discussed with the other two authors involved in data analysis (four authors contributed to this effort). Many codes were generated based on the differences and interesting features that we saw in submissions (e.g., face shield covers front of face, face shield provides overhead protection, mask comes in multiple sizes). Additional codes were developed based on a review of the literature and co-current media coverage of makers’ response to the pandemic and based on expertise in medical making and 3D printing (e.g., presence of persistent documentation, affiliation of makers). After defining our final set of codes, three of the authors applied these codes in a top down fashion to all 520 face shield, mask, and ear-saver entries in batches of 50 stratified random samples; we repeated this process until saturation across the coders was reached, updating and removing codes based on group consensus. We reached saturation with an average inter-rater reliability of 0.87 (range = 0.64–1.00) using Krippendorff’s Alpha across all 16 accepted codes. These three authors went on to individually review the remainder of the dataset. We met weekly to update each other and discuss any uncertainties that arose.
The themes that emerged in our efforts to understand the impact of safety and review on the designs of makers included: affiliation impacted success in the review process, how tradeoffs between values were made in designs, and an overall convergence of the design space. We developed a shared understanding of the data through weekly meetings where PPE and codes were examined.
5 Results
In our dataset of the 623 submissions between March 20th and January 1st, the designs fell into three main categories: face shields (N = 263/623, 42.2%), face masks (N = 177/623, 28.4%), and ear-savers (N = 80/623, 12.8%) (Figure
2). The remaining submissions (N = 103/623, 16.5%) included mask cases, ventilator parts, or door-openers. In this section, we characterize the dataset of face shield, mask, and ear-savers that we qualitatively coded (N = 520). First, we discuss key properties: how they were designed, manufactured, and by whom. Then, we narrow our focus to key properties for medical making: replicability and safety.
5.1 Temporal Trends
Submissions surged right after the collection was created in immediate response to the pandemic. The total number of submissions steadily increased and peaked in the first week of April (Figure
3) then the submission rate dramatically decreased. It increased slightly with the resurgence of the virus in the United States in May 2020.
Makers tended to submit designs with greater perceived importance or complexity, as was found in other maker communities during the pandemic [
21]. The first submissions (prior to March 29th) were two ventilator valves and one face shield, which are simpler to model and manufacture than a face mask. Clinicians interviewed in contemporary studies express that these were more important for saving lives than ear-savers which only increase mask comfort [
20,
28].
5.2 Material Tradeoffs
Due to resource scarcity induced by the pandemic, makers made careful tradeoffs when selecting manufacturing methods and materials. Makers had to balance between competing goals of broadening participation, using available materials, rapid manufacturing, and the safety of a design. We present three examples below that highlight these tensions and the tradeoffs that we perceived in the designs.
5.2.1 Material Selection, Safety, and Participation.
Many submissions could be made by expert-amateur makers working with their personal fabrication equipment. The most common filaments used were all widely available to consumers: PLA (N = 223/520, 42.9%), PET (N = 34/520, 6.5%), PETG (N = 140/520, 26.9%), and ABS (N = 64/520, 12.3%). PLA, PET, and PETG are common and easy to print with. ABS, however, requires more advanced setups due to toxic off-gassing. Similarly, for designs that specified a particular 3D printer, the majority (279/312, 89.4%) used printers available to consumers for less than $10,000. Many submissions listed multiple filament options (N = 147/520, 28.3%) (e.g., printing a face shield in PLA or PETG). Notably, the most commonly remixed face shields (3DPX-013306, 3DPX-013359) could be made with several variations of PLA or PETG, and could be manufactured on a consumer printer. The prevalence of easy-to-use materials afforded opportunities for hobbyists and broadened participation.
On the other hand, more complex materials or printers could improve safety at the expense of participation. 16% of designs used materials that require special equipment or additional expertise to work with (e.g., TPU, Nylon, PC, ASA). The most commonly remixed mask was printed with nylon, which requires an industrial printer. Nylon was chosen because it can be properly sanitized, unlike PLA or PETG. Therefore, substitutions of other filaments could be unsafe. Similarly, some designs combined multiple filaments to meet particular design goals at the expense of manufacturing ease. For example, the “Helmet-Compatible Community Face Mask” (3DPX-013354) used a rigid material (e.g., PLA, ABS, PETG) for the snout to ensure the filter was held away from the nose and mouth. It used a flexible material (e.g., TPU) where the mask touches the face to improve comfort and air-seal. Choices by some makers prioritized safety features produced with advanced methods over broadening the participation through simplified manufacturing processes.
5.2.2 Powerful Tools that Limit Participation.
A design’s manufacturing method further determines who can make a design and how much work is required. Unsurprisingly, 3D printing was by far the most popular method (N = 482/520, 92.7%) followed by laser cutting (N = 49/520, 9.4%) and injection molding (N = 22/520, 4.2%) (see Table
1). While 3D printers are relatively slow and require post processing, they are widely available. Injection molding, on the other hand, is fast but inaccessible to most hobbyists. Maker participation in PPE manufacturing was broadened by the majority of designs which supported 3D printing by hobbyist makers.
Several entries listed more than one manufacturing method (N = 173/520, 33.3%), and often, multiple manufacturing methods were provided. For example, the “NAVAIR - TDP for 3DVerkstan Protective Face Shield” (3DPX-014090) lists that the submission can either be “ printed on non-industrial 3D printers or laser cut”. Makers who designed for multiple manufacturing methods could end up supporting a broader range of makers and/or increasing manufacturing efficiency.
Other designs utilized multiple manufacturing techniques for the same design. For example, the “Southern Tier Face Shield” (3DPX-014082) was one of several face shield designs that required a 3D printed frame that goes across the wearer’s forehead and a laser cut PC barrier to prevent droplets from reaching the face. Materials like PC can increase production efficiency because they can be quickly and automatically cut. Alternatively makers may increase post processing requirements to avoid using additional manufacturing machines. For example, office hole-punchers could be easily used with transparent, plastic, 3-ring binder sheets to create the clear face shield without laser cut plastic (N = 63/520, 12.1%). The “Livingston Shield v2.2” (3DPX-014416) instructs users to use a hole-puncher to create four holes in a transparency sheet to attach to the 3D printed face shield frame. Though the materials were common and unlikely to run out, this design requires more manual post processing to punch and attach the sheets to the 3D printed frame than laser-cut alternatives. Makers traded-off increases in production speed through advanced manufacturing with a slow, manual process that broadened participation.
5.3 Community Members
Prior work positions maker communities as mainly hobbyists working on independent projects in a shared space [
25]. In contrast, the NIH 3D Print Exchange was built to support the open exchange of designs and potentially foster collaboration across stakeholders (e.g., healthcare professionals, universities, companies, entrepreneurs, hobbyist makers). A total of 448 unique authors submitted designs. The median number of designs submitted per person was 1 and the range was 1–9. Our qualitative review revealed that most (N = 344/520, 66.2%) authors listed no affiliation with their submission. We suspect this indicates a lone maker who is not affiliated with a relevant organization. Those submissions with listed affiliations had team members from industry (N = 84/520, 16.2%), academia (N = 67/520, 12.9%), and the healthcare facilities (N = 59/520, 11.3%).
Numerous designs (N = 30) were the result of collaborations within and across institutions (Figure
4). The most common type of collaboration was between universities and health care facilities (N = 19/30, 63.3%). The “Stopgap Surgical Facemask” (3DPX-013429) lists 59 team members from for-profit institutions, universities, hospitals, the FDA, and the VHA. While makers affiliated with institutions (e.g., universities, hospitals) often worked in teams, unaffiliated makers rarely noted any collaborators.
5.4 Replicability and Documentation
For the NIH 3D Print Exchange to be useful, makers need to be able replicate submissions. We found evidence of remixing behavior (179/520, 34.4%), but only found 61 (11.7%) entries that comments reported as successfully replicated. Thus, remixing was prevalent, but its unclear if they were manufacturing others’ designs. We have no way of measuring the number of people who made a design and chose not to share that on this site. Thus, we examine other factors which may influence replicability (e.g., documentation and ease of manufacturing). We expect that submissions with more complete documentation and that are easy to make would be more readily adopted. Other factors, such as media attention or affiliation with famous groups (e.g., Prusa, e-NABLE) are also likely contributors but are beyond the scope of this study.
5.4.1 Prototype Remixes.
The NIH 3D Print Exchange facilitated collaboration and iteration for “remixing”, similar to other popular maker forums like Thingiverse and Instructables [
7,
45], respectively. A total of 131 out of 520 (25.2%) entries were listed as remixes or “other versions” of models on the NIH 3D Print Exchange. Figure
5 presents the remixing network (186 designs), omitting submissions that are neither a remix nor remixed. Many nodes (56, 30.1%) were only remixed once. There were notable outliers: one design, the “3DVerkstan 3D printed face shield head band” (3DPX-013306), was remixed 12 times and 4 additional designs were derivatives of those remixes. Another, the “DtM-v3.1 Face Shield PPE” (3DPX-013359), was remixed 16 times with 6 additional derivatives. Both of these designs were made by affiliated makers (team members came from a European 3D printing company, Microsoft, three universities, and three hospitals). The mask and ear-saver that had the highest number of remixes were the “Stopgap Surgical Face Mask” (3DPX-013429) (five remixes), which was made by an expansive team crossing companies, universities, and hospitals, and the “Surgical Mask Tension Release Band for Ear Comfort & Extended Use” (3DPX-013410) (six remixes), which was designed by a VHA employee. It is important to note that three of these four designs were rated for clinical use, and that no designs in our remixing graph were given a warning usage rating, indicating that safety may have been a consideration in choosing what designs to remix.
Some remixing behavior is not captured by explicit links between submissions. For example, many designs shared a similar shape to the popular “Montana Mask” (3DPX-013443) which was spotlighted on Good Morning America on April 12th [
16]. Furthermore, not all makers attributed credit. For example, the maker of the “3 Hole Punch Minimal Face Shield” (3DPX-013501) found that someone had remixed their design by putting two copies of the original design in their printing file without attributing. They commented: “
at least credit the creator”.
Our qualitative analysis showed that remixes were primarily incremental changes to support alternative manufacturing techniques. Few changes were intended to significantly influence use or efficacy. A total of 54 of the remixes listed a change in materials, of which 20 added new materials not mentioned in the original design. A total of 25 designs strictly limited the number of materials options/materials used in a design. However, the majority (N = 17) of these designs only removed filaments that are difficult to use (e.g., TPU, Nylon, ABS). A total of 21 remixes used different modeling software than the original submission, which may make it easier for makers to replicate the design in the CAD tool of their choice. A total of 13 remixes used different 3D printers models enabling more people to manufacture the design and 6 tailored a design suited for “many” printers for an individual printer model. For example, the “FDM Printable version of Stop Gap Mask” (3DPX-013771) remixed the popular “Stopgap Surgical Mask” to “allow printing on hobby style FDM printers (PLA, PET-g etc)”. The original design required an industrial Powder Bed Fusion Nylon printer. Note that this change effects the mask’s porosity, making it harder to disinfect which, in turn, effects the design’s safety. Other common reasons for remixing designs included adjusting designs to fit different size print beds, take less time to manufacture, require less material, or to print more than one design at a time. Occasionally, designs affected comfort or ease of use in small ways (e.g., “[This change] makes it a bit more comfortable for different head sizes” (3DPX-013659)). While some of these changes may effect safety, none constitute divergence from the original design. On the NIH 3D Print Exchange, remixing behavior was almost exclusively small designs to broaden participation in the manufacturing process.
There are a few examples of substantial feature changes, often motivated by local user feedback. One face shield design, “Anvil Verkstan Visor” (3DPX-014089), significantly modified the popular “3DVerkstan V3 - Face Shield” based on community feedback: “The entire visor has been redesigned and model[ed] from scratch so there will be variances in widths, curves, length, etc. when compared to the original. We re-made this model to better support our local community in our efforts to help the workers on the front lines”. Another design, the “Surgical Mask Tension Release Band with Hair Stabilizer” (3DPX-013819), iterated on a clinically reviewed design to improve it based on issues experienced by clinician users: “They requested a way to keep the band from moving around/flying off while attempting to put on or take off their masks. I incorporated a section of hair pick so that the part can be inserted into the hair, where it will stay on it’s own, allowing both hands to be used for putting on or taking off the mask”. We observed few remixes like these, which implies that makers either created designs from scratch when addressing more significant design requirements (e.g., clinical usage, fit) or that more makers were interested in tweaking designs to support manufacturing under their resource constraints.
5.4.2 Documentation.
It is crucial that entries to the NIH 3D Print Exchange be well documented to foster communication between makers, reviewers, manufacturers, and PPE users. Documentation was often presented as static documents (N = 183/520, 35.2%) (e.g., PDFs), and video links (N = 31/520, 6%). Images were also a popular form of documentation. All entries included at least one thumbnail image, by default a view of the 3D model, and the majority included additional photographs or diagrams (N = 429/520, 82.5%). A majority of entries included at least one web link (N = 315/520, 60.6%), often to a portfolio or alternate repository (e.g., Thingiverse, GitHub). External website content is dynamic, but the NIH required static documentation to be included on the exchange itself. We stopped reviewing links because we found several broken links during our qualitative analysis. Finally, a majority of entries (N = 290/520, 55.8%) also included pre/post processing information, such as printer settings, cleaning instructions, and material recommendations, which are critical to ensure proper manufacturing and safe use. Overall, makers tended to provide documentation that required the least additional work from them, preferring easy to update websites over creating static documents, or adding easy to capture images instead of videos. Documentation did not appear to be a top priority for most makers.
In the medical domain, reproducible testing procedures and results are critical. Test results are needed to quantify the level of protection a design provides. Their importance to reviewers is supported by the correlation between presence of testing and community or clinical approval (\(\chi ^2 = 4.1, p \lt .05\)). Only 44 (8.5%) designs documented rigorous testing results: 6 face shields and 46 mask. A \(\chi ^2\) test reveals that affiliation with healthcare facilities or universities correlated with the presence of testing results (healthcare: \(\chi ^2 = 22.3, p \lt .001\) ; university: \(\chi ^2 = 21.4, p \lt .001\)). This is likely because testing requires specialized equipment that consumers cannot easily access. The community’s emphasis on testing advantaged affiliated makers over unaffiliated makers.
5.5 Convergence of Designs
Our dataset was characterized by rapid convergence of design ideas; there was little exploration of new forms of PPE. The COVID Collection was broad in its call for design, stating that it was created to “inform decision-making on PPE and medical device production, without stifling innovation”. Interestingly, the community who submitted to this collection narrowed its focus to the production of three types of PPE: masks, face shields, and ear-savers; 520 of the 623 total submissions (83.5%) fell into these three categories. The 103 “other” submissions focused on meeting a range of needs (e.g., ventilator parts, shoe covers, gowns, hand-less door-openers, nasal swabs).
We further saw convergence of designs within these three overarching PPE categories. Consider face shields. In our preliminary analysis of a random sample of face shields, the only common difference between the designs was protection from liquid droplets from above (Figure
6(a) and (b)). Besides this feature, face shields almost exclusively consisted of a 3D printed frame that braces against the forehead and a clear plastic sheet that attaches to the front of the frame to protect the face. The two most commonly remixed face shields (“3DVerkstan 3D printed face shield head band” and “DtM-v3.1 Face Shield PPE”) followed this archetype. The convergence to only a few archetypes over a period of about a month is unusual. Generally, makers are espoused for their creativity and presentation of novel, innovative, even wild ideas [
51]. However, this creative design divergence was largely absent from the NIH 3D Print Exchange.
There was one notable design for a combined face shield-mask that starkly deviated from this norm: the “Five-minute zero-print full-face snorkel mask with filter” (3DPX-013396), shown in Figure
6(c). It required no 3D printing and only attachment of filtering material over the spout of a full-face, sealed, snorkel mask. There were 13 other scuba-mask-based designs that all used the same concept but used a 3D printed adapter to attach the filter material. Across our entire qualitative review, this was the only archetype that varied significantly from a design that was reviewed for clinical use before the rapid drop off in submissions in April.
5.6 Safety
The review system is the core component that distinguishes the NIH 3D Print Exchange from any other maker repository. The process enforces clinical norms of safety and quality. Overall, the risks associated with different types of PPE was the primary determinant in review status. More subtle details that contribute to safety and quality were difficult to analyse because, to date, 81.3% of designs have not been reviewed. However, some traits that we expect contributed to a design’s safety-level could be found across the whole dataset. Though we are not experts in the safety of PPE, we identified three relevant safety traits through our analysis: coverage, fit, and the presence of cleaning instructions. The safety criteria for masks and face shields differ, and so we discuss them separately below. Ear-savers, on the other hand pose little risk
2 as an accessory to improve comfort, so we do not discuss their safety features.
5.6.1 Usage Ratings and Safety Results.
The NIH 3D Print Exchange created four different usage ratings to classify entries based on the prototype’s level of safety (Table
2). The vast majority of entries (N = 464/520, 81.3%) had a “prototype” status, which is the default rating of submissions uploaded, indicating that the submission has not been fully reviewed. Though not an official rating, we did note that 8.8% (N = 41/464) of these submissions had received notes from the reviewer which indicates that these submissions were not acceptable given the level of documentation included in the submission. The other three statuses dictate the level of trust reviewers had in the designs’ safety and efficacy. Design affiliation with healthcare correlated with both likelihood of receiving reviewer attention (
\(\chi ^2 = 11.4, p \lt .001\)) and community or clinical ratings (
\(\chi ^2 = 11.2, p \lt .001\)). This may indicate that affiliated makers were sought out for review and were better suited to submit designs that reviewers viewed favorably (i.e., considered safe).
A total of 29 entries (5.6%) received clinical usage ratings, meaning the entry had been evaluated in a clinical setting, and reviewers deemed it appropriate for healthcare workers in contact with COVID-19 patients—their highest mark of safety. For example, the “Stopgap Surgical Face Mask (SFM) Revision B” (3DPX-014168) was evaluated in a clinical setting and was given a clinical usage rating. Others (N = 25/520, 4.8%) received a community usage rating, meaning that the entry is suitable for workers in retail stores, law enforcement, and other community activities. Two entries (0.4%), both of which were masks, received a “warning” rating, indicating that the entry needed FDA approval or had design flaws that make it unsafe to use. Outside of our dataset of masks, face shields, and ear-savers, 5.5% of all submissions (N = 34/623) had a warning rating. A majority of these entries with warning status (N = 15/34, 44.1%) were ventilator parts. Many of these entries had notes from the author saying the entry had not been tested; for example, the author of the “Ventilator Circuit Splitters - reinforced & thicker walls” (3DPX-013347) stated that they “make no representations as to the safety of this device”. Other entries with the warning status included parts for other respiration devices and mask sanitizers. As we expected, classes of devices that pose more risk (e.g., masks, ventilator parts) received more scrutiny, and devices that pose less risks (e.g., face shields, ear savers) received less scrutiny.
There were 41 submissions that were not yet reviewed but had reviewer notes. The reviewer notes in a majority of these submissions (N = 31/41, 72.1%) requested documentation, specifically best printing parameters or use instructions. Some reviewer notes (N = 8/41, 18.9%) pointed out that the printing instructions and instructions for use were on external links and this documentation needed to be statically embedded in the submission to prevent modifications after review. Other reviewer notes (N = 4/41, 9.3%) requested that submissions be renamed so as not to imply incorrect usage and protection properties. For example, reviewers asked for “respirator” to be taken out of the title of the submission “3D Printed Respirator Mask, 4 sizes, XSM, SM, M, L” (3DPX-013948) because the term “respirator” is a medical term that implies a specific level of protection that this mask did not meet [
42]. Two reviewer notes on masks requested testing information. For example, the “The Unity Mask PRO”(3DPX-014364) listed that the mask met or exceeded
National Institute for Occupational Safety and Health (
NIOSH) N95 filtration criteria, but did not provide the test results. These reviewer comments suggest that makers’, particularly unaffiliated makers, and reviewers’ value of documentation were misaligned.
5.6.2 Characteristics of Safe Designs.
Based on the designs that we reviewed, we could only identify three characteristics of designs that we have a high confidence could influence whether or not their reproduction is safe for clinical use: (1) mask sizing/fit, (2) face shield coverage, and (3) the presence of cleaning/disinfecting instructions.
The main safety trait that varied across masks was the inclusion of different size 3D models. For masks, one size does not fit all; sizing is a key factor that influences fit and fit ensures safety. Facial features do not scale uniformly, so scaling model sizes is not a solution. Most masks will not create a secure air seal on a diverse set of human faces. In these cases, contaminated air could enter through the gaps between the mask and face, rather than through the filter. Different sizes are needed to ensure that people of different ages and genders are protected. Only 43 (25%) of the masks offered at least two sizes. In practice, users may find that different designs fit them better, but most wearers do not have the opportunity to print a range of masks and pick the best fit. The lack of sizing features in this dataset shows that most makers were not considering this key safety feature in their design process.
The main safety trait that varied across face shields was forehead coverage. Face shields at a minimum need to protect the front of the face (eyes, nose, and mouth) from liquid droplets, and almost all designs did so; two did not. However, several designs (192/263, 36.9% of face shields) also protected the wearer from liquid droplets from above by covering the forehead. Most designs either created a “visor” to connect to the top of the face shield or a closed gap so that there is no open space between where the frame touches the forehead and the clear sheet (see Figure
6). Forehead coverage may not be as critical as mask sizing for safety, but the additional feature demonstrates that many designers were considering increased safety when designing face shields.
A final piece of information that was critical to safe use of reusable PPE in a pandemic was cleaning instructions. Cleaning instructions are necessary to ensure proper disinfection and safe reuse. Only 84 (16.2%) submissions included cleaning instructions in their static documentation. A \(\chi ^2\) test reveals that affiliation with a health care organization or a university correlated with the presence of cleaning instructions (healthcare: \(\chi ^2 = 5.0, p \lt .05\) ; university: \(\chi ^2 = 11.8, p \lt .001\)), and the presence of cleaning instructions was correlated with a community or clinical usage rating (\(\chi ^2 = 33.3, p \lt .001)\). Many cleaning protocols are based on common protocols for medical devices that are already in clinical use. Notably, while cleaning instructions effected the review process, there was no submission field for including them explicitly.
Overall, there were only a few features that we could demonstrate impacted the safety rating of submissions. This may be because of the small sample of reviewed designs, making it difficult to identify common flaws in makers designs or characteristics of high-quality designs.
6 Discussion
The NIH 3D Print Exchange’s COVID-19 Collection is a unique open source platform because of its review process and focus on ensuring safety. Our study of the submissions made in 2020 helps us better understand how values of safety and openness impact maker communities. Prior work around medical making [
29] and PPE in the COVID-19 pandemic [
21,
33,
54] found that safety was a core value in these communities. However, other work does not show how safety was manifested in the final designs. Our work demonstrates that safety was upheld in the articulation work associated with each design (e.g., usage instructions, cleaning instructions, testing results). The NIH also conveyed this safety norm through the creation of a reviewing process, which is unique among maker platforms. We found that affiliated makers were more successful in creating clinically and community rated designs. They remixed designs more often as a group, while unaffiliated makers largely made small, manufacturing-focused alterations to a few affiliated designs.
As maker culture and open source design permeate more safety-critical activities (e.g., PPE production), we need to create tools that support makers in documenting, sharing, and customizing designs while upholding safety standards. Our work uncovers key questions and considerations in accomplishing these goals. Additionally, our work provides insights into the differences between affiliated and unaffiliated makers’ behaviors and suggests considerations for democratizing medical making to allow for broader participation amongst makers with diverse backgrounds, which is a challenging task [
35].
6.1 Creating Safety through Articulation Work
Open source repositories support knowledge sharing and help community members build off each others’ ideas. When safety is a top-priority for a community, this means that designs in the repository must provide all of the information necessary to manufacture and modify a design, while retaining its safety features. Articulation work (i.e., thorough documentation of a design) is the primary factor that determined if designs were marked as safe (e.g., clinical-review). Because affiliated makers come from communities that practice this form of safety-focused articulation work, they were often more successful at communicating how safe their designs are and received more positive reviews.
6.1.1 Documentation Indicates Safety.
Safety is a top priority in medical maker communities, such as those that use the NIH 3D Print Exchange. Prior work found that one of the main belief systems in PPE maker communities during the pandemic was rapid response or action, but not at the risk of safety [
21]. We found that safety was a top priority on the platform based on explicit solicitation for safety-centric designs and through the broad remixing of clinically rated designs that were more likely to be thoroughly documented.
Reviewers established a norm that if a design is not fully documented (i.e., not articulated well), it is not safe. The articulation work of documenting correct manufacturing and usage instructions was critical, and deviations from safety-critical steps could result in unsafe PPE (e.g., using the incorrect material, using corrosive cleaners). The NIH 3D Print Exchange explicitly stated that this clear articulation of the design and relevant procedures was valued: “Designs that have been marked “Reviewed for clinical use” are a great resource for understanding what good documentation looks like”, but specific standards that each piece of PPE had to meet were not listed, to our knowledge, on the site [
42]. Furthermore, reviewers prioritized designs that were thoroughly articulated, as demonstrated by the significant effect of the presence of testing results and cleaning instructions on the approval rating. While designs that did not include this articulation work could have been safe, these were critical factors that aided the reviewers in making safety-conscious and risk averse decisions.
6.1.2 Affiliation Affects Articulation.
Affiliation with a safety-focused community affected how makers articulated their designs and resulted in better adherence to the NIH 3D Print Exchange’s standards for safety, as demonstrated through higher usage ratings. However, makers outside those institutions did not readily prioritize this norm as evidenced by their lack of documentation (e.g., 16.2% of designs that had cleaning instructions, the 8.5% that had testing, and the 55.8% that had clear printing settings). Alternatively, our results suggest that makers who were affiliated with a university, healthcare facility, or for-profit company better shared the reviewers’ value of documentation. Presence of cleaning instructions and testing results was significantly more common among university and for-profit company affiliated makers. Similarly, affiliated designs received more reviewer attention and higher usage ratings, likely for more successfully adhering to the NIH’s value of safety, as expressed through their articulation work.
Affiliated makers had access to resources pools that aligned with the NIH’s clinical expectations of safety. For instance, interviews with makers working in close proximity to healthcare workers helped affiliated makers evaluate PPE usability [
28]. Additionally, they could access rare medical expertise (e.g., infectious disease teams). In terms of material resources, universities and for-profit companies often had resources like 3D printers, filament, and CAD and fabrication experts. Furthermore, many healthcare and university workers had access to testing facilities, which explains the statistical correlations between these affiliations and presence of testing results. Finally, affiliated makers had access to teams of specialized experts when attempting faster iteration; for example, the most popular and clinically approved mask had over 50 collaborators listed. Access to such resources is demonstrative of an institutional culture that supports and demands safety through documentation, a culture and infrastructure which many unaffiliated makers do not have access to.
6.1.3 Articulation Work is Required for Review Processes.
As thorough documentation of a design is critical to ensure safety, articulation work is required for reviewers to make accurate judgements of submissions. However, the NIH 3D Print Exchange’s website did not clearly list required submission elements that reviewers’ needed for review. While thorough documentation increased the likelihood of clinical or community usage ratings, details such as testing results and cleaning instructions were not listed in instructions or required in the submission form. These missing details led to wasted reviewer time, which was scarce to begin with. If designs received reviewer comments, they were often short: requesting additional details (e.g., printing instructions, testing results) or modifications to specific language that over-promised on the safety standards the designs met. Especially when contributors include people who are new to review processes, clear guidelines around what documentation to include that are built into the submission process can lead to more productive review cycles.
Outside of a pandemic context, safety is a high priority in many domains that attract makers (e.g., crisis-response). Thus, maker repositories focused on these domains should be designed in ways that support and prioritize safe making, which often includes sharing of documentation and safety information. Moreover, this case study demonstrates that in other making contexts that include a formal review process but also aim to broaden participation, platforms need to ensure that unaffiliated makers understand both the community’s priorities and the concrete practices (e.g., articulation work) that are key to achieving those priorities. Finally, platforms can consider how to communicate adjusted priorities (e.g., reviewers might change what is required for review overtime).
6.2 Advantages of Small Changes
Makers are often characterized as creative and resourceful people who come at a problem from unique angles, and the NIH explicitly defined a goal of not “stifling innovation” on the forum; therefore, in our analysis, we viewed diversity of the solution space as valuable. Indeed, in a context of failing supply chains, innovation around PPE devices, like face shields, benefited from having many minds from different backgrounds ideating on how to solve the same problem in different ways. However, in our dataset, we saw that makers largely converged on only three types of medical devices and proceeded to make many remixes to support more diverse methods of manufacturing. We now discuss the benefits and costs of the rapid convergence of the design space and divergence in manufacturing space.
There were two main phases where makers could innovate: designing the PPE archetype and developing different manufacturing methods. This distinction between how to best support co-designing and co-manufacturing is motivated by our case, and it is further explored in other literature [
38,
39]. The affiliation of makers affected which design process they are able to easily participate in: The development of the PPE archetypes was limited to affiliated makers, while unaffiliated makers focused their efforts on increasing the diversity of manufacturing methods.
Due to the nature of the pandemic and the specialized expertise needed in a medical making context, participation in the problem discovery and definition phase of PPE making was largely restricted to affiliated makers. This is valuable because affiliated makers have greater access to the resources and knowledge to efficiently design safe devices. Quickly converging on safe design archetypes was valuable in a crisis context. However, this pushed many unaffiliated makers out of the design ideation process even though most makers are attracted to this type of innovation. Indeed, in contemporary studies of a maker community with affiliated and unaffiliated makers, unaffiliated makers became discontent with their limited role in device design [
20].
On the other hand, unaffiliated makers often acted as PPE manufacturers during the pandemic rather than designers. As such, they tended to contribute remixes of well-rated designs created by affiliated groups. Their changes preserved the overall design while making small modifications to meet localized manufacturing needs (e.g., availability of printer filament) [
39]. For example, we found that many remixes were edits that made manufacturing a design more efficient or manageable on a broader set of consumer-machines. Thus, unaffiliated makers expanded the group who could produce PPE, which was valuable in a time of great shortage. At the same time, if these manufacturing changes were not made with care, they could risk the safety of the original design (e.g., using a more common filament with different safety properties). If those changes and their effects are not articulated well, the risks they pose may be missed by reviewers.
The effects of reviewing and the norm of safety affected who participated in what ways on the forum. The trends that arose on the NIH 3D Print Exchange suggest questions that should be considered when building open source platforms that support safety-critical domains. What types of designing and remixing behavior should be encouraged of all makers? Can unaffiliated makers create safe designs from scratch? If we think that large design changes can be safely achieved by unaffiliated makers, then how can makers be encouraged to document their work for review without discouraging makers with less resources or experience? Such questions pose opportunities to build new systems and communities for safety critical making and must be considered if we aim to reach the goal of encouraging broader participation in medical making and other safety critical domains.
7 Design Recommendations
The COVID-19 collection within the NIH 3D Print Exchange has been an experiment in online sourcing of community medical device designs. In many ways, this tool follows Lakshmi et al.’s recommendations to use “partially-open repositories” to collect, review, and regulate medical makers’ designs [
29]. We understand, from the repository’s statement, that it is intended to create an environment for purpose-driven contribution from amateurs and experts [
27], thereby increasing those who can participate in medical making. Other domain-specific maker-repositories can learn from the outcomes of this PPE-focused case study.
Based on our findings, we conclude there are gaps for medical making platforms to bridge in realizing this goal of broadening maker participation. We found that this iteration of the exchange was not able to review designs fast enough, and makers tended to submit risk averse designs rather than proposing novel and unique designs. While this tradeoff may be inevitable, we expect that a balance between novelty and review could be struck with interface variations that support and reinforce community norms like safety and innovation and providing extra support to makers unfamiliar with new procedures like a review process. We offer three technology-based design suggestions for repositories hosting medical making designs that lie at the intersection of collaboration across expertise groups (e.g., reviewers and unaffiliated makers) mediated by an open-source platform.
7.1 Provide Support for Understanding Review Criteria
A reviewing process is novel for maker repositories, and makers need clarifications to use it effectively. Confusion could minimized by including reviewer comments in accepted designs. These comments are needed on incomplete and accepted designs so that makers can differentiate between the two. Makers would benefit from information about what made a design safe or effective, and they needed this information to be more readily available (e.g., not listed in FAQ pages, but rather built into the submission workflow). An additional way to clarify requirements is to mark critical fields as mandatory for submission. However, the NIH 3D Print Exchange may have limited requirements in an effort to not overwhelm makers. As a compromise, we propose that fields be marked as “recommended for review” to communicate reviewer priorities with makers. This option would still highlight safety and documentation requirements without disenfranchising makers who prioritize sharing a design over achieving a usage rating. Reviewer time would be conserved since they could easily prioritize designs that have the details needed to replicate a design.
7.2 Prescribe and Support Documentation for Remixes
In its current form, the NIH 3D Print Exchange does not include a structure for annotating how a design was remixed. While this reduces burden of documenting changes, it also obfuscates “material” [
47] distinctions between designs. Without a way to describe changes, it is up to makers to add this information in unrelated fields or to omit it entirely. A medium to convey such changes could be as simple as a required “summary of changes” field for remixes or tools that better support the annotation of changes to 3D models such as elements of the physical structure, materials used, fabrication procedure, or use and care instructions. Moreover, requiring these details could conserve reviewers’ efforts by reviewing just the highlighted changes rather that re-reviewing derivative work.
7.3 Encourage Design Innovation
Though we witnessed manufacturing diversity, design archetypes on the NIH 3D Print Exchange rarely diverged from a few common forms. Instead, it seems that, while innovation and creativity were deemed important, safety was nonnegotiable. However, we expect two strategies can encourage innovation while ensuring safety. First, more diverse designs can be encouraged through explicit calls from the platform creators and makers (e.g., “build a better face shield”). Second, innovation can be rewarded along with safety, clarifying to makers that the two do not have to conflict. Safety was rewarded with clinical usage ratings. Similarly, we recommend that commendable innovation and creativity be noted in the review process. A “uniqueness votes” or “tags” system could encourage makers to explore new ideas. Reviewers could prioritize reviewing highly innovative designs over ones similar to reviewed designs. Since makers tend to modify designs with positive reviews this could have a snow ball effect where more makers remix designs that are increasingly divergent.
8 Limitations and Future Work
Our choice of methods and dataset limit the scope of findings for our study. First, this study looks at making PPE and related regulations and government agencies in the United States. While other countries may have similar agencies, future work could investigate how/which of these findings apply in other contexts. Second, while there are other maker repositories which hosted PPE designs (e.g., Thingiverse), we limited our focus to the NIH 3D Print Exchange to examine a reviewed, safety-focused space. Future work could investigate how the submissions on the NIH 3D Print Exchange compares to medical designs on other forums. Third, the main focus of our analysis was on the three main categories of design, which contained 83.5% of submissions. The sample size of any of the types of devices in the “other” category (e.g., nasal swabs) was less than the size of the three main types of submission (see Figure
3), and was too small to perform statistical analysis. However, these other submissions could be investigated further to understand how and why they were created. Fourth, our study looked at the resulting design artifacts rather than interviews, due to coverage in prior work [
6,
20,
21,
28,
29,
44]. While this gave us rich insights into how safety and review affect design artifacts, it does mean that we do not have explicit knowledge of reviewer and maker motivations outside of what they have posted in text on the NIH 3D Print Exchange and what has been learned in prior work. Fifth, we were limited by the information that was listed on the NIH 3D Print Exchange. Though many submissions contained external links, we chose to scope our project in this manner since it matched how reviewers reviewed submissions. This may mean that there is more complete documentation or affiliation information that was not included in our study. Finally, we treat the NIH as an authority on safety because we have no alternative at this time, but future work should explore the longitudinal effects of medical making in regards to safety, what constitutes safe medical making, and who makes these determinations.
Our work uncovered several questions around who contributes what in medical making infrastructures and how to support unaffiliated makers if they are to be more involved. Our work found that affiliated makers tended to do more of the design ideation while unaffiliated makers took these designs and performed manufacturing diversification. One problem that emerged in this workflow was that designs created by affiliated makers did not always encode clearly why design choices were made, including omitting if a design choice was safety-critical. Unaffiliated makers’ remixes of these designs then often lacked necessary documentation for the review process. Future work can investigate how to better encode design decisions in open source repository submissions and if technology can support unaffiliated makers in creating the thorough documentation needed for medical making.
Finally, we recognize that our work was affected by our own experiences. Three of the authors were deeply involved in making PPE and contributed to designs submitted to the NIH 3D Print Exchange. Though we have established relationships with the creators of the exchange, in this article, we only draw from publicly available evidence. As researchers in computer science fields our recommendations focus on the design of tools and interfaces, but we recognize that public policy determines what designs can be created and when and how they can be used. While we have engaged in wider discussions of policy [
3], they are out of the scope of this article. In particular, we have avoided making judgements about what makes designs safe or who should be doing this work. We leave such questions up to medical makers and suggest tools and interfaces that could bolster these critical conversations.
9 Conclusion
The NIH 3D Print Exchange houses 623 makers’ designs for PPE, the results of one of the most expansive efforts of medical making yet recorded. The forum was created to strike a balance between providing guidance through a formal review process and not stifling creativity. Our analysis of these 623 submissions reveals makers’ misconceptions about the review process and criteria which lead to a rapid convergence of the design space. A few key designs created by university, for-profit company, and clinically affiliated makers received clinical usage ratings. Following submissions, particularly those made by unaffiliated makers, were derivatives of these designs that focused on increasing manufacturability. Often these submissions made small changes to optimize or increase flexibility of manufacturing. Overall, few designs were reviewed, and several of the designs that received reviewer attention were missing key pieces of information that prevented full review for clinical use.
In summary, our results suggest that affiliated makers received more positive ratings and more reviewer time than non-affiliated makers due to the knowledge and practices they bring from their clinical work. At the same time, many makers, particularly unaffiliated makers, often left out key pieces of information from their design submissions, leading to wasted review cycles. To make a more efficient and understandable review process without stifling maker creativity, we make three recommendations. First, prioritize unique designs for review to provide more examples of divergent and safe designs. Second, pose explicit requests to the community calling for diverse design ideas. Finally, establish clear metrics of safety through review criteria. These changes aim to bridge the gap between the NIH’s goals and unaffiliated makers’ understanding of the review process and the values it implies.