Keywords

1 Introduction

While many modeling languages, in particular their visual notation, used to be designed by committees of experts, recent years have seen an increasing number of cases where notational design has been grounded in frameworks and theories for ensuring their quality. A particularly prominent theory over the last years has been the Physics of Notations (PoN) [7], for which it has been noted that the number of citations thereof has steadily climbed, while at the same time citations to other works have declined [2].

Several concerns regarding both the PoN itself, as well as its applications have been noted in literature (cf. [3, 10]). Given its popularity, and the tendency of its wielders in using the PoN’s scientific grounding as a justification for their notation being well constructed, it is important to ensure that this strategy indeed leads to designs suited to the notation’s users. Several principles of the PoN are concerned with cognitive factors that require tailoring to the specific group of users [14], such as for example whether a symbol suggests its meaning, or whether the representational medium is appropriate for its users and tasks. Because of this, involving users in applications of the PoN is important, as in vitro applications cannot provide such tailoring.

The seemingly obvious way to do so is to involve them in the design process of a notation and elicit any requirements they may hold, both directly and indirectly. In this ongoing research we investigate the thoroughness of applications of the PoN on several dimensions. To this end, we conduct a systematic literature review for analyzing all reported applications of the PoN. In this paper we focus on analyzing the dimension of user involvement, showing that very few applications of the PoN involve users in determining requirements for a visual notation. Concretely, we ask: RQ1. How many applications of the PoN involve notation’s users in determining requirements? RQ2. For what reasons do applications of the PoN not involve users? To answer these questions we focused on evaluating whether a paper’s application of (parts of) the PoN involves users in determining their requirements for the notation, i.e., if there is an explicit requirements phase involving (intended) users of the visual notation before or during iterations of the notation’s design phase.

2 Reviewing Applications of the PoN

2.1 Literature Review Protocol

In an ongoing effort on analyzing the quality of applications of the PoN, we have published a detailed research protocol and preliminary findings of a systematic literature review (SLR) in [13]. The SLR includes peer-reviewed articles and tech reports published by scientific institutions up to November 26th, 2015; the articles were found to have used the PoN if they either: applied the PoN theory or a part thereof to a visual notation, or discussed an application of the PoN theory or a part thereof. Articles were excluded if they did not apply or discuss (any part of) the PoN framework, were published in a language other than English, were unpublished theses, or were overlapping versions of already including work. Through the search criteria of papers citing the primary PoN publication [7] we gathered a list of 502 articles. This list was analyzed by per-year queries in order to select papers for inclusion in the SLR based on title, abstract and preliminary reading. The initial selection resulted in 41 papers, of which 4 were excluded after analysis of the full paper for lacking application of the PoN, which resulted in a final list of 37 selected papers, in line with expected amounts for SLRs focusing on technology evaluation papers [5].

2.2 Relevant Findings

Given the focus of this paper on user involvement, it is of note that the majority of the selected papers (\(62\,\%\)) reported on novel (versions) of visual notations, whereas the rest of the papers analyzed existing notations. With this number of papers designing new notations, one would hope for a large user involvement as well. Out of the thirty-seven analyzed papers, we found that only 5 of them involved users in setting or determining the requirements for the visual notation. Of those 5, only 2 reported any details, whereas the remaining 3 papers gave only a cursory mention of users. This means that out of all the papers we investigated, a mere \(14\,\%\) involve and report on users for determining a visual notation’s requirements. If we look specifically at only those papers reporting on novel or new versions of notations, only 3 papers reported to some degree on user involvement. Interestingly, papers that did not report on any involvement of users in requirements also often did not justify their use of the PoN: only in \(24\,\%\) of the papers investigated in our SLR was any kind of justification (even if little more than re-iterating its prominence) given for using the PoN.

3 Potential Reasons for the Lack of User Involvement

In this section we will discuss the potential reasons for this significant lack of user involvement as can be inferred from further reading of the selected papers.

3.1 Notation Designers Considered as the Prototypical User

Many papers that propose new notations for conceptual modeling are based on the expertise and experience of the authors themselves. This manifests primarily in choices for the use of symbolic and iconic representation that the authors find most suiting (e.g., using skulls as an icon on misuse diagrams to stress a negative connotation [9]). Given that there are several papers in which the primary affiliation of at least one author is non-academic, this is not necessarily a bad reason, as the notation designers can be either currently active in industry and thus take on the role of (implicitly) providing requirements. This is especially evident in a paper selected in the SLR focused on the creation of an architecture description language, which explicitly noted: “although some ADLs have been put to industrial use in specific domains [...] the majority of ADL projects remain confined to laboratory-based case studies.” [16]. The same authors criticized existing architecture languages such as ArchiMate for being “vertically optimized, limiting their attractiveness in many industrial projects.” [16] Thus, authors originating (or currently active in) industry might simply bring the requirements with them and by doing so chose to forgo explicit user involvement.

3.2 Users are Involved, but Afterwards

Some of the papers selected in our SLR seem to agree on the need for eliciting requirements from users and empirically validating that a notation is suited for them, but tend to do so a posteriori. For some notations this has taken the shape of empirical experimentation comparing a newly designed or enhanced notation with an older one to examine whether a measurable improvement in cognitive factors, like reading speed, can be found. For example, an application of the PoN to feature diagrams was followed up by a paper setting out to empirically validate suggested changes to the notation, indeed noting that: “Despite the proposed new notation being based on Moodys nine evidence-based principles, the proposed improvements will remain as mere heuristics unless they are empirically proven. Empirical validation is performed via two user studies ...” [8].

However, a concern with this reason for not involving users directly in the design of a visual notation is that this a posteriori validation aims at checking whether specific requirements are satisfied, but not whether some requirements were missed and not taken into account.

3.3 Requirements are Derived from Theory

Some papers specifically argue against user involvement, favoring theoretical grounding, primarily reasoning that using existing frameworks to reach an acceptable overall level of quality is more cost-efficient. For example, Thomas stated that “analytic techniques such as Moody proposes can help steer designs in the right direction and provide some preliminary evaluation, particularly in cases where it is envisioned that the proposed representation will be used by a wide variety of users in various contexts to reach various goals.” [11].

Many other papers selected in the SLR were not as explicit, but simply stated that the design of the visual notation was based on the PoN. This ranged from claiming to do so because of a proper grounding, e.g., “For the visual language proposed here, the PoN principles are applied because of their scientific and theoretical validity.” [4], to simply stating that the PoN was used, e.g., “In designing the visual notation we followed Moodys nine principles.” [6].

What seems critical here is that some of the principles of the PoN are inherently subjective due to their nature, and thus likely require user involvement. For example, the principle of semantic transparency states that shapes in the notation (i.e., symbols) suggest their meaning, which can be done by using iconic and pictorial representations. What meaning is suggested by symbols however, is strongly bound to personal and cultural factors, and thus cannot ‘just’ be given as a general rule. For this reason it is important to know how the (intended) users of the visual notation will think, and what connotations they will have to particular shapes and colors if one is to design for them without involving them.

3.4 Existing Theory is Assumed to Cover Requirements

In other cases it is also argued that the PoN covers requirements that can be expected to be elicited from users. A recent study investigated requirements posed by users of business process modeling notations, finding that “Interestingly a lot these non-functional requirements closely resemble the principles constructed by Moody. For example, the demand for descriptive, graphic elements corresponds to the ‘Principle of Semantic Transparency’. Furthermore, demanding non-redundant symbolic corresponds to the ‘Principle of Semiotic Clarity’.” [15].

Indeed, one could be tempted to assume that the PoN and its principles cover most requirements that would be elicited. A paper selected in the SLR [1] explicitly detailed 8 additional requirements next to the 9 PoN principles, but has strong redundancy between the two sets of requirements. For example “R15 Self-descriptiveness (diagram and graphical symbols describe their meaning themselves)” [1] maps quite clearly to the PoN’s principle of semantic transparency. An additional user-elicited requirement, “R16 Simplicity (graphical elements must be easy and fast to draw)” [1], while not directly mapped to an entire principle, is treated by the PoN’s principle of Cognitive Fit, in particular its prescription for adapting to the representational medium, ensuring that symbols are not made up of complicated shapes when needed to be drawn by hand.

Nonetheless, it is dangerous to accept that the PoN, as-is, covers all potential requirements, especially as new requirements might arise over the years with changing business and technology environments.

3.5 Cognitive Effectiveness is an Afterthought

While it is difficult to show that cognitive effectiveness is truly an afterthought in the creation of some visual notations, we feel that it is a valid and evident conclusion that follows from the low level of detail reported regarding the treatment of the PoN in some papers, which dedicate not more than a single line to the effect the PoN has had on the presented notation by stating the design ‘follows’, ‘was inspired by’, or ‘employs’ the PoN, especially if no information is given on any individual principles and how they are satisfied by the notation.

4 A Benefit of Involving Users in Notations Design

So far we have shown that many applications of the PoN do not involve users in setting requirements for the visual notation, that when they do sometimes it leads to redundant requirements, and that in some cases authors even argue against user involvement. Nonetheless, we see a main lesson to be learned from this to the benefit of cognitive effectiveness efforts employing the PoN.

By eliciting requirements from (intended) users of the visual notation to be designed, even or especially when those requirements overlap with the Principles of the PoN, useful information about the relative importance of those principles is gained. One of the shortcomings of the PoN is that there is not a clear guideline or mechanism for attributing weights to the individual principles for an overall analysis, as particular principles may become more or less important for some tasks. In the context of the PoN it has explicitly been discussed how certain principles affect each other [7]. For example, attempting to increase the semiotic clarity of a notation can simultaneously reduce its graphic economy by leading to an increase of graphical symbols. Finding a balance between how ‘satisfied’ the principles are is a complicated task requiring information about what is more important. This can be achieved by eliciting requirements from users, seeing which are most often stated, and comparing them to the PoN’s principles.

This co-incidentally also eases the difficulty of having to deal with creating a workable notation based on a multitude of requirements due to the notations use by many different people, in different contexts and for different tasks that Thomas [11] described. Instead, as many requirements as possible can be elicited, and compared to the PoN principles in order to personalize the PoN by changing the weighting of the different principles for the group of users and, where necessary, even giving rise to different dialects of a notation when particular requirements are favored for a particular task or modeling context. Thus, explicitly involving users and their requirements for the notation does not come in place of the PoN, but actually complements and strengthens its ability to lead to cognitively effective visual notations.

5 Concluding Outlook

In this position paper we have discussed some initial findings from a systematic literature review on the use of the PoN theory. We showed that very little papers applying the PoN to design or analyze of a visual notation explicitly involve users in setting the requirements for that notation, and discussed potential reasons why there is such a lack of user involvement. Finally, we made an argument that contrary to some arguments against user involvement, explicitly eliciting requirements from users does not have to be in place of the PoN, but can actually strengthen its use by allowing one to personalize relative importance and weighting of the PoN principles to the audience of the notation.