Information correspondence between types of documentation for APIs

Abstract

Documentation for programming languages and their APIs takes many forms, such as reference documentation, blog posts or other textual and visual media. Prior research has suggested that developers switch between reference and tutorial-like documentation while learning a new API. Documentation creation and maintenance is also an effort-intensive process that requires its creators to carefully inspect and organize information, while ensuring consistency across different sources. This article reports on the relationship between information in tutorials and in API reference documentation of three libraries on the topics: regular expressions, unified resource location and Input/Output in the two programming languages, Java and Python. Our investigation reveals that about half of the sentences in the tutorials studied describe API Information, i.e. syntax, behaviour, usage and performance of the API, that could be found in the reference documentation. The remaining are tutorial specific use-cases and examples. We also elicited and analyzed six types of correspondences between sentences in tutorials and reference documentation, ranging from identical to implied. Based on our findings, we propose a general information reuse pattern as a structured abstraction to represent the systematic integration of information from the reference documentation into a tutorial. We report on the distribution of 38 instances of this pattern, and on the impact of applying the pattern automatically on the existing tutorials. This work lays a foundation for understanding the nature of information correspondence across different documentation types to inform and assist documentation generation and maintenance.

Appendices

Appendix A: Preprocessing Steps

In general, the following rules and preprocessing techniques for sentence extraction were adhered to:

• Remove HTML tags script, style, table

• Insert a comma after the tokens ‘e.g.’ and ‘i.e.’

• Insert a comma after the token ‘etc.’ if the word following this one began with a lower case alphabet.

• Replace multiple adjacent commas (occurring as a result of previous preprocessing steps) with a single comma.

• Replace newlines with spaces

• Replace multiple adjacent spaces with a single space

• Replace multiple adjacent periods (...) with a single period (.)

• In general, blockquotes, code blocks, images and the equivalents across the files were replaced by a single token BLOCKQUOTE, CODE and IMAGE respectively. These blocks were identified as being of a specific HTML tag type or having a specific HTML class.

• If a list item did not end in a period, the following item would be concatenated to the previous, separated by a semicolon.

• Finally, split on on a period followed by a space (‘. ’) and an exclamation followed by a space (‘! ’) to produce individual sentences

It is important to note here that inline HTML code tags in the sentence (inline and hence, did not involve line breaks) were maintained as is. Usually such pieces were names of the library or method being described. For example, “The java.util.regex package primarily consists of three classes: Pattern, Matcher, and PatternSyntaxException” [15].

Appendix B: Reasons for Similar Correspondences

The reason for a two matched sentences to be considered similar but not equivalent could be one of the following:

• The sentence is a rephrased version of two non-neighbouring reference documentation sentences. As a result, these sentences cannot necessarily be systematically identified and merged without advanced mechanisms to merge the sentences coherently, efficiently and favorably for the reader in a human-like writing style. For example, consider the highlighted sentence in from Java-I/O tutorial [26].

  

  The description in the reference documentation of newByteChannel method to which it refers mentions this information in two separate non-consecutive sentences, as highlighted in the figure below [23].

  

  Combining these sentences to generate a coherent sentence as in the tutorial is beyond the scope of our work.

• The sentence references or is in conjunction with a specific example. Sentences that provide example-specific information are marked as Supporting Text. However, sentences that provide general information about the API in the context of an example are considered to have similar matches. Python-I/O tutorial [14] contains one such instance:

  

  Our preprocessing steps result in the sentence extracted in the following format: “If you have an object x, you can view its JSON string representation with a simple line of code: CODE.” [14]

  The code snippet within this sentence as seen from the screenshot informs that the JSON string representation of an object x can be viewed using the dumps method. The description for the method in the reference documentation [27] states:

  

  Hence, replacing this tutorial sentence by its corresponding reference documentation sentence will result in a loss of the example which is integral for this sentence to provide useful information.

• It introduces a use-case for the reference API documentation. For example, in the Java-URL tutorial, the following sentence exists: “After you’ve successfully created a URL, you can call the URL’s openStream() method to get a stream from which you can read the contents of the URL.” [28] Here, the bold phrase describes when the openStream method can and should be used as opposed to the corresponding reference documentation that simply says: “Opens a connection to this URL and returns an InputStream for reading from that connection.” [25], describing what the method performs.

• The matched API sentence may be providing excessive technical information. For example, the Java-REGEX tutorial states “The regular expression syntax in the java.util.regex API is most similar to that found in Perl.” [15] On the other hand, the reference documentation goes into deeper details: “The Pattern engine performs traditional NFA-based matching with ordered alternation as occurs in Perl 5.” [18] A tutorial author might decide to omit technical details that the reference documentation contains which a reader referring to the tutorial would not be expected to benefit.

Appendix C: Reasons for Unmatched Sentences

We describe the reasons for unmatched sentences in tutorials in detail below based on the ten categories listed in Table 4.

The majority of unmatched sentences in Java-REGEX provide information about the underlying topic, usually describing the general behaviour of the fundamental concept behind the API. The definition and description of a regular expression, its syntax, the behaviour of special characters or definitions of related terminology are examples of this category that we found in tutorial but not in the reference. For example, the following sentence defines a set of methods having similar functionality: “Capturing groups are a way to treat multiple characters as a single unit.” [29]

We discovered that usage, i.e. general information on how an API is expected or intended to be used, is the most frequent category for unmatched in Java-URL, Python-URL, and Java-I/O. The Java-URL tutorial recommends how to handle a MalformedURLException: “Typically, you want to catch and handle this exception by embedding your URL constructor statements in a try/catch pair, like this: CODE.” [17]

In Python-REGEX, the most commonly unmatched sentence category is that of internal behaviour with 30% of the sentences describing such information. For example, the following sentence was found in the tutorial, but not reference documentation: “Regular expression patterns are compiled into a series of bytecodes which are then executed by a matching engine written in C.” [12]

A surprising 52% of sentences describe API behaviour in Python-I/O. It can be expected that descriptions of the way in which an API component performs is presented in the reference documentation, and so this finding is of interest. For example, the following sentence describes a particular behaviour of the read method on a file object: “If the end of the file has been reached, f.read() will return an empty string (‘’).” [14]

We found that sentences describing specific use-cases in which the API could be or is intended to be used, usually with the intention of motivating and justifying the usefulness of the API, were also not matched with reference documentation. This sentence from Java-REGEX is one such example: “The split method is a great tool for gathering the text that lies on either side of the pattern that’s been matched.” [16]

Sentences regarding performance of the API in terms of efficiency and scalability can also be observed in the tutorial, but their match could not be found in the reference documentation. The following sentence from Python-I/O is an example: “This is memory efficient, fast, and leads to simple code: CODE.” [14]

We observed that some sentences providing version information and backward compatibility were not matched in the reference documentation. One example of a sentence providing information regarding content of a particular version is this sentence in Java-REGEX: “As of the JDK 7 release, Regular Expression pattern matching has expanded functionality to support Unicode 6.0.” [30] This is a surprising finding because deprecation and enhancement information are generally specified in the reference documentation, in order to caution developers about no longer supported API components, or introduce them to new ones. We expect that this kind of information can be found in the version release notes and we leave the exploration of this documentation type to future work.

Some of our observations are unique to Java-I/O documentation. This, we theorize, is likely due to its large length and diverse range of sub-topics, providing greater scope for writing style variation. We found non-matched sentences providing environment and platform specific information, API support and input configuration details information only in this documentation. While describing the typical syntax of a file location, the documentation provides the following platform specific information: “In the Solaris OS, a Path uses the Solaris syntax (/home/joe/foo) and in Microsoft Windows, a Path uses the Windows syntax (C:∖home∖joe∖foo).” [31] Another sentence describes whether a file system may be able to support the API components provided: “A specific file system implementation might support only the basic file attribute view, or it may support several of these file attribute views.” [22]

Sentences containing input configuration details information are ones which describe the structure of the input to an API. For example, in the JAVA-IO tutorial, the width is an element of the format specifier in the format API. The sentence provides the following information about width: “By default the value is left-padded with blanks.” [32] The default behaviour of this element of the format specifier is not mentioned in the reference documentation.

We also identified one sentence describing a method for which the corresponding description in the API documentation was not descriptive enough to consider it as a match. While the tutorial states: “visitFile - Invoked on the file being visited.” [33] The description of the visitFile method in the reference documentation is simply: “Invoked for a file in a directory.” [34] While the sentences provide little explanation, the tutorial clarifies that this method is invoked when a file is visited as opposed to the reference documentation. We chose to treat this as an anomaly and not to categorize it as a separate unmatched category because of its low occurrence. Further, we found two more instances of descriptions in reference documentations that could have been matches for an tutorial sentence but were either incomplete or not clear in explanation. We consider both cases as implied matches because their meanings can be deduced given familiarity with the API. One example is shown below:

It is important to note that these categories are not exclusive to unmatched sentences. There may be sentences which are matched to reference documentation that provide information on these categories. We leave the detailed comparison of documentation at the category level to future work.

Appendix D: References to Documentation Sources

Below is the list of web URLs referenced in this paper. In the case of snippets of documentation used as examples, the corresponding URL defines the particular file in which the example text can be found.

• [1] https://docs.oracle.com/javase/8/docs/api/java/net/ServerSocket.html

• [2] https://docs.python.org/3/library/socket.html

• [3] https://docs.oracle.com/javase/8/docs/api/java/util/regex/package-summary.html

• [4] https://docs.oracle.com/javase/8/docs/api/java/net/package-summary.html

• [5] https://docs.oracle.com/javase/8/docs/api/java/nio/file/package-summary.html

• [6] https://docs.oracle.com/javase/tutorial/essential/regex/index.html

• [7] https://docs.oracle.com/javase/tutorial/networking/urls/index.html

• [8] https://docs.oracle.com/javase/tutorial/essential/io/index.html

• [9] https://docs.python.org/3/library/re.html

• [10] https://docs.python.org/3/library/urllib.html

• [11] https://docs.python.org/3/library/functions.html

• [12] https://docs.python.org/3/howto/regex.html

• [13] https://docs.python.org/3/howto/urllib2.html

• [14] https://docs.python.org/3/tutorial/inputoutput.html

• [15] https://docs.oracle.com/javase/tutorial/essential/regex/intro.html

• [16] https://docs.oracle.com/javase/tutorial/essential/regex/pattern.html

• [17] https://docs.oracle.com/javase/tutorial/networking/urls/creatingUrls.html

• [18] https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html

• [19] https://docs.oracle.com/javase/8/docs/api/java/net/URI.html

• [20] https://docs.oracle.com/javase/tutorial/essential/regex/matcher.html

• [21] https://docs.oracle.com/javase/8/docs/api/java/util/regex/Matcher.html

• [22] https://docs.oracle.com/javase/tutorial/essential/io/fileAttr.html

• [23] https://docs.oracle.com/javase/8/docs/api/java/nio/file/Files.html

• [24] https://docs.oracle.com/javase/tutorial/networking/urls/urlInfo.html

• [25] https://docs.oracle.com/javase/8/docs/api/java/net/URL.html

• [26] https://docs.oracle.com/javase/tutorial/essential/io/file.html

• [27] https://docs.python.org/3/library/json.html

• [28] https://docs.oracle.com/javase/tutorial/networking/urls/readingURL.html

• [29] https://docs.oracle.com/javase/tutorial/essential/regex/groups.html

• [30] https://docs.oracle.com/javase/tutorial/essential/regex/unicode.html

• [31] https://docs.oracle.com/javase/tutorial/essential/io/pathClass.html

• [32] https://docs.oracle.com/javase/tutorial/essential/io/formatting.html

• [33] https://docs.oracle.com/javase/tutorial/essential/io/walk.html

• [34] https://docs.oracle.com/javase/8/docs/api/java/nio/file/FileVisitor.html

• [35] https://docs.oracle.com/javase/8/docs/api/java/nio/file/FileVisitResult.html