Task estimation for software company employees based on computer interaction logs

Abstract

Digital tools and services collect a growing amount of log data. In the software development industry, such data are integral and boast valuable information on user and system behaviors with a significant potential of discovering various trends and patterns. In this study, we focus on one of those potential aspects, which is task estimation. In that regard, we perform a case study by analyzing computer recorded activities of employees from a software development company. Specifically, our purpose is to identify the task of each employee. To that end, we build a hierarchical framework with a 2-stage recognition and devise a method relying on Bayesian estimation which accounts for temporal correlation of tasks. After pre-processing, we run the proposed hierarchical scheme to initially distinguish infrequent and frequent tasks. At the second stage, infrequent tasks are discriminated between them such that the task is identified definitively. The higher performance rate of the proposed method makes it favorable against the association rule-based methods and conventional classification algorithms. Moreover, our method offers significant potential to be implemented on similar software engineering problems. Our contributions include a comprehensive evaluation of a Bayesian estimation scheme on real world data and offering reinforcements against several challenges in the data set (samples with different measurement scales, dependence characteristics, imbalance, and with insignificant pieces of information).

Appendices

Appendix A: Details of Ground Truth Annotations

Table 14 depicts details on ground truth annotations regarding the two subjects. This table indicates that both the developer and the leader have quite an imbalanced distribution of tasks. Namely, the developer carries out a Test 82% of the time, whereas the leader performs Documentation 78% of the time. In addition, Test and Administration are found to be performed by both subjects, but at different rates, whereas Programming is carried out only by the developer, and Leisure and Documentation is realized only by the leader.

Table 14 Distribution of the ground truth labels assigned by the coder

Appendix B: Details of Association Rules and Key Phrases for Window Titles

Table 15 depicts the entire set of association rules defined by the expert. Please note that rule numbers simply refer to the row number in the table and do not have any significance from a semantic point of view. Similarly, also the order of candidate tasks does not have any significance. Note that in the main track we provide some sample rules after translating their window titles to English, for the sake of clarity and uniformity. Here, we provide them in their original form, i.e. including Chinese and Japanese characters.

Table 15 The set of association rules defined by the expert

It can be seen in Table 15 that out of the 20 rules, 13 of them are definite and 7 are indefinite. Moreover, only two rules involve both application and window title in their antecedents, while 12 rules involve only application, and 6 rules involve only window titles as antecedents. The indefinite rules with a single term in their antecedents (e.g. rule 15), can be considered to pose the most serious risk from the point of view of (un)certainty of estimations.

In addition, as mentioned in Section 5.3.1, since there is virtually an infinite number of window title possibilities, we only consider a set of 50 key phrases, which commonly appear in window titles. In this study, the set of key phrases is defined by our expert. Nevertheless, this step can be also performed automatically, for instance, by integrating a standard clustering method integrated with natural language processing tools. Specifically, the expert provided a list of 50 key phrases, some of which are redundant. Namely, among the 50 key phrases defined by the expert, 10 key phrases are utilized by the developer and 25 key phrases are used by the leader. In addition, some key phrases appear in the same window title.

To solve this issue, we revised the set of key phrases by considering those that appear together as additional variable values. In that manner, we consider the developer to utilize a total of 11 distinct window titles and the developer to utilize 31 distinct window titles (aside from the alien titles).

Appendix C: Details of Estimation Performance Upon Direct Application of Association Rules

Table 16 demonstrates the number of actions, which receive 0$\sim$3 candidate tasks by direct application of the rules. These results indicate that the rules are able to give a single candidate task (i.e. estimation) for only 152 actions out of 1283 (i.e. 12% of the cases) regarding the leader, and 131 actions out of 1921 (i.e. 7% of the cases) regarding the developer. This means that the remaining 88% and 93% of the actions of the leader and the developer, respectively, need to be revised such that a -single- task is assigned to them.

Table 16 The number of actions estimated to have 0$\sim$3 candidate tasks by direct application of the rules

Appendix D: Details of Post-processing of Benchmark Method

Step-1 of post-processing:

If two subsequent actions are uncertain and have a single candidate in common, then their estimated tasks are determined as this candidate.

Algorithm 1 outlines Step-1 of post-processing as a pseudo-code and Table 17 illustrates its execution on a sample -hypothetical- piece of data¹⁹.

Table 17 Example for the application of Step-1 of post-processing

Step-2 of post-processing:

For each estimated action, the task associated with it is propagated to its preceding (not-estimated) action, provided that that task appears among the candidates of the preceding action.

Algorithm 2 outlines Step-2 of post-processing as a pseudo-code and Table 18 illustrates its execution on a sample -hypothetical- piece of data.

Table 18 Example for the application of Step-2 of post-processing

Step-3 of post-processing:

Step-2 is repeated beyond immediately preceding and succeeding actions, until it reaches an estimated action. Subsequently, the same is applied for the succeeding actions.

Algorithm 3 outlines Step-3 of post-processing as a pseudo-code and Table 19 and Table 20 illustrate its execution on a sample -hypothetical- piece of data with backward and forward propagation, respectively.

Table 19 Example for the application of Step-3 of post-processing

Table 20 Example for the application of Step-3 of post-processing

Step-4 of post-processing:

For each estimated action, the task associated with it is propagated to each preceding (not-estimated) action irrespective of their candidates, until an estimated action is found. Subsequently, the same is applied for the succeeding actions.

Algorithm 4 outlines Step-4 of post-processing as a pseudo-code and Table 21 illustrates its execution on a sample -hypothetical- piece of data (with backward propagation).

Table 21 Example for the application of Step-4 of post-processing

Appendix E: Details of Performance After Causal Post-processing

Table 22 demonstrates estimation accuracy of the benchmark method together with causal post-processing operations. The most frequently occurring task of the developer (i.e. Test) is achieved with a high accuracy (i.e. 0.92), raising his overall estimation accuracy (i.e. 0.86). On the contrary, for the leader, the most frequently occurring task (Documentation) is detected only by 0.58, leading his overall accuracy to be only 0.63. In addition, from Table 22, it is clear that there is a larger margin of improvement in estimation of the actions of the developer (see also Table 6).

Table 22 Performance of the benchmark method after causal post-processing for (a) developer, (b) leader and (c) both

Appendix F: Details of Benchmark Performance After Non-causal Post-processing

Table 23 presents the estimation accuracy of the benchmark method followed by non-causal post-processing. It can be observed that non-causal post-processing achieves a higher accuracy both for the developer and the leader. Namely, for the developer the accuracy increases from 0.86 to 0.92 and for the leader it increases from 0.63 to 0.85. making non-causal post-processing more beneficial for the leader.

Table 23 Performance of the benchmark method after non-causal post-processing for (a) developer, (b) leader and (c) both

Appendix G: Statistical Properties of Descriptor Values

Table 24 presents the minimum, maximum, mean and standard deviation values relating the four ratio scale descriptors, i.e. duration δ (in sec), and number of key strokes κ, left clicks c_L and right clicks c_R. Although the tasks of the developer and the leader are distributed in quite a different manner (see Table 14), no significant distinction is present between descriptor statistics of the two subjects. In other words, the variation between the mean value of any ratio scale descriptor is within a single standard deviation concerning either of the subjects.

Table 24 The minimum, maximum, mean and standard deviations for the four ratio scale descriptors

Appendix H: Details of Normalized Entropy Distances

Table 25 presents normalized entropy distance values computed separately for each user. This table ascertains that there is a higher degree of dependence (i.e. lower distance) between application α and window title ω for both subjects.

Table 25 Normalized entropy distance between pairs of descriptors regarding (a) the developer (b) the leader at Stage-1 of hierarchical classification

Here, it is noteworthy to discuss the possible drawbacks of having a 2D variable space instead of two 1D spaces, due to the obvious dependence of α and ω depicted in Table 5 and in more detail in Tables 25 and 26.

Table 26 Normalized entropy distance between pairs of descriptors regarding (a) the developer (b) the leader at Stage-2 of hierarchical classification

Clearly, building a 2D space introduces a larger number of bins (i.e. cells) than two 1D spaces.

However, taking a closer look at the distribution samples in the 2D variable space, we can actually confirm that this increase in number of bins does not pose a serious issue for our particular data set. Namely, the developer uses a total of 11 applications and utilizes 10 distinct window titles (including the alien titles), which leads to a variable space with 110 bins. Among those 110 bins, only 14 are observed to have nonzero observations. Since the number of samples relating the developer is 1921 (see Table 14 of Appendix Appendix), we consider the sample size to be sufficient to populate the effectual 2D variable space.

On the other hand, regarding the leader, there are 16 applications and 31 window titles (including the alien titles), which lead to a 2D space with 540 bins and 36 of them have nonzero samples. Since the leader has 1283 samples (see Table 14 of Appendix Appendix), similar to the case of the developer, we consider these to be sufficient to populate the effectual 2D variable space. Therefore, although the contribution expected from dimensionality reduction is not regarded to be high for the particular data set of interest, we still consider this attempt to understand the dependency and our strategy based on normalized entropy distance, to be potentially beneficial for other data sets and to expand the application capabilities of the proposed method.

Appendix I: Pre-processing of Ratio Scale Variables

The pre-processing operation applied on ratio scale variables is explained in Section 5.3.1. Figure 3 illustrates an example of that pre-processing operation by presenting its details on the δ descriptor relating the developer.

Fig. 3

An example for the pre-processing of the δ variable concerning the developer

The quantiles Q_i of that descriptor are determined by following the procedure summarized in (6) and are illustrated on the x-axis of Fig. 3 (and listed in column 2 of Table 27). Moreover, the number of observations in each of these clusters are as given in column 3 of Table 27. As explained in Section 5.3.1, since the observations are substantially imbalanced, the number of observations in these quantiles does not approximate a uniform distribution.

Table 27 Cluster edges and number of observations in each cluster for the δ variable of the developer

Appendix J: Assessing Variables’ Relevance

Section 5.3.2 elaborated on the independence of pairs of variables, whereas this section investigates the correlation of variables and tasks. It is known that automated collection tools

may register too much and too detailed information requiring a serious amount of time for the analysis (Karahasanović and Heim 2015). Namely, certain descriptors may be highly correlated with particular tasks, while some others may not present a distinction with respect to the nature of the task. In order to assess the relevance of each descriptor in task estimation, we use Cramér’s V, which is a measure of association between two variables based on Pearson’s χ² statistics (Ramachandran and Tsokos 2014) and attains a value in the range [0, 1]. In explicit terms, Cramér’s V is computed as,

$$ V=\sqrt{\frac{\chi^{2}/n}{\min(k-1,r-1)}}, $$

(7)

where χ² comes from Pearson’s test, n is the number of observations, and k and r are number of values that each variable can attain²⁰.

As seen in Table 28, α and ω have by far the highest V values. Therefore, considering the significant difference between the values of V regarding these two descriptors and the remaining ones, we suggest using α and ω in task estimation and omit the others. In this respect, in this work we

Table 28 Assessing the relevance of descriptors with Cramér’s V s

prefer using V for measuring the degree of correlation and determine our set of relevant descriptors in rather an empirical manner.

In addition to assessing the relevance , we used Cramér’s V also for determining the optimum number of quantiles which yields the highest correlation between the relevant descriptors (chosen as above) and tasks. Namely, we compute V by distributing the ratio scale variables into 1 ≤ q ≤ 6 quantiles and pick the optimum value of q, which yields the highest V for each variable (see Appendix Appendix). In this respect, Table 28 presents the highest values of V for each ratio scale variable.

We compute Cramér’s V for the set of tasks and each single descriptor, which has been pre-processed as described in Section 5.3.1.

10.1 J.1 Cramér’s V for Nominal Scale Variables

As mentioned in Section 3.1, the nominal scale variables are application name α and window title ω. Computing Cramér’s V accounting for the considerations explained in Appendix J, we achieved the values presented in Table 29. We observed from this table that V relating α is higher than V relating ω for both subjects. In addition, the difference in V between α and ω is larger for the developer than the leader. For instance, at Stage-1 the developer attains 0.55 and 0.26 (with a difference of 0.29) for the two descriptors, whereas the leader attains 0.83 and 0.65 (with a difference of 0.18). All tasks are correlated more strongly with α, and the effect is more pronounced for infrequent tasks (i.e. Stage-2). Moreover, V relating ω is lower for infrequent tasks (i.e. Stage-2) for both subjects (i.e. 0.26 > 0.15 and 0.65 > 0.34).

Table 29 Cramér’s V regarding nominal scale descriptors

10.2 J.2 Cramér’s V for Ratio Scale Variables

In practice, there are a number of points that we need to be careful in assessing the relevance of ratio scale variables. Specifically, unlike nominal scale variables, relevance of ratio scale variables is assessed based on their empirical cdfs rather than the -raw- values. In that respect, we first need to carefully decide on the bin size of the empirical distributions, since the choice may impact on Cramér’s V.

The essential challenge in choosing the bin size relates the concentration of descriptor values in narrow ranges as mentioned in Section 3.3. One important factor leading to this issue is that the number of key strokes κ and left/right clicks c_L, c_R, attain a value of 0 for most of the actions. Thus, in building their empirical distributions (i.e. histograms), it is plausible to consider the case of having a zero value on its own. Namely, we first distinguish zero values, and use a bin dedicated to them and group all non-zero values in one cluster. This corresponds to the Q = 0 case in Tables 30, 31 and 32 below. Besides, any number Q >= 1, indicates that the nonzero values are grouped among them (into Q + 1 clusters).

Table 30 Cramér’s V regarding ground truth tasks and ratio scale variables clustered into various quantiles for (a) the developer and (b) the leader and (c) both, when no hierarchy is considered.

Table 31 Cramér’s V regarding ground truth tasks and ratio scale variables clustered into various quantiles for (a) the developer and (b) the leader and (c) both at Stage-1 of hierarchical classification.

Table 32 Cramér’s V regarding ground truth tasks and ratio scale variables clustered into various quantiles for (a) the developer and (b) the leader and (c) both at Stage-2 of hierarchical classification.

Another impediment is that we cannot distribute the descriptor values into any number of bins. Specifically, given a particular Q, it may not always be possible to have distinct edges for each cluster. Namely, some quantiles may be the same, if the distribution is too imbalanced (i.e. too many values in the first few bins and too little values towards the tail). In that case, the method explained in Fig. 3 prohibits using that particular number of quantiles Q.

The final issue relates the number of clusters that we can possibly use, in particular for integer variables. Namely, the number of clusters depends on the number of distinct observations for such variables. This can be understood easily by watching, for instance, the c_R values illustrated in Fig. 4g and h of Appendix H. The number of right clicks is either 1 or 2 for the developer, whereas it is some value between 1 and 7 for the leader, although values higher than 2 are observed quite rarely. From this figure, it is clear that we cannot categorize the values in more than 2 bins for the developer and more than 7 bins for the leader (though a large number of categories are not expected to introduce significant information)²¹.

Fig. 4

Histogram and cumulative histogram of duration δ for (a) the developer and (b) the leader. Similar pairs of figures for (c, d) number of key strokes κ, (e, f) number of left clicks c_L, (g, h) number of right clicks c_R.

When interpreting the information presented in Tables 30, 31 and 32, we should note that if a certain descriptor is correlated with a particular task for one subject, but with another task for the other subject, computing Cramér’s V for both subjects at once yields a lower value than computing it for each subject on its own. As an example, consider that, for the developer, firefox.exe is highly correlated with Test, while for the leader, it is highly correlated with Leisure. This means Cramér’s V can be -relatively- high for α computed for one subject only (i.e. only the developer or only the leader). But if it is computed for both subjects at once, it decreases, since α is associated with a larger variety of tasks.

In the light of these observations, we decided to use Cramér’s V as a guideline to determine the number of clusters of empirical distributions in addition to deciding the relevance of variables.

To determine the best number of clusters such that the correlation between the descriptors and tasks is highest, we compute Cramér’s V for the number of quantiles 0 < Q < 5 as presented in Tables 30, 31 and 32. Subsequently, we pick Q, such that it yields the highest possible V (marked in boldface in these tables).