Keywords

1 Introduction

The purpose of this study is to analyze the usability of Interaction Design Association’s website ixda.org [1]. IxDA is a free membership based group for professionals and enthusiasts in the field of interaction design. Two methods were applied; first one being the System Usability Scale (SUS) and the second one is an expert review guided by the Expert Review Checkpoint workbook. SUS consists of ten questions that cover different aspects of a system’s usability as perceived by the user with a possibility of five answers from strongly disagree to strongly agree. It was developed 29 years ago and since it has become the questionnaire of choice for various software, hardware, websites and other “systems”[2].

The SUS test can be customized for a specific task within the system, where the user fills out the questionnaire and is then interviewed about his experience. Since this simple test is meant to evaluate the system’s overall usability, giving instructions to the user interferes with the goal that John Brooke, the SUS creator had in mind when labeling it “quick and dirty”. This kind of adjustment is more applicable during agile website development, where specific functions of the website are tested as they are coded. This method protects against major mishaps during development but in most cases doesn’t replace usability testing of the website as a finished product.

Another type of traditional or discount usability test [3] is the Rapid Iterative Testing and Evaluation (RITE) method that was championed by Dennis Wixon and used on PC games. During this iterative usability method changes to the system occur as soon as issues are reported by a participant and a solution has been identified by the usability research team. The now updated prototype is being tested by the next participant until the system is deemed usable. These newer tests are meant not just to assess usability of a system but also to discover specific problems [4].

Other types of tests that discover specific problems are the expert review and heuristic usability evaluation. These tests use guidelines about user interface design based on principles established by usability authorities such as ISO, Jakob Nielsen and Don Norman. There are quite a few collections of guidelines for user interface design in existence that contain anywhere from few to 944 guidelines. To ease evaluation, sometimes the guidelines are presented as sets of questions grouped under specific usability theme. One such set of web usability guidelines was created by David Travis of Userfocus under the name of Expert Review Checkpoint [5]. These guidelines are to be used by an expert reviewer during the evaluation of a website's usability to discover specific problems and offer tangible solutions.

2 Analysis

2.1 SUS Method Analysis

Five PhD students at The Hong Kong Polytechnic University School of Design browsed ixda.org for on average fifteen minutes and then filled out the System Usability Scale (SUS) test. This number of study participants meets the ideal number of five test users as established by Jacob Nielsen [6]. According to him, testing more people will just yield similar results at a significant loss of time. There was no specific goal presented to the test subjects. They were just to look up an informational website of a professional group meant at promoting the field of interaction design. Most users will browse ixda.org for usability information to add to their current knowledge, search for conference opportunities, look up professionals who might become a potential contact, or search job leads if in the market.

Each of the user’s response on the SUS questionnaire is valued from 1 to 5. Every odd question is graded at user’s response minus 1 and every even question gets the score of 5 minus user response. So each answer can get the score from 0 to 4. Once all scores are added they are multiplied by 2.5 in order to have a range compatible with from 0 to 100. This doesn’t create a percentile scale, where 100 presents the best. Interpretation of the score is based on mass studies that suggest a score of 68 to be an average, meaning 50 % of all websites are at this level of usability [7]. Without any gouging, the result of this SUS test was exactly 68 points (Fig. 1). According to the above references study by Sauro J., the conclusion is that ixda.org doesn’t set up an example of good usability nor it is severely unusable. Other researchers such as Tullis T. and Albert B. report a larger variation in the SUS results and suggest interpretation for a score of below 60 % as relatively poor usability and for a score above 80 % as a pretty good usability [8].

The SUS test contains questions that imply use of a system for a specific purpose, where the user is performing an on the job task with its help. When browsing websites, especially ones set up by volunteer base professional associations there is no specific task at hand. Therefore the user cannot “feel very confident using the system” (question 9) when she doesn’t know what to expect from the system. A professional website is to incite interest in its content, prolong browsing by clicking on its tabs and links, and eventually create desire to join the group or to contribute to its content by commenting on a post. Therefore, a more sophisticated design and engaging content is necessary in order to render this kind of a website “sticky” [9].

Fig. 1.
figure 1

Line chart of SUS scores for ixda.org

Full size image

2.2 Expert Review Checkpoint

The Expert Review Checkpoint test is an expert usability test consisting of 247 questions subdivided into nine themes or categories that are indicative of a certain aspect of a webpage’s usability. The first theme addresses the website’s Home Page and consists of 20 questions, while the last one is about Help, Feedback and Error Tolerance and contains 37 questions. The expert reviewer starts by filling in the fields next to the statements about specific website characteristics with 1 for yes, 0 if it is met half way and −1 if the characteristic is not there. If some criteria don’t apply to that specific website the reviewer should not grade it. There is also a comment section to the right of the checkpoints for noting specific issues. This section comes in handy later on when writing the actual usability review. The results are then automatically calculated and summarized in a table by categories with scores listed for each category including the overall score. Both the table and a spider web graph of the results are created on the Results Page.

The overall usability score was calculated at 70 %, which is close to the 68 % derived from the SUS test. The Expert Review Checkpoint creator Travis D. in our e-mail conversation from February 13, 2015 expressed skepticism about the proximity of the two results stating: “the margin of error on each score is probably larger than 2 %”. He further stated: “I think you should use the scores only as a comparative measure, not as an absolute statistic. A web site that scores 80 % will probably be better than one that scores 60 % (assuming the same reviewer).” The Expert Review Checkpoint overall score can be validated by having few expert reviewers fill in the checkpoint for the same website. But this is also very costly and contradictory since the main purpose of The Expert Review Checkpoint is to assist with an expert review and not as a freestanding test. It is up to the expert reviewer to determine the checkpoint’s shortcomings and to alter and interpret this test according to the website’s specifics (Table 1 and Fig. 2).

Table 1. Summary of results table from Expert Review Checkpoint for ixda.org
Full size table
Fig. 2.
figure 2

Summary of results chart from Expert Review Checkpoint for ixda.org

Full size image

2.3 Expert Review

The Expert Review Checkpoint starts with 20 questions about ixda.org’s Home Page and this theme received the lowest score of 35 %. Two questions are in regards to its value proposition, one about it being clearly stated with a welcome blurb and another about the user understanding how valuable this website is. While the value proposition is well written and follows a standard template [10], one can only get so impressed with the home page considering there is a broken picture link underneath the value statement. The link does take you to the local leader spotlight as stated in the alternate text but the picture isn’t viewable. In its center, the homepage streamlines user experience by dividing users into two categories: those that are new to design and seasoned IxDA pro users. Left column contains links to jobs, school, tool and design techniques and a right column contains info about keynote speeches and awards.

The left column content is from 2007, what gives an impression that this website isn’t maintained on a regular basis. The right column content is recent, from 2014 and suggests the opposite. Nevertheless, the content under both categories fails to represent the wealth of information that one can access within this website’s ornate structure. Ixda.org’s homepage contains six menus, most of them cascading into few more submenus and sub submenus. The latter are not accessible unless we are on the corresponding page. Use of dropdown menus would have made it possible to see all the pages from the main menu [11]. On most pages there are hyperlinks that lead to pages with even more hyperlinks that can take the user to a very remote place.

The Checkpoint statement sometimes isn’t site specific and the reviewer is puzzled about how to mark it. For example “the home page contains a search input box” received a 1, even though two search boxes on the homepage speak of bad design. Same rule against repetitive content applies to having a job board menu item and IxD jobs tile on the homepage where both link to the same content. The issue repeats with having a discussions menu in addition to IxDA Discussions tile. Another quality criteria that’s not met is “navigation choices are ordered in the most logical or task-oriented manner (with the less important corporate information at the bottom)”. IxDA.org’s menus are not in a logical order. Discussions usually don’t get a menu tab and especially not one right next to the logo. All items under this tab have the discussions.ixda.org web address, whereas ixda.org/discussions follows the right naming convention. On the discussions page the search box at the top right corner is now replaced with a loupe symbol under the header right. The resources menu item was left out from the discussions page, making it inconsistent with the homepage.

Next to Home Page is the Task Orientation theme structured around the task of purchasing products on ixda.org . Even though e-commerce is not its business model, ixda.org has few products to sell. As per this quality “when graphs are shown, users have access to the actual data (e.g. numeric annotation on bar charts)” it is only partially met. Under the Local menu there is a Network submenu containing a Google world map with electronic pins. Clicking on a pin corresponding to a local chapter is not very intuitive and the small pin size makes it difficult to pick the correct location. There is also an option to use the local directory filter placed in the right sidebar. One discovers that most chapters have been inactive for few years and their organizers don’t have a contact e-mail address listed. Next to the Network submenu there is a Directory submenu with all local groups listed and Events submenu with only one future event listed. The same information repeats under each of the sub menus, in different parts of the page. Inclusion of three submenus under Local is unnecessary and confusing to the user. Therefore the guideline of “information is presented in a simple, natural and logical order” received a minus one, for the inclusion of unnecessary and poorly executed submenus. Obviously, the “less is more” principle established by Nielsen J. was not applied during this website design [12].

Under the Navigation and IA theme, few questions address the existence of a sitemap. A forum on www.ixda.org questions the need of a sitemap inclusion on a webpage but this important debate wasn’t on the discussions page but on this page http://www.ixda.org/node/24433 accessible only through a hyperlink. Opinions are divided on this issue and some suggest that a sitemap only helps SE indexing. However, when analyzing a website a sitemap is very helpful. Otherwise one will assume that ixda.org was built and maintained from multiple sources and this made it impossible to have a sitemap. The Conan Design document accessible from page http://www.ixda.org/node/21287 contains a collection of wireframes and a sitemap at the end of the proposed improvements for ixda.org . In the words of Elizabeth Bacon, IxDA’s Vice President and Conan lead from 2009: “This material represents the fruits of over 10 months of work from the all-volunteer IxDA Conan team. We investigated our community’s needs, developed user experience requirements, conceived interaction design and visual design solutions, and conducted a concept feedback round” [13]. The Conan document features the principal design as a foundation for the expected iterative and collaborative future development. Growing Venture Solutions did the ixda.org coding in Drupal.

The statement “there is a site map that provides an overview of the site’s content” received a score of zero, because the sitemap doesn’t correspond exactly to the current layout of this website. The next statement “the sitemap is linked to from every page” received a score of minus one since the sitemap is hidden inside a .pdf document and not as a separate page linkable within the website. Finally, the question “the sitemap provides a concise overview of the site, not a rehash of the main navigation or a list of every single topic” received a minus one, due to its inconsistency as compared to the actually website.

Navigation and IA received a somewhat low score of 52 %, mostly due to slight inconsistencies in the visual design, which according to the sitemap appears to be a deliberate choice of the design team. Page two of the Conan document states: “On the home page, the colored triangles in the upper-left corners are hard-coded as special panel styles. Throughout the rest of the site except for Local, they are the IxDA Aqua. On the Local page, they are the IxDA Orange. There’s a striped background image throughout the whole site” [14]. The aqua and orange colors don’t mix well and interchanging them for buttons and links in an inconsistent manner creates confusion for the user. The striped background image is missing from the discussions page, and the logo here is also smaller than on the other pages.

There is not much discussion on the website after the Conan Design document was made available some five years ago and the call for reviewers went out. Looking for documentation on website updates is the next logical step, since this was a joint project and one would assume through communication was in place. Bugs database is mentioned with an inactive webpage link http://tracker.ixda.org/ . A collaboration forum created on www.getsatisfaction.com has a last update from two years ago. Link to a blog specific to the implementation phase leads to page not found: http://www.ixda.org/blog/2009/09/conan-project-update-implementation-phase-begins. An effort was made at crowdsourcing design, with no documentation to support the fact that this actually took place. The slight deviations from this document to how this website looks today therefore cannot be explained.

On the bright side, the Forms and Data Entry theme received 100 % with fourteen out of 23 questions found relevant to this website. There are only few forms to fill out on this webpage and the process is smooth and error free. Chrome Autofill provided for most of the automated data input, so the question “users can complete simple tasks by entering just essential information (with the system supplying the non-essential information by default”) was a function delegated to the browser. Google forms were used to create application to start a local group as well as to apply for the speaker network. The first form opened in another window but the second one opened in the same window.

When done applying for speaker engagements, the only return option is by clicking on the back arrow, where a return button would serve the purpose better. Another statement under this theme makes one ponder for a while “forms allow users to stay with a single interaction method for as long as possible (i.e. users do not need to make numerous shifts from keyboard to mouse to keyboard).” Some explanation on why this is important and what functions is this statement referring to would have been helpful. But the author only shared a numbered checklist of the guidelines under each theme instead of more specific explanation under: http://www.userfocus.co.uk/resources/formschecklist.html.

Trust and Credibility is an important theme in the Expert Review Checkpoint, especially since this is a voluntary membership based organization website. It received 92 % due to the fact that some of its content has not been updated for the last few years. This issue is brought up twice, question number one “the content is up-to-date, authoritative and trustworthy” and question number ten “the content is fresh: it is updated frequently and the site includes recent content”. As per “the visual design complements the brand and any offline marketing messages” the branding is there but the visual shift between pages makes the user feel like she is drifting away from the original website. “The site is free of typographic errors and spelling mistakes” received a minus one due to few minor spelling mistakes. The site offers contact information but the feedback is inconsistent, therefore the zero grade for this quality staple.

A very strong one was given to the statement “the site avoids marketing waffle” and “the site avoids advertisements, especially pop-ups”, a rather refreshing quality in this age of constant bombardment with online ads. “Delivery costs are highlighted at the very beginning of checkout” received one as well, since no delivery costs apply to the purchase of the IxDA logo embossed notebook.

In the Writing and Content Quality theme many checkpoints aren’t that important. For example “pages use bulleted and numbered lists in preference to narrative text” is not a quality criteria since there are instances when bulleted and numbered lists cannot replace narrative text. It all depends on the context, which also defeats the purpose of “each content page begins with conclusion or implications and the text is written with an inverted pyramid style.” All of these questions don’t get grades due to being irrelevant or simply outdated in terms of responsive content organization. This site “avoids cute, clever or cryptic headings” is relevant in the context of IxDA and thus received a grade of 1. But this kind of headings can be useful for a more informal website, or one intended for kids.

“Text links are long enough to be understood, but short enough to minimize wrapping (especially when used as a navigation list)” – there are instances on the home page when entire question is made into a link to a page that contains the answer, and also an image and its title both being linked. The image title links to a Vimeo streamed videotaped conference presentation and the actual image links to a separate page with a blurb and the same video now streamed on ixda.org . It would have been equally effective if only the image was set as a link to one unique place. The most striking deviation is from the question “link names match the title of destination pages, so users will know when they have reached the intended page.” “Node” and “page” placed after ixda.org with forward slash and a random number create addresses usually for pages that cannot be accessed from the menu but from hyperlinks within other pages. It would have been more effective to name these pages according to the page they derive from and an actual description of their content. Instead of ixda.org/page/interaction it should be ixda.org/conference/interaction .

Page Layout and Visual Design is a theme covered in the Expert Review Checkpoint and the website received an unexpectedly high score of 74 %. The reason for the high score is that the questions didn’t address issues relevant to this website. For example “clickable images include redundant text labels” received a one, even though the fact that there is a broken image link on the home page can prompt some other reviewer to give this question a minus one. “Fonts are readable” received a zero because a similar if not the same san-serif font type is used for all of the text throughout this website. Headings are sized much bigger in comparison to paragraph text, the logo and menu items. “The organization’s logo is placed in the same location on every page, and clicking the logo returns the user to the most logical page (e.g. the homepage)” is accomplished on this website. However, the header and the footer on the discussions page differ from the homepage, the logo is smaller and the aqua blue diagonal stripes across the header are missing. Use of bright orange color for the paragraph headings and hypertext under the local menu feels unpleasant next to the pastel blue color of the logo and the background stripes.

The Search theme is at 89 % with twenty out of twenty statements found relevant. Several searches were conducted and the obtained results were satisfactory. The “templates, examples or hints on how to use it effectively” appeared as soon as the search didn’t yield any results. There were always options presented in tiled windows in the right sidebar to “sort by”, “filter by author” and “filter by title” with different options and categories listed, so this question scored a 1. The only negative point went to “the search box and its controls are clearly labelled (multiple search boxes can be confusing)” due to the existence of two search boxes on the main page, one in the top right corner and one in a tiled window. While on the discussions page, search is presented just by the loupe symbol below the header on right.

The last theme is dedicated to Help, Feedback and Error Tolerance. The “FAQ and forum guidelines” in the footer links to guidelines on how to conduct itself in the discussion forum but it does not contain a “step by step instruction” as expected by the expert review checkpoint. It is not clear what kind of instructions should be included since there are no specific tasks to be performed by the user. Out of thirty-six questions under this theme only nine were relevant to this website. This theme received a score of 79 %, which is due to the fact that most questions are for a website where the user has a specific goal, like buying a computer, or filling out a form. “The site uses a customized 404 page”, which includes tips on how to find the missing page and links to “Home and Search” – this guideline received a score of one due to a uniformly designed “page not found” that contains a search box and logo linked to the home page. This 404 page is accessed by searching ixda.org/two . But searching disque.ixda.org leads to a browser generic “this webpage is not available” webpage. Again, http://discussions.ixda.org/ doesn’t follow the naming convention and the server won’t associate it with www.ixda.org if the address is mistyped. Half of the questions under this theme are outdated since contemporary websites are expected to function smoothly and any error messages or extensive feedback interfering with the browsing will prompt the user to leave the website.

3 Discussion

The difference between SUS and Expert Review Checkpoint is that the first one is testing the user experience where the second one is used for a website review by an expert. In both cases the results are subjective and they can be made more reliable by adding more participant. This can easily be done with SUS since it is a rather inexpensive test. This test can also be applied during agile website development to test user experience for specific functions of the system [16]. The Checkpoint can also be taken by few experts and then the results compared but this is a more expensive process. Both tests give us a rough usability score but the Checkpoint review also provides a list of issues that can later be addressed during website improvement. The Checkpoint also has shortcomings, such as the same issue being repeated in statements from different themes.

The Checkpoint is obviously designed with an e-commerce website in mind, since an emphasis is placed on the existence of a shopping basket, product information, checking out and payment processes, searching etc. Some of the statements can be rephrased to better suit the business model of a professional group website. It is good to have the option of not answering statements that don’t apply to the website, and they are consequently omitted in the scoring process. Using the Expert Review Checkpoint during an expert review is more helpful than just making a laundry list of issues. An expert with a background in graphic design would probably focus on the page layout and visual design and forget to test some of its functional characteristics. If another expert with an engineering background also reviews the website the average overall score derived from both reviews will be considered more valid.

4 Conclusion

The overall impression is that ixda.org is functional but still needs work to improve esthetic and navigational issues. This website’s average usability was implied by the SUS score of 68 % it received, but this score can also suggest that users are more tolerant when it comes to esthetic and navigational issues. The similar score of 70 % for the Expert Review Checkpoint also tells us that this website is as functional as most others. If there was a continuation in testing Heuristic Walkthrough will be the next test. However, enough issues have been identified with the Expert Review Checkpoint and the accompanying expert review. Instead of assigning more assets to testing it will be more effective to assign assets to fixing the existing issues. Even though tests have been developed to prove the objective character of usability, the decision on what issues need fixing and what is the best way to do it still remains subjective.

In order to consolidate all the content and improve usability for this webpage one needs to start from adjusting the old sitemap to match the current website. Making changes and improvements to the updated sitemap according to the research findings will be the next step. It is one of the principles of heuristic research as stated by (Pólya, G., 1957) – “if you are having difficulty understanding a problem, try drawing a picture” [15]. Final step will be to change the code and transform ixda.org where it meets all of the standard website requirements as established by usability.gov [16]. In order to best reflects the Interaction Design Association’s value proposition, attract more members and energize participation ixda.org should be transform into an exemplary website.