Skip to content

API Usability Evaluation Guide

Jump to bottom
Tao Dong edited this page Mar 9, 2021 · 11 revisions

This document defines a set of rubrics for evaluating the usability of routing APIs for Flutter based on the Cognitive Dimensions of Notations framework.

Core dimensions

We will focus on the following 7 dimensions in our evaluation:

  • Role expressiveness: do the API names clearly communicate what the APIs do?
  • Domain correspondence: does the API directly map the concepts the programmer thinks in the application domain?
  • Consistency: is the API consistent with its own surface and across the API surface of the framework?
  • Premature commitment: does the API require the programmer to make some upfront decisions which are hard to reverse?
  • Abstraction level: does the API allow the programmer to achieve a set of common goals with just a few components or many building blocks?
  • Viscosity: does the API allow the programmer to make changes to their code easily? Is there a “domino effect” when refactoring code?
  • Work-step unit: how concise the code is for implementing common API usage scenarios.

The rest of the document describes how to assess each dimension.

Role Expressiveness

Role expressiveness is about how well the API’s names and usage communicate what the system does. We aim for transparent role expressiveness. In other words, “say what you mean and mean what you say!”

Assessment procedure

  1. Select a scenario from our common scenario catalog

  2. Examine a code snippet implementing that scenario using the API in question

  3. Try to summarize what each code block does in your own words. When reading code that uses the API, was it easy to tell what each section of the code does? Why? Which were the parts that were more difficult to interpret?

  4. When reading the code, how often do you feel you need to check a class or a method’s API reference to understand what it does?

    • Frequently
    • Occasionally
    • Rarely

  5. Repeat the above for another scenario

Assessment results

Overall rating:

  • Transparent: the code can be interpreted correctly without reading documentation
  • Plausible: the code can be interpreted correctly after reading documentation
  • Opaque: the code cannot be interpreted correctly after reading documentation

Domain Correspondence

Domain correspondence refers to how well the API maps to concepts the programmer needs to manipulate in the application domain.

Assessment procedure

Domain correspondence can be assessed via a concept mapping exercise, following these steps:

  1. Describe a specific app usage scenario from the end-user’s perspective

  2. Describe the same scenario in generic technical terms

  3. Examine a code snippet implementing that scenario using the API in question

  4. Identify classes and methods exposed by the API

  5. Discuss how related those classes and methods are to the generic technical terms in step 2. (For example, if an API for file I/O exposes a File class, does that map well to your understanding of a file?)

  6. When considering an app usage scenario, was it easy to map from that scenario to API code?

    • Very easy
    • Somewhat easy
    • Somewhat hard
    • Very hard

  7. Repeat the above steps for another scenario

Assessment results

Overall rating:

  • Direct: the mapping makes sense without reading documentation
  • Plausible: the mapping makes sense after reading documentation
  • Arbitrary: the mapping does not make sense even after reading documentation

Consistency

Consistency in an API allows users to make use of what they have learned about how one part of the API works when they start working with another, similar part of the API. There are two aspects of API consistency we need to consider: within-API consistency and cross-API consistency. They are further explained in the assessment procedure.

Assessment procedure

Please assess the following three aspects of API consistency based on the API usage.

Within-API consistency

  • Examine the snippets of using the API to implement different scenarios, identify any conceptually similar user goals implemented differently. Are there good reasons for the differences?

Cross-API consistency

  • Identify new coding patterns, if any, introduced by this API that are not used anywhere else in the Flutter framework. Are the benefits of introducing those new patterns outweigh the cost of learning them?

  • Identify new concepts, if any, introduced by the API that do not exist anywhere in the Flutter framework. Will these new concepts cause confusion?

Assessment results

Overall rating:

  • Full: API achieves a high-level of both within-API consistency and cross-API consistency
  • Core: API achieves a high-level of within-API consistency
  • Arbitrary: API fails to achieve a high-level of within-API consistency nor cross-API consistency

Premature commitment

This dimension assesses the consequences of making premature decisions in API usage and the cost of reversing a decision. We aim to present the user with a small number of choices about how to accomplish some goal. We also aim for reversible choices, so users can recover easily.

Assessment procedure

  1. Select a scenario from our common scenario catalog

  2. Examine a code snippet implementing that scenario using the API in question

  3. List all the choices the developer needs to make when writing this snippet. Draw a decision tree if applicable.

  4. Describe how the developer is supposed to determine what to do next at any branching point in the decision tree drawn at the previous step.

  5. Was it obvious that you needed to make those decisions or did you learn about this through trial and error?

    • It was obvious
    • I had to learn through trial and error

  6. If the developer made a bad choice, what does the developer need to do to recover from that? Was the effort of reversing a bad choice reasonable?

  7. Repeat the above for another scenario.

Assessment results

Overall rating:

  • Minor Reversible: If the API presents the user with a small number of choices about how to accomplish some goal and if the differences between any alternatives are minimal
  • Major Reversible: If the API presents the user with a significant number of choices about how to accomplish some goal and if the differences between the alternatives are minimal
  • Arbitrary: If the API presents the user with a significant number of choices about how to accomplish some goal and if the differences between the alternatives are significant

Abstraction Level

This dimension is a measure of the type and number of abstractions that the developer has to work with. We aim to lower the barrier to entry with aggregate components. Provide Layered and Progressive Abstraction Architecture. In addition, abstraction level needs to fit the target programmer’s work style.

Assessment procedure

  1. Select a scenario from our common scenario catalog

  2. Examine a code snippet implementing that scenario using the API in question

  3. Count the number of classes that have to be used to implement the scenario

  4. Given the number of user-facing concepts in the scenario, does the number of classes seem to be reasonable?

    • more than what I would expect
    • About the same as what I would expect
    • Fewer than what I would expect

  5. How would you describe your experiences with respect to the types of classes used to implement this scenario?

    • They were just as I expected
    • They were too low level
    • They were too high level

  6. Describe the parts of the implementation you feel the system should just take care of for the developer.

  7. Repeat the above procedure for another scenario.

Assessment results

Overall rating:

  • Aggregate: I was able to use one component to achieve a set of goals
  • Coordinated: multiple classes need to be used together to achieve a particular user goal, but it’s easy to discover what else is needed from a single entry point.
  • Primitive: Individual components exposed by the API do not map on to unique user tasks and it’s hard to find what is needed to accomplish a user task.

Viscosity

API viscosity measures the resistance to change of code written using a particular API. When you need to make changes to code that you have written using the API, how easy is it to make the change? Beware the 'domino' effect of any change.

Assessment procedure

  1. Select a scenario from our common scenario catalog

  2. Examine a code snippet implementing that scenario using the API in question

  3. Introduce a requirement change to the scenario. In the context of routing, this could be changing a route, adding a sub-menu, adding a 404 page, re-organizing pages on a site, etc.

  4. Examine the code snippet rewritten to reflect the change.

  5. Count the amount of changes using diff tools

  6. When you need to make changes to code that you have written using the API, how easy was it to make the change? Why?

  7. Are there particular changes that were more difficult or especially difficult to make in your code? Which ones?

  8. Repeat the above steps for another scenario

Assessment results

Overall rating:

  • Low: the changes are local and small.
  • Medium: the changes spread out in a few different code blocks but they do not affect the app architecture.
  • High: the changes require adjustment of the app architecture or they are large enough to be characterized as having a “domino effect” from a seemingly simple user-facing change.

Work-step Unit

This dimension is about how concise the code is for implementing common API usage scenarios.

Assessment procedure

  1. Select a scenario from our common scenario catalog

  2. Examine a code snippet implementing the main navigation tasks in the scenario using the API

  3. Count the lines of code for implementing that task

  4. How would you describe the amount of code that you had to write for the task? Did you have to write more code than you expected or did you have to write less code? Please explain.

  5. Repeat the above for another scenario

Assessment results

Overall rating:

  • Local Incremental: the amount of code for accomplishing a task is proportional to the size of the task, and the code is completely contained within one local code block.
  • Functional Chunk: code for accomplishing a task is not local, but in clearly connected code blocks (e.g., via a call back function)
  • Parallel Components: the amount of code for accomplishing a task is out of proportion, and the code also spreads across multiple code blocks or classes.

Additional dimensions

There are 5 additional dimensions in the Cognitive Dimensions of Notations framework, but we don't plan to use them in this evaluation due to help scale the effort across multiple packages. For completeness, those dimensions are listed below:

  • API Elaboration: Were you able to use the API exactly ‘as-is’ or did you have to create new objects by deriving from classes in the API? How easy is it to extend the API for advanced use cases?
  • Progressive Evaluation: How easy was it to stop in the middle of the task you were working on, and check your work so far?
  • Working Framework: How many classes you had to work with simultaneously to accomplish the task?
  • Learning Style: which learning style does the API support? Trial and error, example-centric, or systematic?
  • Penetrability: How much of the details of the API do you have to understand in order to be able to use the API successfully?

References

Clone this wiki locally