Usability Issue Reporting Format

(1) Description

When [PRECURSOR], the [COMPONENT] is / has / is experienced as / results in / etc. [PROBLEM] which [CONSEQUENCE].

COMPONENTS of the intervention should be detailed using the same structure reflected in the DDBT intake form, including (1) content elements (discrete techniques), (2) structures (processes that guide the selection and delivery of content), (3) artifacts (tangible, digital, or visual materials), or (4) parameters (static properties that define and constrain the intervention or service “space”).

Examples:

When writing your description, avoid the following pitfalls:

• Don’t start with [THE INTERVENTION]…
  • Instead, start with [COMPONENT OF THE INTERVENTION]
  • So, instead of “Problem Solving Therapy was experienced as….” start with “The problem solving processes introduced in PST…” (content element) or “The number of sessions required for PST…” (parameter)

• Don’t focus just on one consequence if there are multiple consequences. Various consequences affect different stakeholders differently.
• Don’t be vague in problems or consequences
  • Avoid vague language about “difficulties,” “problems,” etc..If you don’t have the information to be more specific, more user evaluations may be needed and, should be recommended as a next step.
  • “This takes too much time” is too vague because you don’t know what to do next or what the consequences of it taking too long are. If could mean:
    • “The structure of the therapy doesn’t allow enough time to get to know to my patients”
    • “Takes more sessions than I have with my patients, since they don’t see me regularly”
    • “Takes longer than the session hour, so clinicians and patients don’t get everything done in a session”
    • “Takes longer than the session hour, so clinicians often have to take over parts of the therapy that my clients really should be leading”
• Don’t presuppose or imply a solution. For example, “When mental health services are offered only in-person, potential patients are uncomfortable presenting themselves for care, which results in people not accessing care from which they may benefit” implies that the problem is only in person care and that other modalities are necessary. While other modalities may be a great way to address this problem, reframing this issue as “Potential patients are uncomfortable presenting themselves for in-person care because of concerns about seeing people to whom they do not want to disclose their mental health concerns at or near the clinic” leaves the solution space more open, such as designs that create greater privacy when accessing in-person care.
• Don’t accept at face value reports from one stakeholder group about usability issues that might be experienced by another group. A stakeholder group (e.g., clinicians) often can describe experiences of other stakeholders (e.g., patients), but they are often only able to see an incomplete picture or it may be based on biases. You will ultimately want to test any assumptions about how a particular user group will respond by engaging in user research with that group.

Additional guidance:

• If indicated, you may reference how the information was obtained in the description of the usability issue (e.g., “clinicians reported… ”)
• Especially later in a redesign process, usability issues may be described in a way that is comparative (i.e., “X is less XXXXX than Y because XXXXXX”). Be specific about the comparison.

(2) Severity

Severity helps prioritize the fixing of problems and allocation of resources to fixing them. There are various scales, some developed more for interfaces, others that apply more broadly. Example scales include:

• 3 point: disaster, serious, cosmetic
• 5 point: catastrophic, major, medium, minor, cosmetic

We generally recommend Dumas and Redish’s four-level scale, with a modification to account for how some usability issues can cause harm:

• Level 0 – catastrophic; causes harm; high risk
• Level 1 – prevents completion of a task
• Level 2 – creates significant delay and frustration
• Level 3 – has a minor effect on usability
• Level 4 – subtle problem, points to a future enhancement

It offers a reasonable level of precision, without being overwhelming. Additionally, the descriptors are broad apply not to just to screen-based interfaces.

Because there are multiple scales, we recommend reporting the full descriptor or a short name (e.g., “4 – subtle”) rather than just numbers (e.g., “4”).

We recommend that multiple team members rate each issue, if the team has sufficient expertise and familiarity with the data to do so. If there are disagreements about severity, resolve them through team discussion.

(3) Scope

To what extent is the problem present in the product or service? This can refer to prevalence across the system, service, or artifact as well as the prevalence of how many people it affects.

• Local – isolated to one page or section; for a particular stakeholder group
• Global – throughout the interface or experience; for all participants

We’d love to know number of users and from which groups across your usability testing.

(4) Complexity

How difficult is the problem to understand or reproduce? How easy is it to fix? Generally, more complex problems will take more time and resources to fix, often starting with more detailed study of what is going on.

A low complexity problem is easily explicable. You know what is happening and why. You likely have an idea for how to fix it and that fix seems feasible.

A high complexity problem may have unclear causes. People may be making mistakes and you don’t know exactly why or when. You may not understand how to fix it. Alternatively, the problem may be clear but the solution is either unclear or may require large scale redesign of the intervention (e.g., reordering of all steps in a service) or even rethinking the health system.

(5) Evidence

As a minimum, we ask that you provide exemplary evidence. The goal of this evidence is to help others understand the problem, and the standard should be that it is persuasive and informative to someone who has to redesign the intervention as a result.

Often, the most informative evidence combines data from different sources: e.g., qualitative and quantitative data from usability testing, quantitative data about the number of patients who complete versus drop out of the treatment after certain number of sessions.

As you decide which evidence to include, consider the other categories. Together, the set of evidence should illustrate the component, the issue, the consequences, their severity, and the complexity.

For center reporting requirements, we do not need all of your evidence on an issue here. That said, if you find it easier to present all the evidence, we’re happy to work with it.

(6) Related research

Does this problem connects to other research? If so, we would appreciate pointers to the literature. If not, that’s where our work as the center methods core will pick up.

(7) Next steps / redesign

As projects move from stage-to-stage, we anticipate that you may revisit and revise your next steps. For example,

• at the end of discover, you might identify an issue but also know that, to address it, you need to do further detailed discovery as part of your design work.
• during test, you may confirm that a proposed solution works. Alternatively, you might identify new information that indicates the usability issue was not what your team first thought.

Please keep us updated as you learn!