-Getting into evaluation and usability engineering methods-
A brief recap of what has happened to bring us to the evaluation conversation
Last quarter we took a head first dive into the myAT&T mobile application. We talked to people about their most painful interactions with this application and plotted our findings against each other.Through this ranking of difficulty, we came to understand the most common pain points. From there we went into the app itself and studied the ways those actions are executed in the current experience. It was a patchwork of an experience. There were multiple ways a simple ‘+’ button was used, to give 1 example. It truly seemed like developers had stitched together multiple different attributes created at different times.
We then went into sense making. Plotting the verbiage that was particularly important to this specific experience and the 6 tasks we had identified. We plotted these nouns in terms of their relationships with each other. These connections needed to be thought about critically. It was not just a site map, it was connections both tangible and intangible, what was the back and forth of attributes to accomplish common tasks? After this initial sense making, we made concept maps which I blogged about and you can find that here. This allowed us to map the connections within the app in a way that imprinted them into our minds. This activity is exponentially important so that moving forward with creation, ideas and redesign we could quickly reference the important details and how they connected to other key components to enable action. After fully understanding the ins and outs we consciously tried to strip away all constructs of what the experience currently looks like. Divergent thinking. Push your mind into new territory. What would simplify? How could this action be streamlined? How can we innovate? In this phase, we allowed ourselves to be unrealistic, no constraints. To translate this expansion back into the applicable context is difficult but needed, to be effective. Here is the blog that I wrote regarding the portion of the process walking you through divergent thinking and into initial paper prototype sketches.
Bringing this comprehension into digital space for the first time, I felt confident. I was quick to learn that I did not have the capability to see all of the details yet. There are things that one cannot be aware of until their attention is explicitly pointed towards the details that are imporant. And often the most important details are the hardest ones to see.
Which brings us to evaluation and product management. In this class, we are learning how to evaluate the learnability, usability and, essentially, effectiveness of a digital interface. The methodologies we are learning are the best-known combinations of effectiveness while also being inexpensive. The methods are called Cognitive Walkthrough, Heuristic Evaluation, and Think Aloud Protocol. These methods, in combination with eachother, provide a holistic view into the issues that prevent ease of use for a digital interface.
Cognitive walkthrough is a method for observing and assessing the learnability of a product. This method is based on a theory about the way that people analyze and solve problems. The way that I have started to think about it is like the signage of a grocery store. If our user wants some cheerios what words are they going to look for? What other things would they naturally associate as surrounding products? etc. It is performed by walking through the digital experience, screen by screen and asking the series of questions as follows:
- Will the user try to achieve the right effect?
- Will the user notice that the correct action is available?
- Will the user associate the correct action with the effect that user is trying to achieve?
- If the correct action is performed, will the user see that progress is being made towards their goal?
From this, a designer gleans the knowledge needed to make sure that the experience invites the user into it in a way that makes sense to them.
Think Aloud Protocol and Heuristic Evaluation are for gauging the overall usability of the interface. Think aloud protocol is a process that a designer facilitates with actual users who have never seen the product before. This is to create an unbiased set of responses. The user is asked to run through a prototype and verbalize the actions they are taking and their thought process to get there. To give an example, if I was opening a door handle my thought process may go something like this “Alright, I am going through the door, I see the door handle, I know that I need to turn it, I turn it, go through it and close the door behind me.” The only words the evaluator uses are “Please keep talking” if the user falls silent. This verbiage is so that they don’t go into introspection, they continue to be focused on the interface rather than think about their answer or if it is good enough. The reason you want to avoid the introspective effect is that this activates a different part of the brain than operational, task oriented thoughts do. The way I think about this one is it is like the grocery store signage designer observing real shoppers walking through her/his systems. Is it being used the way the designer hypothesized? Everybody’s brain works differently, in different patterns, with different values. Then, on the contrary, there are those things that make sense to everyone and everyone can relate to, understand and abide by. Which leads us into Heuristic Evaluation. A metaphor to illustrate the way I think about viewing a digital product through this lens is its like the rules of the road. The heuristics of the road. You have your different kinds of lines (double solid yellow, dashed etc.) on the road that mean different things. There are guidelines for what to do and what not to do. Similarly, there is a common language, and a common list of best practices for mobile applications. Do’s and don’ts. You can build the most gorgeous car ever but if the size of the product is the width of two lanes of the road, your car will not be effective and will never be used. The same goes for heuristics in mobile applications. The heuristics to hold a creation up against are as follows:
- Visibility of system status
- Match between system and the real world
- User control and freedom
- Consistency and standards
- Error prevention
- Recognition rather than recall
- Flexibility and efficiency of use
- Aesthetic and minimalist design
- Help users recognize, diagnose and recover from errors
- Help and documentation
These are exponentially useful for understanding applications in general and the common language that is behind the creation of any digital products. These are the kind of details, again, that are the most important…. So important, in fact, that when they are well executed, the user doesn’t even notice they are a thing that someone has to think about.
The utilization of evaluation
As explained, the 3 kinds of evaluations are valuable for different reasons. I used them in a particular sequence that I found to be very effective.
During last quarter, we did multiple iterations of our screens and in between the last few iterations we used Think Aloud Protocol. This quarter, I knew that I had some screens missing. Such as, when something needs to be typed in with a keyboard, I had the keyboard appear fully expanded without a screen for the tapping action that precludes the keyboard being on the screen. The screen on the far left, below, is the iteration before evaluation. I then added the screen to the screen in the middle to preclude the screen on the right to mitigate this keyboard issue as well as protect users passwords because I realized that showing a users password from the get-go was a major security issue.
So, for this reason, and knowing that Cognitive Walkthrough is for checking the learnability of a product, I used this method to find the places in the experience where I needed to add less important, filler screens such as the keyboard example. This worked well because it opened my eyes to other details that felt abrupt. The things that I didn’t make the user ready for.
After adding these screens and noting learnability issues I moved on to heuristic evaluations. I had 5 evaluators look at my screens and log the feedback in a spreadsheet. As I got comfortable in using this evaluation I started to not have to reference the list of heuristics, I would just start to see the issues and why they manifested that way. I think learning these methods alongside actually building an app is such a valuable cadence for learning because on both sides of the equation; evaluation vs creation, the learning is applicable. I am excited to rebuild the rest of my screens for this AT&T redesign and am even more excited to begin on the creation of my own application from scratch with all of these “rules of the road” ingrained into the way I look at interfaces now.
Lastly, I did a few Think Aloud Protocols saw that in a lot of ways, users were doing what I thought they would do, and in a lot of ways, they weren’t. Work to be done! Improvements to be made!
Breaking down evaluation findings
I’m going to start at a high-level description of the synthesized take always and then break them down in terms of where they came from and how they inform redesign.
The experience felt discombobulated. There is little to no feeling of continuity. The user always feels slightly unsure if they’re doing it right.
Attributes and/or features are not effectively communicating their purpose. The design does not provide clarity.
Visibility, control and freedom are huge overarching issues. Each screen is separate with separate actions and it is relatively arduous to go back a couple steps to change something.
Hierarchy and priority are not clearly visualized. Everything feels the same. The design does not draw the users’ attention to the next step.
From Cognitive Walkthrough I learned a lot about verbiage. The wording and phrasing is so important to afford the right actions findable. This also ties in with difficulties in navigating. Navigation flaws are very often linked with the wording that is used to inform movement. Navigation was also inhibited by screen length, mostly in regards to where the screens were getting cut off. I did not think about screen length or scrolling at all in my first phases of design so this was a huge learning. And lastly Cognitive Walkthrough was an excellent method for finding screens that were missing as I said before.
From the Heuristic Evaluations I began to see the gestault of failure in overall visibility of the app and where users were in completing tasks with multiple steps. I found that my users, often times, did not have full control of their experience. There were actions and buttons that were the end all, be all of a task and the user did not have full freedom, I was mandating them fitting into my ‘template’ of actions. And lastly from Heuristic Evaluation I more fully understood the importance of users being able to recognize something as opposed to recalling it from previous usage. This is especially significant in the context of the AT&T app because I assume that users don’t interact with the experience more than once per month. Compliment this with the fact they users probably want to get in and out fairly quickly, recognition should be highly valued.
Lastly, Think Aloud Protocol made me understand more thoroughly the power and importance of word choice. What is applicable to the greatest number of people? I was approaching the initial creation of the application with a very minimalistic mindset. I felt, and to a large degree still feel, that at this day in age simplicity is largely important. However, when it comes to sensitive information, money, or actions that are difficult to reverse, customers are very tentative and protective of their space. Therefore, enough information needs to be handed to them so that they feel they have full comprehension of what is happening. And sometimes, in the right circumstances, that is a short text block. The combination of explanation and action is important.
Redesign based on evaluation
I will illustrate the redesigns I have made in the context of a flow of screens that are about the action of ‘Upgrading a device’. I chose this flow because it housed all of the issues identified above in some capacity.
First let me break down some of the changed attributes…
I changed the attribute that compares the data used by individual users on the account to the amount of data the account as a whole has. This was informed by a misunderstanding of the comparison of its initial, more boxy, vertical format, with the other ways that timelines and data were visualized.
This first visual is how the billing cycle and data are displayed:
Take that into account when looking at how individual users data usage are compared with each other. The formatting is utterly not the same…
Which brought me to the following design:
I changed the boxy attribute to be the same shape as the other barometers to provide ease of mental digestion of the information.
In terms of wording, I changed quite a few things. To give a few examples, I changed:
Because I learned, in think aloud protocol that this could be perceived as a positive amount of money rather than an amount that needed to be paid.
Another wording change was from ‘Devices’ to ‘Devices & Users’ on the tab bar.
This was significant because people were going to the “plans and usage” tab for actions regarding individual users’ information in regards to data rather than to the ‘devices’ tab. Which is fine, but not intended and in 1 circumstance was detrimental to the desired action to be achieved.
To give you a full sense of the changes I made and how they manifest here are my old and revised flows back to back.
First is the old version of ‘Upgrading a device’:
Then here is the redesign:
The umbrella goal of the revision is to provide ease of back tracking. Buying a phone is a complicated transaction with a lot of steps and a lot of details and it is satisfying being able to see all of the things you have chosen while you are making your next choice.
To summarize my takeaways; this is how I will be thinking about these methods of evaluation going into the future.
What invites users in the experience in a way that makes sense to them?
Think aloud protocol:
What doesn’t make sense at all?
Structure and guidelines – what are people used to?
In combination, these provide clarity and insight into how to make products that people love to use.
You can see my full presentation AT&T Evaluation Deck here.