Join Natalie Garza and accessibility expert Natalie MacLees in the 19th episode of the AAArdvark Accessibility Podcast. In this episode, they discuss the vital practice of user testing, especially with people with disabilities. From defining user testing to its execution and why it’s indispensable, learn how involving disabled users can uncover specific issues that would otherwise be missed. They also explore how to find and recruit disabled testers, the help available from specialized organizations, and the benefits of conducting tests remotely.
Natalie Garza: Hello, everybody, and welcome to the AAArdvark Accessibility Podcast. This is episode 19. I am Natalie Garza, one of the co-hosts, and with me today is.
Natalie MacLees: Natalie MacLees, accessibility expert.
Natalie Garza: And in today’s episode, she’s going to teach us all about user testing, specifically with people with disabilities. So, to get started, what is user testing in general?
Natalie MacLees: Yeah, so when you’re building a product or a website, you can do testing with real users, and that can look like a few different things that we’ll talk about today. But basically, you sit down real users in front of your product.
And have them try it out. And it is one of the most maddening and frustrating things you will ever do because no matter how carefully you have designed something and you think it’s so beautiful, users will not be able to figure out how to do anything and it will drive you up the wall.
But basically, you sit them down and give them a task and say, you know, “Hey, find information about elephants on this website,” or “put a product in a cart and check out.” You give them a task like that to complete, and then you kind of observe as they go through that task. You see where they run into problems, where they run into issues, and where they get confused, and you keep track of how long it takes ’em to complete the task.
Natalie Garza: Yeah. As frustrating as it can be, it’s probably one of the most valuable things you can do.
Natalie MacLees: Absolutely, you can learn so, so much about your product and you could, you could spend hours looking at a screen of your app, for example, and in two or three user tests get way more information on what should be changed and how it should look, by watching some real users attempt to use it.
Natalie Garza: Yeah, it’s a real eye-opener, ’cause you realize not everyone treats technology the same as you.
Natalie MacLees: And not everybody is super tech savvy, you will have situations where you have a giant flashing red button in the middle of the screen that says, “click here,” and users will go,” I don’t, I don’t see where to click. I don’t.”
And it’s really hard to not just do it for them.
Natalie Garza: Yeah, you, you can’t help ’em if they’re just staring blankly. You just have to let them.
Natalie MacLees: Yeah. You can kind of step in and kind of gently nudge them if they start to get really frustrated. But yeah, you should generally try not to participate too much.
Natalie Garza: Yeah, and I would say user testing usually is targeted towards like the audience of the product that you’re working on, like if it’s a business product, you’re gonna get the business people to come test it. Or if it’s for non-tech-savvy people, you’re gonna get non-tech-savvy users.
Natalie MacLees: Yeah, exactly. So you wanna figure out who your user base is, and then that’s who you wanna recruit to come in and do the user test.
Natalie Garza: But often disabled users get overlooked, so why should we test with disabled people as well?
Natalie MacLees: Because they are going to be using different assistive technology or just different features of the devices that they happen to be using, and will uncover barriers and issues that users who do not rely on assistive technology or assistive features of products, things that those users won’t find.
So you could uncover issues where you know a button can’t receive keyboard focus. And in a user testing with users that don’t have disabilities, you might never discover that issue. But if you have users come in who rely on keyboard or users come in who use screen readers, they will identify that issue immediately and find it for you.
Natalie Garza: Yeah. What if you are working on a product and you’re like, “okay, well I have an accessibility auditor gonna come check this product,” why should you still test with disabled people?
Natalie MacLees: Yeah, the auditor is probably gonna do a pretty good job, but most of the time they’re probably not a full-time assistive technology user. And even if they are, they are probably a full-time assistive technology user of one type of assistive technology. And of course there’s all different kinds. You know, people with disabilities are not a monolith.
So there’s not just like, “oh, here we just had one, one user with some kind of disability, come in and use this. So we’re covered now.” There’s so many different ways that people adapt devices and adapt websites, and adapt products to be useful to them, and you need to cover as many different ways as you can.
Natalie Garza: Yeah, we did have another episode where we talked about all the assistive devices and all the assistive tools, too. So if you wanna check it out, put a link up here somewhere or link in the show notes. All right, so how do we find people to test with disabilities?
Natalie MacLees: Yeah, so if you are doing recruiting for user testing, you do maybe wanna ask about that and have that in your recruiting questionnaire, to figure out if this is somebody who uses some kind of assistive technology or assistive features. And you would wanna make sure that you’re including some users from that group in the testing.
If you’re having trouble, there are organizations that you can actually work with who will help you, recruit users, with disabilities for your test. I have personally worked with the World Institute on Disability, which I think is up in San Francisco. But we’ve done everything remotely, so I’m not sure.
But they have been fantastic and they actually can help, recruit the users and run the tests, and they have, I think they have gone to doing it remote now, but prior to the pandemic, of course they had an in-person lab that people would come in and they would have, you know, they’d have each user come in and they would do four or five tests for different clients on the same day.
And they would get paid. Users get paid for doing user testing, and they would record the session and you could watch it live or you could watch the recording later. And then they, give you a report at the end with the findings from the report. There are other organizations as well that can help you out with it.
Fable, who has a website at “Make It Fable”. If you’re in Australia, there’s a company called Intopia. In the UK, a company called Nomensa. And then we also have UsableNet and AbilityNet, which can also help out with this area.
Natalie Garza: Okay. And do they help guide you towards, like, the type of users you need to be in contact with?
Natalie MacLees: Yeah, so they’ll ask you about it. And then, when I’ve worked with the World Institute on Disability, I would always do a phone call with them, and we would talk about what the project was, who I expected the users to be. I know one time I hired them, and it was a website for teenage girls. So, you know, they talk about that kind of thing because they’re gonna try to recruit people from that audience.
Right. If I’ve built a website for teenage girls having, you know, middle-aged men come in and test it is not gonna be the most useful. So we wanna have people who are as close to that demographic as we can. And then we would talk about what are the features on the website and what are the things that we’re concerned about doing?
And they would make some recommendations to me, let’s say like, “oh, we should test two screen reader users and we should test one person who uses a braille display and we should test one person who uses this.” And, they would help me get that all sorted out. And how many testers that we needed to use.
And they also helped to figure out what the tasks were that we should assign to the testers. So they were very helpful, to help with all of that. And for all of that service, it was very reasonably priced.
Natalie Garza: Okay. And then you said, they give you back a report.
Natalie MacLees: Mm-hmm.
Natalie Garza: Does it have like, “oh, you broke this WCAG success criterion”, or is it more just like I?
Natalie MacLees: It was the same kind of report you would get from any kind of usability testing, which was this user, you know, we gave each user five tasks to do. And we tested 10 users, and they’ll say, okay, eight users were able to complete task number one. And it took them an average of this many minutes.
This many users could complete task number two, and it took them this number of minutes and then they would give you, the places where people got stuck like, “oh the, you know, people couldn’t complete task one ’cause they couldn’t put a product in the cart and these were the issues that they ran into.”
And they would give you that kind of report back of like, here’s the issues they ran into when they couldn’t complete it. And even when they could complete it, maybe they ran into something that they had to figure out how to work around, and so that’s the kind of information they would give you back in the report.
It would not be like an audit where they were telling you like, “oh, you failed 1.4.3.” It was all very focused on the users and the tasks that they were doing, whether or not they could be successful, and whether or not they ran into problems.
Natalie Garza: Okay. And it had the added layer of like, okay, they used keyboard only
Natalie MacLees: Yeah. Yeah.
Natalie Garza: Okay.
Natalie MacLees: And it would inevitably, you would get back feedback that would say something like, “oh, the screen reader users loved this feature, but this other user hated it and couldn’t find their way around it, and it blocked them,” which is one of, you know, we’ve talked about that.
It’s one of the frustrating things about accessibility, because people with disability are not all the same, right? It’s a group of people with so many different needs and adaptations and, so many different ways to approach things that there’s no, there’s often not like a single solution that’s gonna work for everybody.
And so you just have to figure out how to be flexible and find what is gonna work for most people.
Natalie Garza: And then you said you said that most of it is online now.
Natalie MacLees: Yeah, so usability testing, I think that this is true not just of testing with people with disabilities, but I think it’s also true just in general with user testing in the world. I think we had to, we just have a lot less of people coming into a lab, to do user testing. There might be a little bit of less data as a result of that because in a lab, you know, they have an eye tracker and there are multiple cameras from all different angles tracking everything somebody is doing.
But it’s also like a very, uncomfortable environment, right? You can imagine that if you were sitting at this like teched out desk and you’re in a room by yourself and there’s a one way mirror, right? Where who knows, you don’t know how many people are sitting back there watching you try to use this website.
And then there’s cameras everywhere, every which way, recording everything you’re doing and your eye movements are being tracked and all of those kinds of things. So how realistic that is because that is not how people use things, right? People don’t sit in isolation in a room with no interruptions to use, to use web applications.
That’s just not, that’s not how it happens. So a lot of it has gone to being online with, you know, like, we’re recording this podcast right now online. We’re not in this; we’re not in a room together.
But you can get somebody on like a Zoom session and, you know, walk them through and watch them try to use a product, share their screen with you, and watch them try to get through some tasks in your app.
Natalie Garza: Yeah, in that way, it kind of made it more accessible for everyone to participate. You don’t have to live in San Francisco.
Natalie MacLees: Yeah. Yeah, exactly. You don’t have to live in a certain place. You don’t have to travel to a place. You don’t have to sit in that sterile room, be watched while you try to do something. You can be at home, on your own computer, at your own desk, right, and be a lot more comfortable. Which is better for everybody, I think more convenient for everybody.
Natalie Garza: So going back to them guiding you, like if you go through one of these agencies or one of these, what organizations, going to help you pick out the devices or is that something you have to come in and say like, “I want, I really wanna test screen readers. I really wanna test Braille device.”
Natalie MacLees: So that’s what they’ll help, that’s what they’ll help you pick out. And if you do have something like that in mind, like you’re like, “well, I’m just really curious how somebody using a Refreshable braille display would use my app.” Definitely let them know, because they can help you recruit a user with that.
But generally, generally I would say approach it, with an open mind and let them suggest to you kind of, you know, let them ask you a few questions about your application, what it’s for, who’s using it, and let them suggest to you who maybe you should test with.
Natalie Garza: Alrighty, well thank you so much for going over user testing with user’s with disabilities. So, to wrap up, where can people get started with digital accessibility?
Natalie MacLees: Yeah, well come on over to aaardvarkaccessibility.com. You can test your homepage for free, find out what issues it might have, and get suggestions for how you can fix them.
Natalie Garza: Yes. And so with that, thank you for watching. This is the 19th episode of the AAArdvark Accessibility Podcast. We will talk to you guys next time.
