I Spent Years Refusing to Build an Accessibility Score

For years, the answer was no.

The question came up constantly. From my business partner. From customers. From people watching a demo who just wanted to know where their site landed. Every time, I said no, and not because it was hard to build. I said no because I knew what would happen the moment we shipped it. Someone would see 94 out of 100 and stop working.

I wrote about this last October:

I’ve resisted adding a score to AAArdvark because it would give people false confidence.

I still believe that. And we are shipping an accessibility score.

Here is what changed.

Why a score bothered me

The score becomes the answer instead of the starting point, and the work stops exactly where the hard part begins.

An automated accessibility scan tells you exactly one thing: whether or not the scanner found anything it knows how to look for. It does not tell you whether someone using a screen reader can finish your checkout. It does not tell you whether a keyboard user can get past your navigation menu. It does not tell you whether your error messages make any sense to a person who cannot see the red outline around the field.

Automated testing catches a real but narrow slice of accessibility problems. Everything else requires a human being making judgment calls.

That is not how a number reads, though. People see 94 and think “almost there.” They see 100 and think “done.” The score becomes the answer instead of the starting point, and the work stops exactly where the hard part begins.

That was my objection, and I held onto it for a long time.

Nathan never agreed with me

My business partner has believed we needed a score the entire time. He agreed to launch without one. He never once treated the question as settled.

We disagreed about this for years, and we disagreed well. Nobody went around anybody. We would talk it through, fail to convince each other, and come back to it a few months later.

That turned out to matter more than either of us intended. Because the question stayed open, I never stopped researching it. I read the academic literature on accessibility metrics, including the work on why raw issue counts mislead and why conformance levels are too coarse to compare one page against another. I studied how other products score sites and where their models fall apart. I collected approaches from accessibility professionals whose judgment I trust.

So when we finally decided to build one, I was not starting from an opinion. I had years of notes.

Onboarding worked. Everything after it did not.

We were handing people a map of the terrain with no route drawn on it.

Signing up for AAArdvark goes well. You create an account, add your site, run your first scan, and get results. That part is smooth.

Then people stop.

Not because the results are wrong, but because a list of issues does not tell you what to do with your Tuesday afternoon. You are looking at a scan full of findings. Some are straightforward. Some need a decision from a person. And an entire category of problems never showed up in the scan at all. So where do you start? How do you know when you have done enough of one thing to move on to the next?

We knew the answer, and that was the problem. The workflow lived in our heads, in our documentation, and in the way our support team answered questions. It was not in the product. We were handing people a map of the terrain with no route drawn on it.

Accessibility testing has an order

Here is the route, and it is not complicated.

1. Fix what automated testing found. These are the known quantities: missing alt text, low contrast, unlabeled form fields.
2. Review what needs a human decision. Automated tools flag plenty of things they cannot resolve on their own. Is that image decorative or meaningful? Does that alt text describe the right thing? Somebody has to look.
3. Do the manual testing. Keyboard navigation, screen reader testing, focus order, and all the things no scanner will ever catch.

Most accessibility tools show you the first stage and then go quiet. That is precisely why a clean scan feels like a finish line. The tool stopped talking, so you assume you are finished.

We had built all three stages into AAArdvark. We had just never made them visible as stages.

The score stopped being a verdict

I was arguing against a score as a verdict. What I had not considered was a score as a position.

This is where I was wrong, and it took me an embarrassingly long time to see it.

I had been arguing against a score as a verdict, a judgment handed down at the end that tells you how you did. That version really is dangerous, and I would still refuse to build it.

What I had not considered was a score as a position: where you are on a path.

That reframe changes the whole design. Automated cleanup alone tops out at 85, a B. You cannot get past it by fixing scanner findings, no matter how thorough you are, because the ceiling is set by how much of the workflow you have completed rather than by how clean your scan looks. When you hit that plateau, the plateau is the message: you have earned everything this stage has to give, so go do the next one.

The top grades open only when the real work is done. Manual testing performed. Flagged items reviewed. No unresolved critical issues.

And the grade never appears on its own. It always carries what it covers, so it reads as “B, automated only” or “A minus, manually tested.” A screenshot of a B cannot be mistaken for a certificate.

Best of all, it moves while you work. Fix a batch of contrast issues and watch it climb. That is the part I had never given any credit to. A number that responds to your effort tells you the effort is working, and people who can see progress keep going.

My first version would have failed everybody

A measurement that is technically correct and makes people give up has failed at the only job it had.

Then Nathan was right a second time.

My first scoring model was ruthlessly honest. Scored strictly against what real accessibility requires, almost nothing clears 30 percent without manual testing. Our own modeling had nearly every site we would ever grade landing on an F.

I thought that was integrity. Nathan said people would look at an F, decide the situation was hopeless, and quit.

He was right. Someone who has poured a year into a site they care about does not read an F as “here is your next step.” They read it as “you failed, and you cannot fix this.” A measurement that is technically correct and makes people give up has failed at the only job it had.

So the model is deliberately generous inside an honest frame. We retired the F completely. Honesty is carried by the workflow gate, by the coverage label attached to every grade, and by the product telling you plainly what is holding you where you are. It is not carried by a punishing number.

What it still will not do

A few things we will not budge on.

It will never say your site is compliant or accessible. Even an A is described as strong progress, because there is no honest way for any tool to certify that a website works for everyone.

It will never show a bare number. The coverage always travels with it.

And it can go down. A new scan finds new issues. A fix regresses. Content ships. A score that only climbs is marketing, not measurement.

The number is not the point

The work is the point. What changed my mind is understanding that a score can point at the work instead of standing in for it.

I spent years worrying about the person who sees a high number and stops. I did not spend nearly enough time thinking about the person who runs a scan, stares at a list of two hundred issues, has no idea what to do next, and quits. That person was never protected by our refusal to show a number. That person was just lost.

Both of those failures come from the same place. Accessibility is genuinely complicated, and any tool that pretends otherwise is going to mislead somebody. A number that says “you are finished” misleads in one direction. A wall of findings with no order to it misleads in the other, because it suggests the work is endless and unwinnable. Neither one is honest about what is actually being asked of you.

So what I want now is narrower than what I was arguing for before, and more useful. Not a number that judges your site. A number that knows where you are standing and tells you what comes next.

I was wrong about this for years, publicly and with a lot of conviction. I am not embarrassed about that. Accessibility has been proving me wrong since the first time somebody told me the sites I was building did not work for them, and after twenty-five years I have learned that being wrong is usually the sign that I am about to learn something new that’s worth knowing. The harder skill is noticing when the thing you are protecting people from is not the thing actually hurting them.

If you have been stuck somewhere in the middle of your own accessibility work, I would genuinely like to hear where. It is the most useful thing anyone tells me.