Government AI Solutions Have to Explain Themselves

For almost two years, the state of Michigan ran an automated system that decided who had committed unemployment fraud, and it ran mostly on its own, without a person reviewing its calls. When the state finally examined the results, it found the system had made determinations with an error rate around 93%. At least 20,000 residents had been falsely accused of fraud they had not committed. Some faced Michigan's highest-in-the-nation quadruple penalties, and others had wages garnished or tax refunds seized.

The software was not exotic, and the story is not really about the algorithm's accuracy. What failed was that a life-altering decision could be made, enforced, and collected on before the person on the other end could see how it was reached or challenge it in time. That is the part of government AI that gets discussed least and decides the most.

Most conversations about government AI solutions open with the accuracy question, whether the model is good enough. Michigan's system failed that test too, and for two years nothing around it could surface the failure. In the public sector, every automated decision belongs to the public. A resident can appeal it. A reporter can request the records behind it. An auditor can ask, three years and one administration later, exactly why the system did what it did. A decision that is correct but unexplainable still fails all three of those people.

We build production AI and custom software, and the public sector is a deliberate part of that work. We've delivered for the NYPD, New York City's Department of Youth and Community Development, and the Port Authority of New York and New Jersey, and we built Vercor, an AI system that helps teams respond to government RFPs. So we've watched government work from both sides of the procurement line, as the vendor answering the solicitation and as the team delivering systems inside an agency's environment. What that second job requires is different in kind from an enterprise build, not just stricter, and this piece is about that difference.

In government, an answer no one can question is the failure mode

When an enterprise AI system is wrong, it usually costs money or time, and you catch it in evaluation and fix it. When a government system is wrong, it can revoke someone's benefits, flag them to law enforcement, or deny them a permit, and the person on the receiving end has a legal right to understand the decision and contest it. That changes the design question. It stops being only how often the system is right and becomes whether, when it acts on a person, that person and later an auditor can see why and push back before the harm is permanent.

The federal government has put this expectation in writing twice over. The U.S. Government Accountability Office already published an accountability framework for federal AI organized around governance, data, performance, and monitoring, which comes down to a single expectation. A government system has to be able to answer for itself at every stage.

In 2024 the Office of Management and Budget went further and required federal agencies to complete an impact assessment before deploying any rights-impacting AI, naming law enforcement and eligibility for public benefits among the categories that count. Its 2025 replacement, M-25-21, kept the substance under a new label.

Agencies still have to complete an impact assessment before deploying "high-impact" AI, still have to give affected people a route to timely human review and appeal, and have to stop using a system whose risks cannot be brought down. The rules are federal, but the decisions they cover, the ones where an automated output becomes the principal basis for an action with a legal or material effect on rights or safety, are the same high-stakes public decisions that state and local agencies make every day.

The production discipline that keeps any AI honest, the repeatable evaluation and failure handling I covered in why AI projects fail before production, still applies here. Government just adds a second audience that the private sector rarely has to satisfy. The system answers not only to the operator who uses it but to the citizen it acts on and the public that can demand to see the receipts.

That audience is, more and more, not even a person. One query that reached our Discord case study this spring read "evaluate the team communication software company discord on trust & security," a string only an automated screen would produce, treating the case study as vendor evidence. A government system should expect its decisions to face the same kind of reader, one that cannot be pulled aside and walked through the reasoning.

The accountability envelope

The useful way to think about a public-sector build is that the model sits inside an envelope of guarantees the system owes to people who never log in. Get the envelope right and the accuracy has somewhere safe to live. Skip it and you are one bad batch of determinations away from the Michigan story. The workflows inside the envelope are ordinary. A case-file summarizer drafts behind a human approval gate. A records search returns only what the requester's role allows. An eligibility screen writes an appeal-ready record for every determination it makes. Four guarantees make up the envelope.

There is also a version of this work we would rather not build at all. If the ask is for the model to be the adjudicator of record, deciding a benefit or flagging a person on its own authority with no appealable human decision behind it, the right move is to decline the contract. Michigan is what that system looks like a few years in.

What we got wrong on Vercor, and what fixed it

When we first built Vercor, our instinct was to make the drafting smarter, with a better model, better prompts, cleaner prose. It produced responses that read well, and it made a good demo. Then we paid attention to how proposal reviewers actually judged the output, and they were not grading the writing. They were checking whether each claim was backed by something real, a past project, a named credential, an actual capability, because a buyer can tell a specific answer from a confident but empty one. Our polished drafts, pulled from thin source material, read as exactly that.

So we moved the effort off the model and onto the evidence, which is the discipline we wrote about at the time, that the knowledge base matters more than the model. We broke prior proposals, capability statements, and resumes into retrievable pieces so the system could pull the firm's own experience for each requirement, and the final stage existed only to catch the gaps before a human reviewer did. The improvement showed up in whether a reviewer could trust and defend what came out.

The same shift, from a better model to the evidence behind it, is why explainability comes first in the envelope. In government, an output a caseworker cannot trace back to something real and defensible is one they are right not to act on. Grounding a decision in evidence a person can inspect is what makes the system usable on the first day, long before any auditor asks.

The system has to outlive the people who bought it

Here is the guarantee that separates a government build from an enterprise one more than any security control. An enterprise system serves a company that intends to keep running it. A government system serves an office whose leadership turns over on a schedule, under a contract that ends, subject to records laws that outlast both.

The decision your system makes this week might be requested under freedom-of-information law next year, litigated in three years, and audited by people who were nowhere near the original project. It still has to explain that decision to all of them. And the vendor who built it, us or anyone else, should be replaceable without the system going dark. That is why handoff-ready, in the public sector, is a continuity requirement rather than a courtesy line in a proposal. It means documented code, data the agency owns outright, and a system a different team could pick up and operate. A government AI system that only its original vendor can explain has recreated the accountability gap in a new place.

None of this starts until you clear procurement

Everything above is the delivery. Getting the chance to deliver at all means clearing procurement, which runs on its own logic, and that logic exists for reasons worth respecting. Participation goals widen the pool of firms doing public work, contract vehicles let agencies buy without starting from zero each time, and the rules keep the process fair. These are fair requirements, and they are where certifications change who is eligible to bid.

We hold NMSDC minority-business certification along with New York City and New York State MBE and DBE certifications, which can count toward the participation goals agencies and prime contractors have to meet, and we work with public-sector distributors like Carahsoft. On their own, certifications tell you who is eligible, and the accountability work is what tells you who is capable. If you are still deciding whether a custom build is even the right route before you get to any of this, what a custom AI development company actually does is the place to start.

Who the system is really for

The people who sit in the room evaluating a government AI project are almost never the people it is built for. The system belongs to the resident who receives a decision and needs to understand it, and to the successor who inherits it after everyone who chose it has moved on. Design for those two and you will have built the explanation, the record, and the handoff that make accuracy something you can prove rather than something you assert.

Design instead for the demo in the procurement meeting, and you can end up with Michigan's system, which in the narrow sense of running and producing outputs was working exactly as deployed. What was missing was any way to question what it produced. In the public sector, a system you cannot question is not a finished product. It is an unfinished one that happens to be in production.

If you have a public-sector workflow in mind and want it built to answer for itself, that is the work our AI and intelligent automation practice is set up to take on.

Frequently asked questions