AI Built My Exam System — Until It Silently Broke

The Promise Was Perfect — At First

When developers reach for AI to accelerate software projects, the early results often feel like magic. Structured outputs look correct, test cases pass, and everything appears production-ready. But a growing number of engineers are discovering that LLMs harbor a subtle and dangerous flaw: they are almost right, almost every time — and that 'almost' can be catastrophic.

One developer's recent account of building an AI-powered exam system illustrates this problem with uncomfortable clarity. The project didn't even start as an exam platform. It began as an experiment in using AI to generate structured data for APIs. And for a while, it worked beautifully.

When 120.5 Is Not 120.50

The first cracks appeared in production, not in testing. The AI-generated responses were returning values like 120.5 instead of 120.50. To a human reader, these are identical numbers. To a downstream system expecting a strict two-decimal-place format, they are entirely different strings — and cause silent failures or outright rejections.

This is the kind of bug that doesn't show up in casual review. It passes eyeball tests. It even passes many automated tests unless they are specifically designed to catch formatting precision. The developer described these as 'small' issues, but in production, small issues compound into system-wide unreliability.

The core problem is deceptively simple: LLMs are probabilistic text generators, not deterministic data formatters. They don't 'understand' that a financial API requires exactly two decimal places. They produce what statistically looks correct based on training data — and most of the time, that is good enough. Until it isn't.

Why Structured Output From LLMs Remains Fragile

This experience points to a broader industry challenge that companies like OpenAI, Anthropic, and Google are actively working to solve. OpenAI introduced 'Structured Outputs' for its GPT-4o API in mid-2024, allowing developers to define JSON schemas that the model must follow. Anthropic's Claude offers similar tool-use and structured output capabilities. Google's Gemini API supports response schemas as well.

But even with these guardrails, edge cases persist. Schema enforcement can guarantee that a field exists and is the right type, but it cannot always guarantee the precise formatting of a string value or the semantic correctness of the content. A number returned as a float instead of a formatted string, a date in ISO 8601 instead of a locale-specific format, an enum value with slightly different casing — these micro-failures are the landmines of AI-driven data pipelines.

Developers building production systems on top of LLM outputs are essentially introducing a non-deterministic component into what is typically a deterministic workflow. Traditional APIs return the same output for the same input every time. LLMs do not.

The Exam System: A Case Study in Compounding Errors

When this same approach was applied to building an exam system — generating questions, answer options, scoring rubrics, and structured metadata — the fragility multiplied. Each question required multiple fields to be precisely formatted. A misaligned answer key, a scoring value off by a rounding artifact, or a malformed question ID could corrupt an entire exam session.

The developer reported that the system 'worked' in demos and early testing but began producing strange, hard-to-trace errors once real users interacted with it at scale. This is a pattern familiar to anyone who has shipped LLM-powered features: the demo is flawless, the production deployment is a debugging nightmare.

What This Means for AI-Powered Development

The lesson here is not that AI is useless for building software — far from it. The lesson is that LLM outputs require a validation layer that is at least as rigorous as the generation layer. Engineers need to treat AI-generated data with the same suspicion they would treat user input: validate everything, trust nothing implicitly.

Several best practices are emerging in the developer community:

• Schema validation on every response — using tools like Pydantic, Zod, or JSON Schema validators to catch formatting issues before they reach downstream systems.
• Deterministic post-processing — reformatting AI outputs through traditional code to enforce exact specifications.
• Regression test suites focused specifically on output format, not just content correctness.
• Fallback mechanisms that retry or substitute when validation fails.

Outlook: Reliable AI Outputs Are an Unsolved Problem

As AI becomes embedded in more critical workflows — from exam platforms to healthcare data pipelines to financial reporting — the tolerance for 'almost right' drops to zero. The industry is moving toward better constrained decoding, stricter output schemas, and hybrid architectures where LLMs handle creativity while deterministic code handles precision.

But today, the gap between what AI produces and what production systems require remains one of the most underappreciated risks in software engineering. The developers who succeed will be the ones who plan for failure — not just hope the AI gets it right.