After ten years of building models with finance teams, I’ve seen the same problem repeatedly: an output can look right and still be wrong.

I tested this with a cheaper, faster LLM, asking it to build a 12-month cash flow forecast based on several variables. The result looked passable at first, but when I checked the calculations, I found a significant error. I specified monthly sales growth, but the LLM applied that growth once for the entire year. By December, revenue was about 16% too low. Two errors happened to offset, and ending cash was within 1% of the correct figure.

If you were reviewing the forecast by looking at the headline number (ending cash) you could reasonably conclude that the model was working. It wasn’t.

1. Affordable models fail

As a first step, we ran the same forecasting task twenty times on two different AI models at three task complexity levels, then checked every cell by recomputing it instead of just reading the output.

Once the task starts to resemble real work, the cheaper model is wrong about half the time.

When you use generic AI systems, you don't always get to pick the LLM, and the money saved on cheaper models results in losses from errors. At this point, the obvious takeaway is to just utilize the frontier LLMs. So we did, and it still didn’t solve the accuracy problem.

2. Frontier models alone don’t solve the accuracy problem

We gave the frontier LLM a more difficult task: to verify a full operating financial model. We ran this analysis twenty times. Eighteen came back correct, one came back clearly broken, and the last one balanced on every statement while letting cash fall below the floor its own revolver existed to protect.

These outputs all look the same - even those with errors. This wasn’t a particularly complicated model, either; our test model is simpler than what real teams rebuild every month.

"Usually right" isn’t good enough for your financial forecast. Nothing short of 100% is a passing grade.

Choosing the right LLM is the baseline. What matters is what happens after the file exists, so we tested what checking one actually takes.

3. Clean spreadsheets can be wrong

We built complete, linked financial models, planted an error in each, and tried the three mechanisms people use to "verify" forecasts.

The first looks at whether the file looks clean. The second hands the file to another LLM, which catches the broken analyses but also raised incorrect objections about a quarter of the time.

Only the third works: rebuild the math independently, then confirm the statements agree. It found every planted error.

Summation runs an independent check on every file it produces.

How Summation is Different

A few core components have to be true in any financial model. This is true no matter how nuanced and complicated the model is.

Summation checks these core attributes on every output. It understands your data, but doesn't take your source material or existing models at face value, and if it can't verify an output, the output doesn't get our verified badge.

Any LLM can build a financial model. The argument is about whether it is correct. Think about the last model you acted on: the one behind a new hire, an adjusted price, a loan term, or a number you gave the board.

What actually convinced you it was right? If the answer is that it tied out, make that your baseline, then let Summation run those checks outside the AI.

We built Summation to be the AI analyst you can trust. Want to give it a try? Visit https://www.summation.com/ai-analyst

—

Chris Kim is a Forward Deployed Data Scientist at Summation. He has 10 years of data science experience at companies like Coinbase and Oscar Health. He holds an M.S. in Computer Science (AI/ML) from Georgia Tech and a B.S. in Information Systems from NYU Stern.