AI-Generated Impact Reports: A Grant-Maker's Guide to Evaluating What Charities Submit
Charities are submitting AI-generated impact reports. Many are doing it openly: they describe using large language models to draft narrative summaries, synthesise programme data, or produce the prose sections of their annual reports. Some are doing it less openly — the finished document looks like it always has, but the writing was assisted, checked, and sometimes heavily shaped by AI tools the charity did not mention.
This is not inherently a problem. AI can genuinely help small charities communicate their work more clearly and efficiently. But it creates a new category of risk for grant-makers: the polished, fluent report that reads like strong evidence but is not grounded in the data it appears to describe.
Understanding how to evaluate AI-generated impact reports is now a practical skill for anyone making funding decisions. This guide covers what the risks actually are, what to look for, and how to use a structured AI process on the funder side to level the playing field.
Why AI-generated reports look more credible than they are
The specific risk with AI-generated content is not that it is poorly written. It is that it is often very well written in ways that mimic the markers of rigour without containing the substance of rigour. A language model will produce fluent sentences about "measurable outcomes" and "robust evaluation methodology" whether or not the charity's underlying data supports those phrases. It will generate plausible-sounding statistics. It will structure arguments to answer the questions a funder is likely to ask.
This is sometimes called hallucination — the model generating confident-sounding content that is not grounded in verifiable facts. In the context of a grant application or impact report, hallucination does not always mean fabrication. It can mean inflation: a 30% improvement in beneficiary wellbeing cited without a source, or a claim about independent evaluation that refers to an internal survey. The text is not false in a way that is easy to spot, but it is not supported in the way it implies.
The practical problem for grant-makers is that the traditional signals of a well-evidenced report — clear structure, confident language, specific numbers — are exactly the signals that AI is good at producing. Surface quality is no longer a proxy for evidence quality.
What to look for: three diagnostic checks
Citation grounding. Does every specific claim in the report point to a verifiable source? Not a general reference to "our monitoring data" or "findings from the evaluation," but a specific document, dataset, or independent study that you could ask for or check. AI-generated reports frequently use the language of citation without the substance: "evidence shows" and "research indicates" appear throughout, but nothing is actually cited. When you see this pattern, ask the charity for the underlying data. The response will tell you a great deal.
Outcome specificity. Impact reports that describe real change tell you who changed, by how much, over what period, measured how. Outputs — "we ran 48 workshops" — are easy for any system to state and easy for any funder to mistake for outcomes. Outcomes require measurement infrastructure: a baseline, a follow-up, a comparator group, or at minimum a validated self-assessment tool. AI-generated prose can describe this infrastructure with great specificity while the actual measurement is far thinner. Ask how the reported outcomes were measured. If the charity cannot describe their evaluation methodology clearly, the numbers in the report are not reliable.
Evidence triangulation. A single self-reported metric is a weak signal. The same finding, corroborated by independent evaluation, a peer-reviewed study, Charity Commission data, and beneficiary testimonials is a strong one. AI-generated reports tend to be constructed from a single source — usually the charity's own data or narrative — and then written to sound multi-sourced. Look for evidence that the claims in the report are consistent with what the charity has filed with the Charity Commission, what its auditor has said, and what other funders have documented in their grant records. Inconsistencies are worth following.
The particular challenge of third-party AI tools
The charity assessment guidance published by most sector bodies was written before the widespread adoption of large language models by small charities. It assumes that the main source of inaccuracy in charity-submitted documents is poor data collection or honest error. It does not account for documents where the language has been generated to fit the expected profile of a strong application.
This creates an asymmetry. Charities with AI tools can produce documentation that reads like evidence of a mature evaluation culture. Grant-makers without their own structured AI process are left comparing documents by surface quality — which is exactly the measure that AI optimises for.
The response to this asymmetry is not to become suspicious of all AI-assisted communication. Most charities using AI tools are doing so to communicate genuinely good work more clearly, not to misrepresent themselves. The response is to build a funder-side AI process that is grounded in verifiable public data — data that the charity does not control.
How a structured funder-side AI process counters these risks
When grant-makers use AI to gather and assess evidence from public sources — the Charity Commission register, filed annual accounts, SORP-compliant financial statements, independent evaluations — they are working from data the charity cannot edit or inflate. The AI does not read the impact report and score it highly because it is well written. It reads the accounts, the filing history, the regulatory record, and the web presence, and scores against criteria the funder has defined in advance.
This approach has three properties that matter in the context of AI-generated submissions. First, it is source-independent: the quality of the charity's own communications does not directly affect the assessment outcome. Second, it is anchored to verifiable data: every scored finding points to a checkable source. Third, it is consistent: the same criteria applied the same way to every charity on the shortlist, regardless of how compellingly any individual submission is written.
You can still read the impact report. You should. But you read it after the structured assessment, which means you read it knowing whether the claims in the report are consistent with what the public record shows. Inconsistency between a charity's self-reported narrative and its filed accounts is one of the more informative signals available to grant-makers, and it is much easier to spot when you have done the structured assessment first.
What to do when you are uncertain
If a charity's impact report contains specific outcome claims that you cannot verify from public sources, ask directly. Request the underlying data. Ask who conducted the evaluation and whether a methodology report is available. Ask whether the impact report was AI-assisted and, if so, which parts.
This is not a hostile question. Most trustees and programme staff at well-run charities will answer it straightforwardly. An organisation that is defensive about its evaluation methodology is telling you something important. An organisation that shares its data willingly — including the parts that show mixed results — is telling you something important too.
The goal is not to penalise charities for using AI. It is to ensure that the documents they submit are grounded in real evidence that can be checked. That standard is no different from the standard that applied before AI tools existed. The methods for meeting it have just become more important to apply consistently.
Building a more reliable evaluation process
The funders best positioned to evaluate AI-generated impact reports are those who have defined, in advance, what evidence they accept, how they grade it, and what weight they give each criterion in their assessment. When that structure exists, an AI-assisted impact report is just another input — one that gets checked against public data, compared to the charity's financial history, and triangulated against independent sources before it influences any decision.
Read more about ClearGiving's evidence grading methodology, or explore what a working AI charity assessment template looks like for grant-makers who want to build this kind of structured process. If you are assessing a specific shortlist and want to see a structured AI assessment in practice, request access to the platform.