How to Choose Legal AI Without Falling for Hype: The Seven Questions

Choosing a legal AI tool requires seven checks: disaggregated sources (which ones, from what jurisdiction, updated as of when), verification with a link to the official source, benchmarking with a public methodology, contractual data destination, provider's declared limitations, due diligence record, and exit conditions. Peer-reviewed evidence (Universidad de Stanford, Journal of Empirical Legal Studies, 2025) shows that specialized tools reduce errors compared to generalist chatbots —in their assessment, between 17% and 33% failure rates versus 43% for generalists— but do not eliminate them: that's why "zero hallucinations, guaranteed" is the first red flag. In this article: the seven detailed questions to bring to your commercial meeting, and why a Spanish court has already formally established the cost of due diligence.

When you're pitched a legal AI tool, the salesperson's first sentence tells you almost everything. If it starts with 'zero hallucinations, guaranteed,' you can probably cut the meeting short. Not because the tool is necessarily bad. But because anyone promising what published evidence doesn't support is asking you for faith — and in law, we don't work with faith. We work with sources.

What the evidence says (the real, peer-reviewed kind)

The reference academic study was published by Stanford University in a peer-reviewed legal journal. It measured AI-powered legal research tools — the specialized paid ones, not generalist chatbots — and found two things simultaneously:

The good news: specialized tools significantly reduce errors compared to a general-purpose chat.

The uncomfortable truth: they don't eliminate it. In their measurements, the specialized models failed in a range of 17% to 33% of cases, compared to 43% for the generalist model they used for comparison.

Honest disclosure about scope: they measured American tools, concerning Anglo-Saxon law. I am not aware of an equivalent measurement for the Spanish market published with open methodology. This is not reassuring: it means that here we buy with less information — and that the following questions are more necessary, not less.

Wait: what is a 'wrapper'?

The word you need to know before the list. A wrapper is a product that merely wraps a third-party model —the same chatbot you already use for free— with a layer of legal marketing on top, without relevant proprietary technology: no verified corpus, no anchoring to official sources, no logging. It's not illegal to sell it. But you should know what you're buying, because the price of the paint job is sometimes significant.

The seven questions (take them to the meeting)

1 · What sources does it work with — which ones, from what jurisdiction, updated as of when?

'Millions of documents' is not an answer: it's a headline. A serious answer breaks down: what types of documents, what validity, how it's updated.

2 · Does each statement link to its official source?

The five-second test: you ask for an answer with case law and check if each citation opens the official resolution or if you have to search for it manually and hope for the best.

3 · Do you publish a benchmark with methodology — or do you ask me to just believe it?

A serious benchmark states what was measured, how, with how many cases, and where it fails. A round number without methodology is marketing with decimals.

4 · Where does my data go?

Where are they processed, if they are used for training, who can view them, how they are deleted. Contractually, not verbally. (If you read the article on AI and professional secrecy, you know why this question is non-negotiable.)

5 · What limitations do you declare?

A serious tool publishes what it DOESN'T do. One that 'does everything' — search, drafting, accounting, and even coffee — usually doesn't do anything well. Admitting limitations is the best sign of honest engineering there is.

6 · What record is kept of my due diligence?

The day someone asks you if you verified — and this series started with a court asking exactly that — do you have a record, or do you rely on memory?

7 · What happens if I leave?

Data export, format, deadline, cost. Ease of exit is the best measure of a provider's confidence in its product.

The price of due diligence (set by a court, not a vendor)

You might recall the details of the case of the 48 non-existent rulings: when the Canarian court calculated the fine for the lawyer, it referenced the cost of reliable legal AI tools — it stated 'plans from 70 euros per month', according to the official statement from the Judiciary.

So the question of price is framed incorrectly. It's not 'Is 70 euros a month expensive?' It's: compared to what? Compared to the hours you lose manually checking what a free chat gives you? Compared to the day you don't check? A court already did that calculation in writing. It came out in favor of the tool.

Market price references change; the figure cited is what the court order itself stated in writing in February 2026 — this article does not recommend any price or provider.

The part where this examination tests me

I'll be transparent: for weeks I've been telling you I'm building an AI-powered workflow system for law firms. These seven questions also apply to me. What I'm building will have to answer them in writing — broken-down corpus, benchmark with public methodology, data architecture, and logging — or it won't deserve to be sold. Publishing the exam before taking it is the 'no smoke and mirrors' part that can't be faked.

Frequently Asked Questions

What is a 'wrapper' in legal AI?

A product that merely wraps a third-party model —the same generalist chat as always— with a layer of legal marketing, without relevant proprietary technology: no verified corpus, no anchoring to official sources, no activity logging. It's not illegal to sell it, but it's wise to know what you're buying: the seven questions in this article can detect it in a meeting.

What is the error rate of legal AI tools?

The only peer-reviewed academic measurement (Stanford, Journal of Empirical Legal Studies, 2025) found between 17% and 33% errors in specialized paid tools, compared to 43% for the generalist comparison model — regarding American tools and Anglo-Saxon law. They reduce error; they don't eliminate it. No equivalent measurement with open methodology exists for the Spanish market.

What should a serious legal AI benchmark include?

What was measured, how it was measured, with how many cases, of what types, and where the tool fails. A round percentage without published methodology is not a benchmark: it's marketing with decimals. If the provider cannot show the methodology, the honest answer is that they don't have one.

How much does serious legal AI cost?

Market references change, but there's a figure with special value: when calculating a penalty for fabricated citations, the High Court of Justice of the Canary Islands referred to tools with 'plans starting from 70 euros per month' (court order from February 2026). More useful than the absolute price is the comparison: how much does the alternative cost in hours of manual verification — or on the day it's not verified.

What should I ask about my data before contracting?

Four things, in writing and not verbally: where they are processed (jurisdiction), if they are used for training (and how to opt out), who can access them, and how and when they are deleted — including export if you decide to leave. Ease of exit is the best measure of the provider's confidence in its own product.

Sources for this article

• Magesh et al., Journal of Empirical Legal Studies (Stanford, 2025).
• Official note from the Judiciary (order from the TSJ de Canarias).
• From this series: the 48 non-existent rulings and AI and professional secrecy.

Sources verified on Jul-6-2026. This blog doesn't ask for faith: it links to the source.

Every two weeks I publish an analysis like this in the PYME Digital PRO newsletter: digitalization for law firms, consultancies, and professional SMEs, without the hype. Subscribe on LinkedIn.