Running a Local LLM for Lawyers: Ollama, Hardware, and What You Give Up

A practitioner on r/legaltech wrote that they had “bought a strix halo to do this locally with llama 3.3”. A blogger who tested that kind of set-up for legal work in April 2026 reported the other half of the story: “Three of the five cases it gave me either didn’t exist or had holdings that said the opposite.”

Both are true at once, and together they are the whole case for and against a local LLM for lawyers. Nothing leaves your machine, so privilege, § 203 StGB and vendor terms stop being your problem. And the model on your machine is smaller, slower and more confidently wrong than the one in the cloud. For some tasks the trade is worth it, decisively.

Why lawyers consider local: no third party at all

Every cloud tier, however good the contract, involves a third party. The DACH rules make that the entire question: BRAK’s reading of § 203 StGB is that it does not matter whether the provider actually reads the prompts; “Ausreichend ist … dass sie die Möglichkeit dazu haben” (BRAK Hinweise). A model on your own hardware gives nobody that possibility. The Swiss SAV guidance lists on-premise as the first of its three permissible set-ups. In the United States, Florida Bar Opinion 24-1 says that where a tool “does not involve the disclosure of confidential information to a third-party, a lawyer is not required to obtain a client’s informed consent”.

The 2026 case law points the same way. United States v. Heppner turned on a consumer platform’s privacy policy allowing third-party access. Morgan v. V2X (D. Colo., 30 March 2026) amended a protective order to bar AI platforms unless the provider is contractually prohibited from training on inputs and disclosing them, which the court accepted “practically bars the use of most ‘low-to-no-cost’ AI tools” (Akin). A local model has no provider to contract with.

Ollama in fifteen minutes

Ollama is a free command-line tool that downloads open-weight models and runs them on your computer. The set-up, from the April 2026 walkthrough:

# 1. install Ollama from ollama.com
ollama pull llama3.1:8b      # ~5 GB download
ollama run llama3.1:8b       # chat in the terminal

For a ChatGPT-style window instead of a terminal, Open WebUI runs in Docker at localhost:3000; LM Studio is the alternative for people who never want to see a command line. Nothing in this stack sends your documents anywhere; the network cable can be unplugged throughout.

Model sizes: 8B on a laptop, 70B needs 40-48 GB

Open models come in sizes measured in billions of parameters, and size sets both the hardware you need and the quality you get.

The figures are the walkthrough’s, using 4-bit quantised (“Q4_K_M”) models, the normal way to fit them into consumer memory. For text work, “No dedicated GPU needed”; Apple’s unified memory is what makes 70B reachable on a small box.

Hardware budget: the Mac mini and the alternatives

The walkthrough recommends a Mac mini with the M4 Pro chip and 64 GB of unified memory as the practical 70B host and puts it at about $1,400. That is Apple’s launch price of $1,399 for the 24 GB base model; the 64 GB configuration was about $1,999. Budget two thousand dollars, once, no subscription, for a machine that serves a 70B model to a small office over the local network.

Between the two sits sovereign hosting. Noxtua moved its Beck-Noxtua platform into Deutsche Telekom’s AI Factory in March 2026, citing the US CLOUD Act, so a German firm can use a large legal model on German infrastructure without running anything itself. That is the firm-scale answer; the box under the desk is the solo-scale one. The DACH tools guide compares them.

What local models do well

The sweet spot is document work where the answer is in the text you supply and a fluffy sentence costs nothing.

• Anonymisation before anything else. The best single use. Run the placeholder prompt from the anonymisation guide locally, keep the key table, then send the placeholder version to whichever frontier model you are allowed to use.
• Triage and extraction. “Which of these forty emails mention the delivery date?” A small model with a source fence does this well enough, and you check the hits.
• Summaries and plain-English rewrites. The walkthrough’s own prompt list is exactly this: NDA review for unusual clauses, deposition summaries, client emails “in plain, non-technical language”, proofreading a brief.
• Drafting from facts you provide. A demand letter from a fact pattern and a template; a first-draft chronology from a bundle.

You are helping a [jurisdiction] lawyer triage an NDA. Use only the text between the tags. Do not draw on any knowledge of case law or statute, and do not cite any authority.

<nda>
[paste]
</nda>

List, with the clause number and the exact words quoted: (1) the definition of confidential information and any carve-outs; (2) the term and the survival period; (3) any obligation on the receiving party beyond confidentiality (non-solicit, non-compete, exclusivity); (4) remedies and any indemnity; (5) anything one-sided or missing that a receiving party would normally expect. Where a point is not in the text, write "NOT IN DOCUMENT".

From the documents between the tags, build a table with the columns: Document | Date | Parties | One-line subject | Page. Quote dates exactly as written; do not convert or infer them. If a document has no date, write "UNDATED". Do not add documents that are not in the text, and do not summarise beyond the one-line subject.

<bundle>
[paste]
</bundle>

What they do badly: reasoning, long context, hallucination

Reasoning. On LEXam, the Swiss-built benchmark of 7,537 law-exam questions in English and German, GPT-5 scored 70.2 on open questions and Gemini 2.5 Pro 67.4. The open models a lawyer could plausibly run at a desk scored far lower: Apertus-70B 34.7, EuroLLM-9B 22.95, Ministral-8B 14.88. That is not a gap you prompt your way across. A local 8B model is a text tool, not a substitute for legal analysis.

Long context. Chroma’s context-rot study tested 18 models, including open-weight Qwen3 models down to 8B parameters, and found that performance degrades as input grows across the models tested, that a single distractor hurts, and that coherent text is harder to search than shuffled text. A small model has less headroom to begin with, so feed it one document and one question, not the data room; the context window guide shows how to chunk.

Hallucination. Privacy does nothing for accuracy. The walkthrough’s verdict: “Three of the five cases it gave me either didn’t exist or had holdings that said the opposite.” And: “There’s no connection to Westlaw, LexisNexis, or any live database.” In Stanford’s “Large Legal Fictions” study, Llama 2 hallucinated on 88 per cent of legal queries, against GPT-4’s at least 58 per cent. Never ask a local model for authority; the fake cases explainer covers why every model does this.

Local models and § 203 StGB: what changes and what does not

For a German or Austrian lawyer, a local model removes the § 203 StGB and § 43e BRAO problem at its root: no provider, so no possibility of access and no outsourcing contract to draft. ÖRAK’s rule that mandate data may not go into “öffentliche oder ungesicherte KI-Systeme” is satisfied because nothing goes anywhere.

What does not change: § 43 BRAO’s duty of final control, and ÖRAK’s demand for “ausnahmslose Überprüfung und Verifizierung aller KI-generierten Ergebnisse”. The KG Berlin’s Leitsatz in 17 WF 144/25, that lawyers must check whether citations are the product of a “fantasierenden” KI, applies with more force to a small local model, not less. And the AI Act’s Article 4 literacy duty applies to deployers wherever the model runs. The BRAK, DAV and ÖRAK comparison has the detail.

Hybrid: local for the sensitive step, frontier for the rest

The workable pattern for most practices is not “everything local”. It is local for the step that touches identifiable client data, frontier for the step that needs judgement:

1. Strip metadata; anonymise on the local model; keep the key offline.
2. Run the substantive prompt on a frontier model in a no-training tier, or on a legal platform.
3. Re-insert the names locally; verify every authority in a database; log it.

That is Recipe 18 in the prompt library, and roughly where the lawyers building their own tools have landed. Clifford Chance’s Jamie Tso argues that “by asking the AI to build a tool rather than answer a question, the result becomes deterministic, significantly reducing the error rate”; a local model is the natural back end for a self-built extraction tool that must never see the internet. The engineering caveat, from Version Story: lawyers “should not trust apps they vibe-code with their clients’ confidential information unless they have the technical expertise to deploy their tools securely”. The vibe-coding guide takes that further.

Replace every personal name, company name, address, account number, case number and unique identifier in the text below with consistent placeholders ([PERSON_1], [COMPANY_A], [ADDRESS_1], [CASE_NO]). Generalise contextual identifiers that would allow re-identification (unusual job titles, unique events, small towns) to a neutral description. Output two things: the anonymised text, and a two-column key table (placeholder | original). Change nothing else.

<text>
[paste]
</text>

A decision checklist

Where to go next: the anonymisation method is the natural companion, is ChatGPT confidential for lawyers covers the cloud tiers a local model competes with, and the confidentiality hub has the rest. In AI Lab for Lawyers we put a local model and a frontier model side by side on the same anonymised document, so you can see the quality gap yourself before deciding what to run where.