Self-hosting an LLM for business applications: lessons learned

“Sovereign AI” is turning into a sales pitch stuck on anything and everything. So let us approach the subject differently: for several months, three of our production applications (the assistant in our CRM, a back office for a non-profit and a lead qualification tool) have been running on a language model we host ourselves, on our own machine. Not a demo, not a proof of concept: production, with its possible outages and its pitfalls discovered along the way. Here is what it really looks like.

Why put ourselves through this? #

• Data, first of all. A CRM assistant sees your customer file go by; a qualification tool reads tenders and sales responses. Sending that to an American API raises the processing and transfer questions we covered in our article on the Cloud Act. Locally, nothing leaves and the question disappears.
• Predictable cost. An API bills per token, and a chatty assistant plugged into a CRM consumes quickly. A machine that has been paid off costs the same whether it handles ten requests or ten thousand.
• Independence. No model deprecated overnight, no changing terms, no surprise quota. The model we validated is the one that runs, full stop.

And honesty demands the counterpoint: an open-weights model with around thirty billion parameters is not a latest-generation large commercial model. For complex reasoning or demanding writing, frontier models stay ahead. Self-hosting wins on well-defined tasks (summarising, extracting, classifying, qualifying, calling tools), which in practice make up most of the AI needs of a business application.

The architecture: one machine, one gateway, three consumers #

The setup has three layers. A dedicated GPU machine runs the model through llama.cpp (llama-server), which exposes an OpenAI-compatible API, the de facto standard every client can speak. In front of it, a gateway (OpenWebUI in our case) centralises what the model server does not handle: accounts and API keys per application, a strict access list (an unknown caller gets a 404, not a login screen) and logging. The applications only know the gateway, never the GPU machine: we can change model, machine or engine without touching the consumers’ code.

The pitfalls theory does not predict #

Structured outputs that drive the model mad #

Tool calling (an LLM calling your application’s functions) relies on JSON schemas. llama.cpp converts these schemas into a formal grammar (GBNF) that physically constrains every generated character. And some perfectly valid schemas produce trapped grammars: an empty sub-schema additionalProperties: {} (generated by very widespread validation libraries) is interpreted as “object required”: the model becomes unable to emit an integer or an array at that point, returns absurd calls, the application rejects them, the model tries again… an infinite loop. Diagnosis: hours. Fix: a filter on the gateway that rewrites schemas before they reach the engine. The moral: when you self-host, the boundary between your schemas and the generation engine is yours, for better and for worse.

Temperature matters more than the prompt #

Our qualification tool scores sales opportunities. First finding in production: two runs on the same data differed on average by 30 points out of 100, enough to tip a decision. We first refined the prompt, without much effect. The real lever was elsewhere: the generation temperature, lowered from 0.2 to 0. Average difference afterwards: less than one point. For any reproducible scoring task, this one-line setting weighs more than hours of prompt engineering, and it is the kind of thing you only discover by measuring.

The single point of failure, accepted and monitored #

A single GPU machine means a single point of failure. We accept it, because the degraded mode is acceptable: if the model goes down, the CRM loses its assistant, not its data or its functions. It is an explicit trade-off: monitoring on it, documented criticality, and no vital function plugged into the LLM. The general rule: self-hosted AI should enrich your applications, never be a condition for them to work.

Where to start #

1. A well-defined task with clear value: summarising records, extracting fields, sorting e-mails, qualification. Not “a chatbot that does everything”.
2. A single well-chosen model, evaluated on your data, rather than a zoo of models impossible to maintain.
3. A gateway from day one: authentication, allowlist, logging, and the layer of indirection that will let you change everything behind it.
4. Measurements: reproducibility, latency and error rate are observed, not assumed.

This is exactly the scope of our AI and language model integration service: connecting a model (self-hosted by us in France, or an API when that is the right choice) to your existing applications, with the guardrails above. A repetitive task that deserves a trial? Describe it to us.