Only once it was clear at the supplier which data stays local did the technology become a simple question. The data class determines the model, not the other way round. This page is the reference for it, not a starting point.
- The data class determines the model, not the computing power. The starting point is therefore data classification.
- Local means calculable costs instead of ongoing licence fees, plus independence from vendor decisions.
- A small model on existing hardware is enough for most first use cases.
- You need the technical details below only once a concrete case is defined.
This page is a reference, not an introduction. If you have not yet clarified which of your data may leave the company, you are better off starting with the data classification. The choice of model then follows almost by itself.
The data class determines the choice of model, not the data sheet
As soon as it is clear which of your data may leave the company, the technical discussion is almost decided. If a substantial part of your use cases falls into a class worth protecting, local operation is set, and the question of the right hardware remains. If the focus lies with uncritical or already public data, you have more freedom and can also start with a hosted offering.
What does not work is to reverse the sequence and begin with the model. Five terms that keep coming up are explained in more detail by the glossary; here a rough idea of parameters, context window, quantisation and hallucination is enough.
What to select a model by
Freely operable models differ less in pure performance than in properties that are decisive for operation. For a viable selection, a few criteria are enough.
- Licence. Permissive or vendor-bound. It decides whether you may later pass a system on to customers, and it cannot be corrected retroactively.
- Language. How well the model works in German and in your technical terms.
- Origin. Whether European origin is a criterion for you, for example for data-protection or independence reasons.
- Context window. Whether a complete specification fits in one pass or only individual chapters.
- Way of working. A sober model for pure document work or a reasoning one for tasks where something is actually calculated or weighed up.
- Size. Matched to the available hardware, not as large as possible.
Which concrete combination holds up for your case depends on your data class, the use case and your hardware. We work that out together. A public recommendation list would be outdated in a few months and would not do justice to your use case anyway.
Why the licence is a selection criterion
Because it cannot be corrected retroactively. A permissive licence usually requires only the mention of the original copyright and the disclosure of changes. Vendor licences go beyond that.
| Criterion | Permissive licence | Vendor licence |
|---|---|---|
| Purpose of use | No restriction, every lawful use permitted | Use bans, competition or training restrictions possible |
| Revocation | Not possible | Termination on breach usually reserved |
| Subsequent change | Excluded | Reserved, continued use counts as consent |
| Passing on to customers | Only enclose the licence text | Passing the restrictions on to every downstream user |
The change clause is often underestimated. There have been cases in which a provider could later adjust the usage policy of a model advertised as open and thereby restrict usage rights. For a system in a production process, that is an operational risk.
Important: the licence applies per model variant, not per provider. The same provider can licence one variant permissively and another with commercial restrictions. Check the licence text of the exact variant, including the version number. And the licence does not replace regulation: the obligations from the EU AI Act apply independently of the chosen licence.
A finding that makes the decision easier
88 percent of companies state that the country of origin of the AI provider is important to them, and 93 percent prefer German providers. In practice, however, the large majority use US services, while European providers are in the per-mille range.
The gap between 93 percent aspiration and a fraction of that in practice is an unsolved problem, not a preference. This is exactly where local AI comes in.
Source: Bitkom, Artificial Intelligence in Germany, study report 2026.
How reliable are the answers?
The practically most important finding runs counter to intuition. Reasoning models that think in intermediate steps before answering often perform worse at summarising a presented document. They invest computing effort in the thinking-through and deviate from the source material, instead of simply sticking to it.
For your selection that means the following. If the model is only to evaluate existing documents, that is, test protocols, specifications or complaints, a sober instruct model is the better choice. Reasoning belongs where something is actually calculated or weighed up.
The most effective lever is in any case not the choice of model, but the architecture around it. A model that answers exclusively from supplied documents and outputs the source reference produces verifiable results even when it is smaller.
Limitation. Uniformly collected reliability figures are not available for freely operable models. Test with your own documents instead of relying on general rankings.
Does the same model deliver the same result twice?
Not without additional measures, and that surprises many companies. A fixed starting value and a temperature set to zero remove only part of the randomness. Because floating-point calculation is not associative, intermediate results are combined in a different order depending on load, which leads to differing answers.
Reproducibility can be enforced technically, but it costs noticeable speed.
Practical consequence. If results are subject to documentation, store the output together with input, model version and timestamp. Do not rely on being able to produce a result identically later.
Which platform does this run on?
Windows, Linux and macOS all work. For companies with an established Microsoft environment that means you need no Linux setup to start.
For a first pilot, a single computer is enough. As soon as several departments access it simultaneously, server operation that efficiently processes many parallel requests becomes sensible. Anyone without a server room who works with a few people finds an alternative in a well-equipped Apple device. The advantage lies in the memory capacity per device, the disadvantage in the lower speed per token.
Which path is the right one for you depends on your existing environment. That belongs in the planning, not in a blanket recommendation.
Which hardware do you need?
| Class | Rough equipment | Suitable for |
|---|---|---|
| Workstation | Well-equipped computer with sufficient memory | Single person, proof of feasibility |
| Department server | Server with a dedicated graphics card | Five to twenty people |
| Company server | Server hardware capable of sustained load | Company-wide use |
| Apple alternative | Device with large unified memory | One to five people, no server room |
Consumer or server hardware? For continuous operation with several users, server hardware is the better choice, because consumer hardware is optimised for gaming, not for continuous inference. Consumer cards tend to throttle thermally under sustained load and offer no error correction in memory. For a system that a shift depends on, error correction is not a detail.
The most common mistake in memory planning. Some models are split into many specialist areas, of which only a small part contributes to each request. That saves computing power and makes the answer faster, but it saves no memory. All the specialist areas have to be available in memory at the same time, so that the right one can step in at any moment.
A picture for it. Imagine a large specialist library. For a single question, someone opens only two books, which is quick. Even so, all the books have to be on the shelves so that the right one is there for the next question. Anyone who plans the room only for two books has calculated too small. So size the memory by the total size of the model, not by the part that is active per request.
Energy and operating costs
In a region where energy costs are a constant topic anyway, this point belongs in the calculation. A workstation system does not carry weight. A department server with a dedicated graphics card is, under full load, in the range of a small workshop machine, but typically does not run continuously under full load, only during requests. An upper-class company server needs suitable cooling and a location that absorbs the waste heat.
In the profitability assessment, calculate with acquisition, energy, cooling and the personnel effort for operation. The last item is forgotten most often and is often the largest.
The personnel effort
A local system needs someone to look after it. Installing updates, exchanging models, maintaining the document base, answering users' questions, checking logs.
The weighing-up looks like this. With a cloud service you carry the risk of data disclosure, but the provider operates the protection. With local operation the disclosure risk disappears completely, but you take on patch management, hardening, monitoring and failover yourself.
If this capacity is not available, a hosted model with a European provider is the more sensible choice than a self-operated system that no one maintains.
An argument that is often missing
Besides data protection, there is a second argument for freely operable models that is immediately understood in manufacturing companies, namely independence.
A model that you have downloaded still works even when a provider changes the pricing model, discontinues a product, no longer serves a region or adjusts the terms of use. Anyone who thinks in terms of spare-part availability over twenty years understands this argument immediately.
Infrastructure and security
| Expansion level | Infrastructure | IT security |
|---|---|---|
| Level 1, Pilot | Single computer, no access to specialist systems | Documented rule on which data may be entered |
| Level 2, Department | Server in its own network zone, connection to document storage | Own network segment, login via existing directory, logging |
| Level 3, Company-wide | Hardware capable of sustained load, separate test and production environments | Authorisation concept, input and output checking, penetration test |
| Level 4, Production-near | Separation of office and production network, operation without internet possible | No return channels, separate approval of every connection |
Five points belong in from the start, because they can hardly be retrofitted later.
- Network segmentation. The AI server belongs in its own zone with documented connections. A system that simultaneously sees the internet, the office network and the production network is, in the event of damage, the bridge between all three.
- Logging. Input, output, user, timestamp, model name and model version. This is at once the basis for traceability and for an incident report.
- Access rights via the data layer. A language model has no rights management of its own. Whoever has access sees everything the connected source delivers.
- Model integrity. Document the checksum of the downloaded weights, record the source, treat updates as a planned change.
- Output control before automation. As soon as an answer automatically triggers an action, you need a check stage in between.
The jump from level 1 to level 2 is the expensive one. Up to then you need no new security concept, only a documented usage rule. From level 2 the system becomes part of your infrastructure and thus part of the audit obligation.
Where do you get the models?
Two notes that are regularly overlooked in companies.
Download weights exclusively from the provider's official account. For every well-known model there are numerous copies and quantisations from third parties. These are not bad per se, but they are not signed and not checked by the manufacturer.
Note, for every deployed model, the exact version designation, the quantisation format and the checksum of the downloaded file. Without this information it cannot be proven later which system produced a particular output.
If you do not want to operate it yourself, European hosters offer a middle way between cloud convenience and data control. Anyone who prefers local operation turnkey rather than self-built finds a further option in a preconfigured AI appliance. Which source and which mode of operation suits you, we clarify in conversation.
Frequently asked questions
Do we need Linux?
No. Windows, Linux and macOS all work. Linux has advantages for high throughput and reproducible inference, but it is not a prerequisite for getting started.
Will it run on our existing Windows Server?
In principle yes, provided a suitable graphics card is installed. For established Microsoft environments there is a dedicated path that requires no Linux setup.
Is a normal office computer enough?
For a test with a smaller model in economical quantisation, a well-equipped workstation is enough. Not for productive multi-user operation.
Does a Mac make sense for an industrial company?
If there is no server room and one to five people work with it, yes. The advantage is the memory capacity per device, the disadvantage the lower speed per token.
Can we use a gaming graphics card?
For tests, yes. For continuous operation, server hardware is the better choice, because consumer cards throttle thermally under sustained load and have no error correction in memory.
Can we operate the system without an internet connection?
Yes. You need internet access only for the initial download of the weights and for updates. Both can be designed as a controlled process.
How long does a model stay current?
The market moves in months, not years. Build applications so that the model remains exchangeable.
The market for language models and their licensing terms changes quickly. Before you decide on a specific model, check its licence and terms of use at the source.
Which models and which hardware hold up for your case is best clarified on the basis of your data classes. Arrange an appointment or first look at the data classification.