A translation pipeline is only as good as its model. We did not pick one by reputation. We ran every chat model on STACKIT AI Model Serving against the same pages and compared the results with the German versions already on this site. Code measured every number below.
The Models on STACKIT AI Model Serving
Section titled “The Models on STACKIT AI Model Serving”STACKIT AI Model Serving offers shared models behind one OpenAI-compatible endpoint in the region eu01. A request and its text stay on STACKIT. Switching the model means changing one name in the request.
The list below comes from the STACKIT documentation and updates itself.
| Model | Capabilities | Context | Max. output | Status |
|---|---|---|---|---|
| Text models | ||||
| Qwen3-VL 235B | Chat, Vision | 200K | 16,384 | Supported |
| Qwen3.8 27B | Chat, Vision, Reasoning | 262K | 16,384 | Supported |
| Qwen3.6 27B | Chat, Vision, Reasoning | 262K | 16,384 | Deprecated |
| Llama 3.3 70B | Chat | 128K | 4,096 | Supported |
| GPT-OSS 120B | Chat, Reasoning | 131K | 16,384 | Supported |
| Gemma 4 31B | Chat, Vision, Reasoning | 256K | 16,384 | Supported |
| GPT-OSS 20B | Chat, Reasoning | 131K | 8,192 | Supported |
| Embedding models | ||||
| E5 Mistral 7B | Embedding | n/a | n/a | Supported |
| Qwen3 Vision-Language Embedding | Embedding | n/a | n/a | Supported |
What is this?
This section is copied from the STACKIT docs automatically, several times a day. It cannot be changed here. Changes belong in the STACKIT docs.
The Test Set
Section titled “The Test Set”- Eleven files: about 62,000 characters. Assets, framework pages, tables, two trails, a how-to, an ISV page and a contributor profile.
- A German twin each: every file already had a German version on this site. It is the reference for the comparison.
- A glossary: derived from 222 existing English and German pairs. It lists the English terms our German keeps, such as “Use Case”.
- Same conditions: every model got the same pieces, the same glossary and the same prompt. Thinking was off.
Eight Chat Models Compared
Section titled “Eight Chat Models Compared”How to read the table:
- Closeness: the chrF score. It cuts the model’s German and our existing German into short letter sequences of up to six letters and counts how many match. 100 means the same text. Two good translations often choose different words, so no model gets near 100. Untranslated English still scores 38.5, because names, product terms and numbers look the same in both languages. That is the baseline to beat.
- Files intact: every piece came back with all its placeholders, so the page could be put together again.
- Pieces left English: text pieces of six words or more that came back unchanged.
- Minutes: model time for all eleven files.
| Model | Closeness | Files intact | Pieces left English | Minutes |
|---|---|---|---|---|
| Qwen3.8 27B | 80.6 | 11 of 11 | 0 | 6.0 |
| Qwen3.6 27B | 79.7 | 11 of 11 | 1 | 4.5 |
| Gemma 4 31B | 79.7 | 11 of 11 | 1 | 5.1 |
| Qwen3-VL 235B | 78.2 | 10 of 11 | 0 | 4.2 |
| Gemma 3 27B | 75.6 | 10 of 11 | 1 | 7.0 |
| GPT-OSS 20B | 75.5 | 10 of 11 | 18 | 1.1 |
| Llama 3.3 70B | 75.3 | 10 of 11 | 0 | 8.5 |
| GPT-OSS 120B | 72.9 | 11 of 11 | 56 | 2.2 |
- The Qwen models lead: Qwen3.8 is closest and keeps all 11 files intact. Four models kept only 10 of 11 files intact, even in small pieces.
- Bigger is not better here: Qwen3-VL 235B scored below the 27B models.
- GPT-OSS is fast but leaves English: up to 56 pieces stayed untranslated.
- Llama 3.3 is the slowest: it also kept only 83 percent of the English terms our German keeps on purpose. Qwen3.8 kept 98 percent.
A Second Model Corrects
Section titled “A Second Model Corrects”In the correction pass Gemma 4 changed 54 of the 545 pieces Qwen3.8 had written. Two of them, shortened. They show the kind of slip it fixes:
| Qwen3.8 wrote | Gemma 4 made it |
|---|---|
| Verwenden Sie das Use-Case-Katalog | Verwenden Sie den Use-Case-Katalog |
| Bestanden mit gemessener Kapazität | Bestanden bei gemessener Kapazität |
Why We Use the Pair
Section titled “Why We Use the Pair”The closeness score compares with one existing version. It cannot tell whether a change makes a sentence better. So for the correction pass we looked at what actually changed.
- Qwen3.8 first: it writes the German closest to ours and keeps all 11 files intact.
- Gemma 4 second: it fixes small grammar slips like a wrong article.
- Not the other way round: Qwen3.8 correcting Gemma 4 changed almost nothing.
- A person last: the reviewer in the pull request has the final word.
Settings That Mattered
Section titled “Settings That Mattered”- Thinking off: Qwen3.8 and Gemma 4 think by default. With thinking on a run took five times as long and used five times the tokens. Gemma 4 ran into its output limit and 165 pieces were missing.
- Short output per call: Gemma 4 and Llama 3.3 write at most 4,096 tokens per answer. Pieces instead of whole files keep answers far below that.
- No deprecated models: Qwen3.6 and Gemma 3 still answer, but Qwen3.6 is marked deprecated and Gemma 3 is no longer in the list. A pipeline should not start on them.
- A tolerant parser: in the correction pass Gemma 4 sometimes returns both languages instead of only the text. The parser accepts both.
The lesson for your own pipeline: measure on your pages with your terms. A public ranking did not predict ours.
Learn More
Section titled “Learn More”Asset historyAdded Oct 30, 2026TMUpdatedNo updates · 1 bar = 1 week i
- TMTobias M.Head of STACKIT Cloud Framework · STACKITOwner
Tobias M.Head of STACKIT Cloud Framework · STACKITOwnerActive 11 of the last 12 weeks · 196 updateswww.linkedin.com/in/tobias-müller-011304172