---
title: "Translating With STACKIT AI Models: What We Measured"
description: "Eight chat models on STACKIT AI Model Serving, one test set from our own pages. Which model translates, which one corrects and which settings mattered most."
scfAsset:
  managed: false
  category: "guide"
  external: false
  tags: ["translation", "STACKIT AI", "Model Serving", "model choice", "evaluation", "Qwen", "Gemma"]
  maintainers:
    - user: "tobias.mueller"
source_url: "https://framework.stackit.cloud/data-and-ai/assetcontainer/stackit/translation-pipeline-stackit-models/"
source_file: "docs/data-and-ai/assetcontainer/stackit/translation-pipeline-stackit-models.mdx"
---

A translation pipeline is only as good as its model. We did not pick one by reputation. We ran every chat model on STACKIT AI Model Serving against the same pages and compared the results with the German versions already on this site. Code measured every number below.

## The Models on STACKIT AI Model Serving

STACKIT AI Model Serving offers shared models behind one OpenAI-compatible endpoint in the region eu01. A request and its text stay on STACKIT. Switching the model means changing one name in the request.

The list below comes from the STACKIT documentation and updates itself.

> From the STACKIT docs: [Available Shared Models of STACKIT AI Model Serving](https://docs.stackit.cloud/products/data-and-ai/ai-model-serving/basics/available-shared-models/) (Source updated 04.11.2026, copied 04.11.2026)

| Model | Capabilities | Context | Max. output | Status |
| --- | --- | --- | --- | --- |
| Qwen3-VL 235B | Chat, Vision | 200K | 16,384 | Supported |
| Qwen3.8 27B | Chat, Vision, Reasoning | 262K | 16,384 | Supported |
| Qwen3.6 27B | Chat, Vision, Reasoning | 262K | 16,384 | Deprecated |
| Llama 3.3 70B | Chat | 128K | 4,096 | Supported |
| GPT-OSS 120B | Chat, Reasoning | 131K | 16,384 | Supported |
| Gemma 4 31B | Chat, Vision, Reasoning | 256K | 16,384 | Supported |
| GPT-OSS 20B | Chat, Reasoning | 131K | 8,192 | Supported |
| E5 Mistral 7B | Embedding | n/a | n/a | Supported |
| Qwen3 Vision-Language Embedding | Embedding | n/a | n/a | Supported |

## The Test Set

- **Eleven files**: about 62,000 characters. Assets, framework pages, tables, two trails, a how-to, an ISV page and a contributor profile.
- **A German twin each**: every file already had a German version on this site. It is the reference for the comparison.
- **A glossary**: derived from 222 existing English and German pairs. It lists the English terms our German keeps, such as "Use Case".
- **Same conditions**: every model got the same pieces, the same glossary and the same prompt. Thinking was off.

## Eight Chat Models Compared

How to read the table:

- **Closeness**: the chrF score. It cuts the model's German and our existing German into short letter sequences of up to six letters and counts how many match. 100 means the same text. Two good translations often choose different words, so no model gets near 100. Untranslated English still scores 38.5, because names, product terms and numbers look the same in both languages. That is the baseline to beat.
- **Files intact**: every piece came back with all its placeholders, so the page could be put together again.
- **Pieces left English**: text pieces of six words or more that came back unchanged.
- **Minutes**: model time for all eleven files.

| Model | Closeness | Files intact | Pieces left English | Minutes |
|---|---|---|---|---|
| **Qwen3.8 27B** | **80.6** | 11 of 11 | 0 | 6.0 |
| Qwen3.6 27B | 79.7 | 11 of 11 | 1 | 4.5 |
| Gemma 4 31B | 79.7 | 11 of 11 | 1 | 5.1 |
| Qwen3-VL 235B | 78.2 | 10 of 11 | 0 | 4.2 |
| Gemma 3 27B | 75.6 | 10 of 11 | 1 | 7.0 |
| GPT-OSS 20B | 75.5 | 10 of 11 | 18 | 1.1 |
| Llama 3.3 70B | 75.3 | 10 of 11 | 0 | 8.5 |
| GPT-OSS 120B | 72.9 | 11 of 11 | 56 | 2.2 |

- **The Qwen models lead**: Qwen3.8 is closest and keeps all 11 files intact. Four models kept only 10 of 11 files intact, even in small pieces.
- **Bigger is not better here**: Qwen3-VL 235B scored below the 27B models.
- **GPT-OSS is fast but leaves English**: up to 56 pieces stayed untranslated.
- **Llama 3.3 is the slowest**: it also kept only 83 percent of the English terms our German keeps on purpose. Qwen3.8 kept 98 percent.

## A Second Model Corrects

In the correction pass Gemma 4 changed 54 of the 545 pieces Qwen3.8 had written. Two of them, shortened. They show the kind of slip it fixes:

| Qwen3.8 wrote | Gemma 4 made it |
|---|---|
| Verwenden Sie **das** Use-Case-Katalog | Verwenden Sie **den** Use-Case-Katalog |
| Bestanden **mit** gemessener Kapazität | Bestanden **bei** gemessener Kapazität |

## Why We Use the Pair

The closeness score compares with one existing version. It cannot tell whether a change makes a sentence better. So for the correction pass we looked at what actually changed.

- **Qwen3.8 first**: it writes the German closest to ours and keeps all 11 files intact.
- **Gemma 4 second**: it fixes small grammar slips like a wrong article.
- **Not the other way round**: Qwen3.8 correcting Gemma 4 changed almost nothing.
- **A person last**: the reviewer in the pull request has the final word.

## Settings That Mattered

- **Thinking off**: Qwen3.8 and Gemma 4 think by default. With thinking on a run took five times as long and used five times the tokens. Gemma 4 ran into its output limit and 165 pieces were missing.
- **Short output per call**: Gemma 4 and Llama 3.3 write at most 4,096 tokens per answer. Pieces instead of whole files keep answers far below that.
- **No deprecated models**: Qwen3.6 and Gemma 3 still answer, but Qwen3.6 is marked deprecated and Gemma 3 is no longer in the list. A pipeline should not start on them.
- **A tolerant parser**: in the correction pass Gemma 4 sometimes returns both languages instead of only the text. The parser accepts both.

The lesson for your own pipeline: measure on your pages with your terms. A public ranking did not predict ours.

## Learn More

<LinkCard title="Pieces and Guard Rails: Keeping a Model in Its Lane" href="/data-and-ai/assetcontainer/stackit/translation-pipeline-guard-rails" />

<LinkCard title="AI-Assisted Translation: Flow and Human Review" href="/data-and-ai/assetcontainer/stackit/translation-pipeline-flow" />

<LinkCard title="Available shared models on STACKIT AI Model Serving" href="https://docs.stackit.cloud/products/data-and-ai/ai-model-serving/basics/available-shared-models/" />
