Skip to content
Beta

Translating With STACKIT AI Models: What We Measured

In 1 trail

Last updated on

A translation pipeline is only as good as its model. We did not pick one by reputation. We ran every chat model on STACKIT AI Model Serving against the same pages and compared the results with the German versions already on this site. Code measured every number below.

STACKIT AI Model Serving offers shared models behind one OpenAI-compatible endpoint in the region eu01. A request and its text stay on STACKIT. Switching the model means changing one name in the request.

The list below comes from the STACKIT documentation and updates itself.

From the STACKIT docsAvailable Shared Models of STACKIT AI Model ServingSource updated 04.11.2026 · copied 04.11.2026
What is this?

This section is copied from the STACKIT docs automatically, several times a day. It cannot be changed here. Changes belong in the STACKIT docs.

  • Eleven files: about 62,000 characters. Assets, framework pages, tables, two trails, a how-to, an ISV page and a contributor profile.
  • A German twin each: every file already had a German version on this site. It is the reference for the comparison.
  • A glossary: derived from 222 existing English and German pairs. It lists the English terms our German keeps, such as “Use Case”.
  • Same conditions: every model got the same pieces, the same glossary and the same prompt. Thinking was off.

How to read the table:

  • Closeness: the chrF score. It cuts the model’s German and our existing German into short letter sequences of up to six letters and counts how many match. 100 means the same text. Two good translations often choose different words, so no model gets near 100. Untranslated English still scores 38.5, because names, product terms and numbers look the same in both languages. That is the baseline to beat.
  • Files intact: every piece came back with all its placeholders, so the page could be put together again.
  • Pieces left English: text pieces of six words or more that came back unchanged.
  • Minutes: model time for all eleven files.
  • The Qwen models lead: Qwen3.8 is closest and keeps all 11 files intact. Four models kept only 10 of 11 files intact, even in small pieces.
  • Bigger is not better here: Qwen3-VL 235B scored below the 27B models.
  • GPT-OSS is fast but leaves English: up to 56 pieces stayed untranslated.
  • Llama 3.3 is the slowest: it also kept only 83 percent of the English terms our German keeps on purpose. Qwen3.8 kept 98 percent.

In the correction pass Gemma 4 changed 54 of the 545 pieces Qwen3.8 had written. Two of them, shortened. They show the kind of slip it fixes:

The closeness score compares with one existing version. It cannot tell whether a change makes a sentence better. So for the correction pass we looked at what actually changed.

  • Qwen3.8 first: it writes the German closest to ours and keeps all 11 files intact.
  • Gemma 4 second: it fixes small grammar slips like a wrong article.
  • Not the other way round: Qwen3.8 correcting Gemma 4 changed almost nothing.
  • A person last: the reviewer in the pull request has the final word.
  • Thinking off: Qwen3.8 and Gemma 4 think by default. With thinking on a run took five times as long and used five times the tokens. Gemma 4 ran into its output limit and 165 pieces were missing.
  • Short output per call: Gemma 4 and Llama 3.3 write at most 4,096 tokens per answer. Pieces instead of whole files keep answers far below that.
  • No deprecated models: Qwen3.6 and Gemma 3 still answer, but Qwen3.6 is marked deprecated and Gemma 3 is no longer in the list. A pipeline should not start on them.
  • A tolerant parser: in the correction pass Gemma 4 sometimes returns both languages instead of only the text. The parser accepts both.

The lesson for your own pipeline: measure on your pages with your terms. A public ranking did not predict ours.

Cloud Framework Pieces and Guard Rails: Keeping a Model in Its Lane Open page Cloud Framework AI-Assisted Translation: Flow and Human Review Open page STACKIT documentation docs.stackit.cloud Available shared models on STACKIT AI Model Serving Open the documentation
Asset historyAdded Oct 30, 2026TMUpdatedNo updates · 1 bar = 1 week i
Maintainer
TMTobias M.Head of STACKIT Cloud Framework · STACKITOwnerActive 11 of the last 12 weeks · 196 updatesSTACKITwww.linkedin.com/in/tobias-müller-011304172Contributed in STACKIT