Benchmark Methodology
How we evaluate LLM translation quality across Asian & ASEAN languages — a multi-judge, MQM-weighted panel over a fixed corpus.
Overview
The Dextalo Translation Benchmark systematically evaluates large language models on their ability to translate English text into Asian and ASEAN languages. Rather than relying on reference-based metrics (BLEU, COMET) that penalize valid alternative phrasings, we use a multi-judge LLM-as-Judge panel to score translations on four quality dimensions.
Test Corpus
The benchmark uses a curated corpus of 80+ English source sentences organized into eight domain categories:
Business
Corporate communications, financial reports, formal correspondence
Casual
Everyday conversation, informal tone, colloquialisms
Technical
Software documentation, IT infrastructure, APIs
Legal
Contracts, compliance, regulatory language
Marketing
Promotional copy, brand voice, persuasive content
Medical
Clinical documentation, pharmaceutical, patient care
Idiomatic
Figurative language, proverbs, cultural expressions
E-Commerce
Product descriptions, reviews, shipping policies
Each domain tests different translation skills: terminology precision, register matching, cultural adaptation, and technical accuracy.
Language Pairs
Models are tested on four English-to-target language pairs:
These languages were chosen to represent diverse writing systems, grammatical structures, and levels of LLM training-data availability.
Translation Process
Each model translates all sentences through the OpenRouter API using domain-specific system prompts. Sentences are batched by domain to provide context, and all models receive identical prompts to ensure a fair comparison.
Performance metrics — time-to-first-token, tokens per second, and cost — are measured via streaming responses and averaged across three runs per model.
Scoring Panel
Each translation is independently scored by three judge models:
Using multiple judges from different providers reduces single-model bias. Final scores use median aggregation across the three judges, which is more robust to outliers than averaging.
MQM Quality Dimensions
Judges score each translation on four dimensions from the Multidimensional Quality Metrics (MQM) framework, each on a 0–100 scale. The composite score is their weighted average, used for overall ranking:
| Dimension | Weight | What it measures |
|---|---|---|
| Accuracy | 0.40 | How faithfully the translation conveys the meaning of the source text. Measures semantic equivalence. |
| Fluency | 0.30 | How natural and grammatically correct the translation reads in the target language. |
| Terminology | 0.20 | How appropriate and consistent domain-specific terms are in the translation. |
| Style | 0.10 | How well the translation matches the expected register, tone, and style for the domain. |
Performance Metrics
Performance metrics are averaged across three runs to reduce variance from network conditions.
Limitations
- LLM judges are not perfect — they may have their own biases and are non-deterministic
- The corpus is curated and may not represent all real-world translation scenarios
- Performance metrics depend on the runner's network conditions and OpenRouter routing
- Only English-to-target translation is tested; the reverse direction may yield different rankings
- Cost calculations are estimates based on published token pricing