←Alignment
    Dispatch · multilingual safety · 4 min

    Translated and localised prompts expose AI safety gaps that English tests miss

    A preprint turns English harmful prompts into fill-in templates for up to 60 languages, and finds some models that refuse in English comply once a prompt is translated.

    14 November 2026 · arXiv preprint, not yet peer reviewed
    inside a tape deck in low light: the head drum with a red glow, tape guides and circuit board
    Photo: Eytane Debruyne

    Most of the data used to teach large language models to refuse harmful requests, and to check whether they do, is written in English and reflects English-speaking concerns. A 2026 preprint by ETH Zurich researchers, supported by the Swiss AI Initiative and not yet peer reviewed, describes a way to produce such data in bulk for many languages. It releases the result as a corpus called MSafeTemp. The authors give four reasons this data is scarce. Native-speaker red-teamers are slow and costly. Human testing rarely reaches low-resource languages, those with little text online, which models therefore see little of in training. Plain machine translation adds no local context. And safety-trained models often refuse to translate or write harmful material at all.

    The pipeline avoids that last obstacle by never asking a model to handle the dangerous part directly. A generator model, Gemma-4-31B-IT, reads a harmful seed prompt, finds the words that make it harmful and replaces them with labelled blanks such as <illicit_substance>. What remains is a template mild enough to translate. A judge model, GPT-OSS-120B, checks that the template is valid. The generator then translates only the text around the blanks. A second check confirms the language, the translation quality and that the blanks survived. Each template has a main form and two paraphrases.

    The blanks are filled in three ways. Generic harmful fills form one branch. Benign fills keep the harmful-sounding wording and are used to measure over-refusal, a model wrongly turning down a harmless request. Culturally local fills come from a per-language "cultural card" of local groups, organisations, events and institutions, generated by Gemma and verified by Gemini Flash 3.6. Test and training sets are split by original seed, so no variant of a test prompt appears in training.

    The evaluation set starts from 200 HarmBench behaviours and spans 60 languages. The training set starts from 2,500 WildGuardMix prompts across 44 languages, giving about 1.8M items, with 500 seeds held out for testing. The authors count over 2.8M distinct harmful entities among the fills. In a human check of 600 judgments across 11 languages, raters scored naturalness at 4.43 out of 5 and cultural fit at 4.46. They endorsed 91% of the harmful labels but only 65% of the benign ones.

    The authors tested eight open models, with responses graded by GPT-OSS-Safeguard-120B. The attack success rate is the share of harmful prompts a model answered unsafely. On HarmBench, Olmo-3.1-32B's rate rose from 0.0 per cent in English to 13.7 after translation. Apertus-70B went from 8.5 to 13.0. The effect was not uniform: Llama-3.3-70B broke more often in English, falling from 13.0 to 5.3 when translated. Qwen3.5-27B and GPT-OSS-20B stayed at or below 1% throughout. But GPT-OSS-20B also refused 31.8 and 45.1 per cent of benign prompts on the two tracks, the highest in the panel. Localised prompts usually lowered success rates, yet they reached different failures from translation: about 45% of the failures specific to one prompt type appeared only with local entities. Scoring a single fill per template missed about half of the templates that broke a model.

    The authors then trained four models of 7 to 8 billion parameters, five checkpoints in all, using supervised fine-tuning followed by DPO, a method that teaches a model to prefer better answers over worse ones. Olmo-3-7B's HarmBench success rate fell from 21.5 to 2.9, and Llama-3.1-8B's from 15.9 to 2.0. Fine-tuning alone pushed Olmo's HarmBench over-refusal from 2.7 to 28.4, which DPO brought back to 19.8. Capability suffered: Llama's GSM8K maths score dropped from 84.0 to 68.5 and its MMLU score from 72.0 to 63.6. External benchmarks were mixed. PolyTox results worsened for the instruction-tuned Apertus-8B and for Llama. Llama's OR-Bench over-refusal rose from 4.6 to 24.8.

    The authors acknowledge that LLM judges run through every stage. A cross-check with judges from another model family covered only 5,000 responses. The human study had only one or two raters per language. Because raters felt many benign controls still read as harmful, measured over-refusal may overstate the real figure. Harm labels are inherited from the English seed, although a local entity can make a variant more severe. All prompts are single-turn. The in-house test tracks come from the same pipeline as the training data, and Gemma serves as both generator and test subject. The text also contains inconsistencies, describing seven models where its table lists eight.

    The dependence on inherited labels echoes a braintankAI report that guardrail benchmark scores can largely reflect agreement with one definition of "unsafe" (earlier story). The pattern of refusal training raising over-refusal and costing accuracy also appeared when refusal signals were carried from text to image and video models (earlier story).

    Paper
    How I Aligned Your Model: Scaling Up Multilingual Safety Training with Templates
    Authors
    Max Neuwinger, Hanna Yukhymenko, Imanol Schlag
    Venue
    arXiv preprint — not yet peer reviewed · 29 October 2026
    Link
    arxiv.org/abs/2610.40516
    Index
    stacks.braintank.ai
    Code
    github.com/swiss-ai/msafetemp
    Cite

    Keep reading

    a bearded doctor studies rows of skull X-rays on a lightbox, vintage colour
    a bearded doctor studies rows of skull X-rays on a lightbox, vintage colour

    A bearded doctor studies rows of skull X-rays on a lightbox, vintage colour · Photo: National Cancer Institute

    Alignment · AI-text detection

    Sparse autoencoders turn an opaque AI-text detector into one auditors can read

    A Google DeepMind and Penn preprint rebuilds a neural detector on features with plain-English descriptions, keeping most of its accuracy and, in some versions, generalising better.

    arXiv preprint · 21 October 2026
    small-plane cockpit: attitude indicator, compass and gauges against clouds
    small-plane cockpit: attitude indicator, compass and gauges against clouds

    Small-plane cockpit: attitude indicator, compass and gauges against clouds · Photo: Rinald Rolle

    Alignment · Agent prompt injection

    Office AI agents run planted scripts when users ask, a shared-workspace benchmark finds

    A 2026 test of 12 AI models doing finance, legal and consulting work found that swapped or attacker-supplied scripts derailed most of them.

    arXiv preprint · 9 October 2026
    close-up of a black control joystick on a teal console, blurred red and green indicator lamps behind
    close-up of a black control joystick on a teal console, blurred red and green indicator lamps behind

    Close-up of a black control joystick on a teal console, blurred red and green indicator lamps behind · Photo: Isawred

    Alignment · Jailbreak robustness

    Rerunning pretraining changes which jailbreaks break a language model

    In 150 small models, which jailbreak prompts succeeded depended far more on the particular pretraining run than on the training data, a preprint reports.

    arXiv preprint · 25 October 2026