اردو Education & Reasoning

A Gemma-3-4B model adapted to Urdu by translating English knowledge corpora into Urdu with Adaption AutoScientist, then benchmarked on UrduMMLU.

Gemma-3-4BAdaption AutoScientist UrduMMLU 46.2%+1.3 vs base

Try it — اردو میں سوال پوچھیں

64 512
0 1

What this validates

We tested whether adapting English knowledge corpora into Urdu with Adaption AutoScientist improves a 4B model on a native Urdu benchmark. It does, for knowledge that is language-independent: every such domain improved and the model exceeded its base overall (44.96% to 46.21%). The effect does not extend to Urdu literature, which is intrinsic to the language and requires native data rather than translation.

+5.9
STEM (pts)
+3.6
Profession
+2.7
Social Sci.
46.2%
UrduMMLU overall
+1.3
vs base overall

Results on UrduMMLU

Method

Most UrduMMLU subjects test knowledge that is largely language-independent: science, mathematics, reasoning, and social studies. We assembled about 40,000 examples from open English datasets covering these subjects, together with native-Urdu instruction and literature data, then used Adaption AutoScientist and the Adaptive Data pipeline to translate and localise each example into Pakistani Urdu, adding a reformulated prompt and an English reasoning trace. Gemma-3-4B was supervised-fine-tuned on the result and evaluated on UrduMMLU, zero-shot.

Boundary of the method. Cross-lingual adaptation improved the science, mathematics, reasoning, and social-knowledge domains, but Urdu literature declined by 2.5 points. That content cannot be produced by translating English sources; improving it requires native Urdu literary data. This is a limitation of available data, not of the adaptation method.