Assignment · Josh Talks · Oct 2026
Akshar Bench
Can AI write India's
shop posters?
A blind human evaluation of whether AI image models can put correct Hindi, Gujarati and Punjabi text on a small shop's festive poster. One missing vowel sign turns a sale into a typo, and existing text-to-image boards don't measure it.
Fixed prompts
Indian scripts plus English twin prompts
Image models compared
Blind rating rounds
- Problem
- Image models now make festive posters for small sellers, but nobody measures whether the Indian-script text on them is right.
- Bet
- Ask the shopkeeper's question first (which poster would you share today?), then audit the text letter by letter.
- Result
- A live rating app, results leaderboard and methodology, with quality control built in.
What it looks like
01
The question
Small sellers advertise on WhatsApp Status and Instagram in the language their customers read. A shopkeeper using an image model for a Diwali poster needs the text exactly right; a dropped vowel sign or broken conjunct makes the poster look careless or say something else.
02
Where it sits
It complements Voice of India's text-to-image cultural-truth board, which judges cultural faithfulness and explicitly does not measure text rendering. Akshar Bench measures only that gap.
03
Prompt design
20 fixed prompts sent verbatim to every model, in three difficulty tiers (T1 a single greeting line; T2 an offer with a number; T3 shop name, claim and a price with the rupee sign). Hindi (5), Gujarati (4), Punjabi (4), plus 5 English twin prompts describing the same scene as a Hindi prompt, so the English-to-Hindi drop measures the script, not the scene. Seller-voice variants (Hindi and Hinglish instructions) test what happens when a shopkeeper types the request themselves.
04
Generation rules
First output only, one image per prompt per model, no rewrites or cherry-picking; regeneration only after a logged technical failure. Every request logged (model ID, parameters, seed, timestamp, provider request ID). Models: GPT Image 1, Gemini 2.5 Flash Image, Gemini 3.1 Flash Image.
05
Two rounds, preference first
Round 1, blind pairs: “You own this shop. Which poster would you rather share with your customers today?” (A, B, Tie, Neither). Round 2, text audit: exact, small mistake, or wrong, tagged with eight failure types (missing vowel sign, broken conjunct, gibberish, wrong script, wrong number or price, missing words, extra text, text cut off). Preference comes first because once someone has hunted for broken conjuncts they can’t un-see them; the audit then shows how often the gut pick was “pretty but broken”.
06
Quality control
Script-matched gold pairs (correct text vs obvious gibberish) that exclude careless raters; a repeat pair with sides swapped to check consistency; a 3-second vote gate enforced on the server to stop reflex tapping; Indian-script text judged only by native readers; images served blind with random IDs and stripped metadata.
07
Metrics
Usable-as-is rate, text accuracy, preference win rate, Bradley–Terry strength, script gap (English minus Hindi accuracy on twin scenes), and the pretty-but-broken rate. 95% intervals bootstrap over prompts, not votes, because prompts are what generalize.
08
Automated check
A vision model transcribes each image and character error rate is computed against the target text, reported next to the human verdict and calibrated against it script by script. A first pass, never ground truth.
An early read, not a verdict.
Ratings are still coming in from a small volunteer panel, so the live leaderboard is directional. See the current numbers on the results page.
10
How it scales
At Josh scale the protocol becomes a pipeline: raters from Josh Jobs routed by the scripts they read, the OCR check as a first pass with humans only where it is unsure or disagrees, all 22 scheduled languages, and a refresh with every major model release.
11
What this shows about how I work
- Designing the eval around the user's real decision.
- Building rater quality control in rather than cleaning data afterwards.
- Reporting uncertainty honestly instead of a single headline number.