LLM Evaluator
Alignment & Evaluation
Rate, rank and red-team model outputs across reasoning, coding, safety and instruction-following rubrics. Ideal for engineers, researchers and strong generalists.
View Opportunities
XUmmia is hiring vetted experts for LLM evaluation, computer vision, multilingual review and red teaming. Remote-first, fair pay, and projects that match your expertise.

Open roles
All roles include paid onboarding, gold-set calibration and ongoing feedback from customer AI teams.
Alignment & Evaluation
Rate, rank and red-team model outputs across reasoning, coding, safety and instruction-following rubrics. Ideal for engineers, researchers and strong generalists.
Localization
Audit translations, generation quality and dialectal nuance in Swahili, Yoruba, Amharic, Hausa, Zulu, Wolof and 30+ languages.
Vision Datasets
Produce bounding boxes, segmentation masks and keypoints on street scenes, agriculture, healthcare and retail imagery with 99%+ IoU targets.
Specialist Verticals
Apply your professional expertise in medicine, law, finance or engineering to evaluate model outputs, write guidelines and catch safety-critical errors.
Safety
Probe models for harmful, biased or misleading outputs and document reproducible harm taxonomies. Strong writing and structured judgment required.
Operations
Own delivery for customer projects: roster management, quality telemetry, reviewer coaching and daily client communication.
Safety
Evaluate model responses against safety policies, flag edge-case harms and write concise justifications that improve guardrails. Similar to specialist evaluator roles on top AI labs' partner platforms.
Coding & Reasoning
Review, rank and explain code generations across Python, JavaScript, C++ and Go. Ideal for early-career SWEs, bootcamp grads and CS students who can spot subtle bugs.
Trust & Knowledge
Verify factual claims, source quality and tone in generated content. Flag hallucinations, outdated references and unsupported assertions for search, chat and research products.
Creative & Writing
Compare model-written drafts, rank clarity and style, and rewrite weak outputs to train more helpful, human-like writers. Strong C1+ English and editing experience preferred.
Advanced Reasoning
Evaluate ML, statistics and systems-design responses; write rubrics and gold explanations that teach models to reason like senior data scientists.
Alignment & Evaluation
A broad entry point for sharp generalists: compare answers, follow complex instructions, spot inconsistencies and provide readable explanations across many topics.
Why work with XUmmia
Rates benchmarked to global remote work, not local minimums.
No arbitrary thresholds. Work this week, get paid next week.
Remote-first with optional co-working in Lagos, Nairobi, Kigali and Cape Town.
Guideline writing, calibration and AI safety training included.
The application process
We review every application, run a paid calibration task, and only onboard experts who meet our customer quality bars. No unpaid trials, no opaque queues.
Every application is read by a real person on our talent team.
No unpaid trial work. Your first evaluation task is compensated.
Get paid in USD or local currency, every week, with no minimum threshold.
Submit a general application and our talent team will match you to the right project based on your background, language skills and availability.