AI Benchmark Engineer | Native Language Specialist - Chinese (Taiwan) - Remote

Building a rigorous, verifiable evaluation suite of Terminal-Bench tasks designed to test the limits of large language models on multilingual software challenges. The goal is to measure multilingual robustness across prompt language effects, non-English data processing, and complex locale/encoding edge cases in terminal workflows. Freelance, remote opportunity for native-speaking software engineers to design, build, and validate these benchmarks. Most tasks require a minimum of 2 hours per day or 15-20 hours per week. Work availability fluctuates with project demand, and as a 1099 contractor, no benefits such as health insurance, paid time off, or retirement contributions are provided, and hours are not guaranteed.

Responsibilities

Task Engineering: evaluating coding agents; Asset Creation: build realistic task environments using datasets and files in your native language, remaining in the target language; Prompting & Translation: finding failure points where AI does not work, in your native language; Implementation & Verification: support development of reference implementations and write reliable, deterministic verifier scripts (rubric-based judging only when strictly necessary); Calibration & Execution: analyze execution logs and calibrate task difficulty (Easy to Very Hard) using standard Terminal-Bench run configurations against various model tiers (Haiku, Sonnet, Opus); Quality Assurance: participate in a 4-layer human quality control process (creation, human review, calibration review, and audit) alongside automated LLM-based checks

Requirements

1+ years of industry experience in software engineering; Proven track record at leading technology companies and/or graduation from top-tier engineering universities; Native or near-native fluency in Chinese (Taiwan), with a deep understanding of its grammar, register, and phrasing rules; High English proficiency; Strong proficiency in Python, standard shell scripting, and data processing; Extensive experience with Terminal/CLI-based development workflows; Working familiarity with coding agents; Deep technical understanding of multilingual text processing pitfalls, including encoding/decoding robustness and Unicode normalization, locale-dependent conventions (collation, casing, non-Gregorian dates), text I/O, toolchain interoperability, and safe string operations; (For specific languages) Bidirectional/RTL handling, font fallbacks, and rendering/typography in UI or artifacts

Benefits

Flexible schedule with no fixed hours, check-ins, or micromanaging; Competitive rates and prompt payments; Access to diverse, innovative projects across industries and domains; Membership in a global community of linguists, subject matter experts, and language professionals