South Asian Software Engineers: Coding Tasks for AI Evaluation

Paid study on the effectiveness of coding evaluation harnesses used to test AI agents. Participants walk through a series of proposed coding tasks and environments during an AI-moderated session and provide technical feedback on structure, difficulty, and realism of programming challenges and suggest improvements to test suites.

Responsibilities

Review proposed coding tasks and programming environments for realism; Provide technical feedback on the structure and difficulty of evaluation harnesses; Walk through approach to verifying complex code challenges; Suggest improvements to make the test suites more robust

Requirements

Experienced software engineer based in South Asia; Strong background in building and reviewing complex codebases; Experience building or reviewing realistic programming tasks; Familiarity with coding evaluation harnesses or test automation; Comfortable discussing technical workflows in an interview setting; Hands-on experience verifying realistic programming tasks in professional environments

Benefits

Paid study participation