LATAM Software Engineers: Coding Tasks for AI Evaluation

We're running a paid study on the effectiveness of coding tasks used to evaluate AI agents. Our team is building a comprehensive suite of programming environments designed to test complex software capabilities. You will review a series of proposed coding tasks and assess their suitability for testing AI agents. The session involves a mix of code review, technical discussion, and direct feedback on the task structures.

Responsibilities

Review realistic programming tasks designed for AI evaluation; Assess the accuracy and structure of various evaluation harnesses; Walk us through your thought process while checking code validity; Provide actionable feedback on how to improve task complexity and realism

Requirements

Professional experience as a software engineer or developer; Located in Brazil or Argentina; Familiarity with building or verifying coding tasks and test environments; Comfortable discussing technical architectures and evaluation frameworks; Direct experience building, reviewing, or testing evaluation harnesses and programming tasks

Benefits

Paid study