Technical Legal Engineer - Evals

We are seeking a tech-savvy lawyer with knowledge of, or a strong interest in tech products, AI and LLM evaluations. Apply your legal training and practice experience to evaluate, stress-test, and improve AI systems used in legal work. This role is ideal for a lawyer who enjoys issue-spotting, quality control, and analytical rigor, and is eager to shape how legal work is performed using modern technology.

Responsibilities

Review AI-generated legal outputs for accuracy, completeness, legal reasoning, and jurisdictional correctness across research, drafting, and analysis use cases; Assess outputs for real-world usability and identify failure modes, edge cases, and legal ambiguities that affect product reliability; Design and build high-quality evaluation datasets, including benchmark cases and gold-standard legal answers; Define clear scoring rubrics and evaluation guidelines for consistent, objective assessment; Continuously refine benchmarks as products, models, and legal requirements evolve; Identify recurring legal errors, edge cases, and failure patterns; Distinguish minor drafting issues from substantive legal risks; Flag AI behavior that could mislead users or create downstream legal risk; Collaborate with senior legal engineers, product managers, and engineers; Translate legal feedback into clear, actionable guidance for non-lawyers

Requirements

Lawyer with legal training and practice experience; tech-savvy; knowledge of or strong interest in tech products, AI and LLM evaluations; issue-spotting; quality control; analytical rigor; eager to shape how legal work is performed using modern technology