QA Automation Engineer

Ask Sage (BigBearAI company) is seeking a QA Automation Engineer that will translate nuanced product requirements and customer workflows into effective test coverage, build and maintain automation with Playwright, and partner with Product and Engineering to make informed release decisions as the platform evolves. The engineer will reason about complex, nondeterministic AI behavior, design risk-based coverage that reflects real customer impact, and produce automation that engineers trust. The engineer will own the automated coverage and the evidence behind release readiness across browser, application, and backend boundaries, where auditability, security, and government compliance expectations are part of everyday delivery.

Responsibilities

Translate requirements into testable outcomes, identifying ambiguities and defining acceptance criteria with Product and Engineering; Own automated test coverage by designing, implementing, and maintaining Playwright tests covering critical user journeys, feature functionality, and regressions, supported by API and integration testing; Develop a risk-based testing strategy prioritized by customer impact, feature dependencies, and the product roadmap; Test conversational AI workflows, including streaming responses, conversation history, file uploads, document retrieval and citations, model selection, and tool execution; Evaluate AI response quality by building evaluation datasets and scoring criteria for accuracy, grounding, instruction following, and handling of unsafe requests; Make automation reliable by integrating tests into CI/CD, investigating flaky tests, maintaining isolated test data, and providing actionable failure diagnostics; Communicate release readiness by reporting defects with reproducible evidence, customer impact, and severity, and explaining coverage gaps and residual risks; Preserve decision history by documenting expected behavior, approved changes, and testing rationale; Cover platform and AI infrastructure surfaces, including backend APIs, authentication flows, billing and token behavior, model routing, file parsing, MCP/tool execution, agentic harnesses, and passthrough APIs; Control test cost and execution footprint by minimizing unnecessary token usage, external provider calls, latency, and execution cost; Automate security-sensitive validation including user isolation, permission boundaries, input validation, sanitization, rate limits, replay prevention, and safe error handling; Apply AI-assisted testing tools responsibly while critically reviewing generated tests and proposed repairs; Scale the automation footprint by maintaining test infrastructure, fixtures, and test data management

Requirements

4+ years of QA automation engineering experience; Demonstrated experience building and maintaining automated tests with Playwright, including fixtures, resilient locators, assertions, network handling, and trace-based debugging; Strong coding ability in TypeScript or JavaScript, with experience writing maintainable test code and reviewing changes through Git; Experience with API testing, CI/CD integration, test isolation, and diagnosing failures across browser, application, and backend boundaries; Ability to reason about complex requirements, explore edge cases, and balance testing depth with delivery priorities; Clear written and verbal communication explaining defects, uncertainty, tradeoffs, and release risk to technical and nontechnical stakeholders; Experience testing authentication, authorization, permissions, and separation of customer data; Familiarity with validating Defense in Depth behavior without being a dedicated DevSec role; Strong troubleshooting and analytical skills, with the ability to work independently and as part of a team; Ability to obtain a Department of Defense Secret clearance; All applicants must currently reside in the United States

Nice to have

Experience testing LLM applications, retrieval-augmented generation, or AI agents; Experience with Python for API testing, test utilities, test data generation, or AI evaluation workflows; Experience with enterprise or government platforms and auditable test evidence; Experience testing model gateways, MCP tools, tool-using systems, or workflow automation and no-code/low-code platforms (e.g., Power Automate, Zapier, Make, n8n); Familiarity with flagship Generative AI provider APIs (Google Vertex AI, AWS Bedrock, Microsoft Azure OpenAI) and models (OpenAI GPT, Anthropic Claude, Google Gemini); Experience with CI/CD pipelines (e.g., GitHub Actions), Docker, Kubernetes, and observability/monitoring tooling; Experience with PostgreSQL and SQL for test data setup, teardown, and validation; Knowledge of government compliance frameworks (FedRAMP, NIST AI RMF, CMMC 2.0); Active DoD security clearance at the Secret level or above