Research Engineer, Web Crawling

We're hiring a Software Engineer to build and own our web-crawling systems, from distributed collection at internet scale through filtering, deduplication, and deciding what data we keep.

Responsibilities

Design and scale the web crawler and ingestion infrastructure that sources Inkling's pretraining data; Build pipelines for large-scale extraction, deduplication, and data quality filtering; Build specialized crawlers for high-value or hard-to-reach data sources; Work with the pretraining team to understand how changes in crawled data affect model performance; Improve the reliability and efficiency of crawling and ingestion infrastructure at petabyte scale; Help set technical direction for this area as it grows, and bring other engineers up to speed on what you've learned

Requirements

8+ years designing, building, and scaling web crawlers, scrapers, or large-scale distributed data-acquisition systems; A track record of owning crawler or data-acquisition infrastructure at internet scale; Strong software engineering skills in a language such as Python, Go, or Rust, with real experience in distributed systems; Working knowledge of the practical and legal considerations of large-scale web data collection (robots.txt, rate limiting, licensing)

Nice to have

Experience applying machine learning to crawl selection, extraction, or data quality classification at internet scale; Experience setting technical direction for a crawling, data acquisition, or search infrastructure team, whether or not that was your formal title; Experience designing systems for petabyte-scale storage and processing; Track record of open-source contributions to crawling, scraping, or data infrastructure tools; Background at a search engine (crawling, indexing) or a frontier AI lab's data acquisition team

Benefits

Generous health, dental, and vision benefits; Unlimited PTO; Paid parental leave; Relocation support as needed; Visa sponsorship