This website is for sale

Member of Technical Staff (Web Access Engineer, Search Crawler)

Perplexity London

Posted 6 days ago · via Ashby

Apply at Perplexity

About this role

Perplexity is hiring a Web Access Engineer for the Crawler team to build reliable fetching and rendering systems for AI-powered search. You'll work on freely available public web content, within site permissions and approved access policies, and develop the infrastructure that supplies content to downstream indexing and retrieval systems. Your focus will be diagnosing network errors, page-loading issues, and incomplete rendering. You'll trace problems from network protocols through client-side execution, distinguish technical failures from intentional access restrictions, and turn findings into production fixes that improve content quality and efficiency at scale.

Responsibilities Own fetching and rendering improvements end to end, from investigation through implementation and production rollout, partnering with crawler engineers and SREs.

Investigate fetch and rendering failures across DNS, TLS, HTTP, caching, redirects, and client-side rendering, and classify root causes.

Improve page loading, completion detection, and content capture to preserve relevant content and handle dynamic pages reliably.

Ensure the crawler respects site-imposed access restrictions and rate limits, identifies itself clearly, and retries only transient technical errors.

Build tools to reproduce failures, compare crawler output against expected page content, and turn findings into regression tests.

Define metrics for fetching reliability, content completeness, and error classification, and use them to prioritize improvements.

Qualifications 5+ years of software engineering experience in web infrastructure, browsers, CDNs, or crawling, with strong programming skills in at least one of Go, Python, Rust, or C++.

Deep understanding of HTTP/1.1, HTTP/2, and HTTP/3, including status semantics, caching, conditional requests, content encoding, and redirects; strong knowledge of TLS, DNS, cookies, and sessions.

Understanding of CDN caching and content delivery, HTTP error handling, retry backoff, and server-defined rate limits.

Knowledge of page lifecycle, network idle, service workers, client-side rendering, and shadow DOM; experience debugging JavaScript and fluency with Chrome DevTools, CDP, HAR files, curl, and packet captures.

Comfort working independently when the path forward is not fully defined.

Nice to have Experience developing CDN, caching, or web performance systems.

Experience operating crawlers, headless browsers, or distributed content-acquisition systems at scale.

Contributions to a browser engine or JavaScript runtime.

This listing is collected from the employer's public hiring board via Ashby and may be abbreviated. See the full posting and apply on the employer's site.

Apply at Perplexity

Related jobs

New Founding Account Executive, EMEA

Perplexity London via Ashby

Perplexity is hiring our founding Account Executive in EMEA. As our first seller in the region, you'll build the business from the ground up, landing our first EMEA enterprise customers, shaping how we go to market, and helping lay the foundation for the team that follows. This role is ideal for a t…

Posted Today

New Senior Data Scientist - Product Analytics

intercom London via Arbeitnow

Fin, now part of Salesforce, is on a mission to help businesses provide perfect customer experiences. Our AI Agent Fin is the highest-performing AI Customer Agent on the market today, enabling businesses to deliver impeccable, always-on customer support across the customer journey, from service, to…

Posted Today

New Applied AI Architect, Partnerships

anthropic London via Arbeitnow

About Anthropic Anthropic’s mission is to create reliable, interpretable, and steerable AI systems. We want AI to be safe and beneficial for our users and for society as a whole. Our team is a quickly growing group of committed researchers, engineers, policy experts, and business leaders working to…

Posted Today

New Manager, Forward Deployed Engineering - London

OpenAI London, UK via Ashby

About the team OpenAI’s Forward Deployed Engineering (FDE) team partners with our customers to turn research breakthroughs into production systems. We embed with users to solve high-leverage problems, move fast from prototype to deployment, and surface reusable patterns that shape our platform. We o…

Posted Today

New AWS Partnerships Lead

Anthropic London, UK via Greenhouse

About Anthropic Anthropic’s mission is to create reliable, interpretable, and steerable AI systems. We want AI to be safe and beneficial for our users and for society as a whole. Our team is a quickly growing group of committed researchers, engineers, policy experts, and business leaders working to…

Posted Today