This website is for sale

Member of Technical Staff (Index, Search Crawler)

Perplexity London

Posted Yesterday · via Ashby

Apply at Perplexity

About this role

AI answers are only as trustworthy as the information behind them. Perplexity grounds its answers in fresh, real-world sources, and the index is what makes that possible. We're looking for an engineer to work on the indexing and data processing layer of Perplexity's Search Crawler. You'll build systems that ingest, process, and organize large volumes of crawled content into efficient, queryable structures, working primarily on offline and nearline pipelines that keep data fresh, complete, and accessible to downstream retrieval. Throughput, scalability, and correctness are critical across datasets that change continuously.

Responsibilities Design, implement, and operate large-scale indexing systems and data pipelines at the core of Perplexity's search infrastructure.

Develop and optimize indexing strategies that balance performance, freshness, and resource efficiency.

Work on storage formats, compaction, and update mechanisms that keep data accessible and current.

Keep pipelines reliable and predictable under high-throughput conditions.

Build well-tested components with clear responsibilities and interface contracts that stay flexible as the system evolves.

Define and implement observability across offline and near line pipelines, including structured logs, metrics, and data quality signals.

Monitor throughput, resource usage, and cost, and drive optimisations as business needs change.

Partner with retrieval, ranking, and ML teams to make sure indexing outputs meet their requirements.

Enable safe experimentation on indexing strategies and processing logic through controlled rollouts and clearly defined quality signals.

Qualifications 5+ years of experience building production backend or data infrastructure systems.

Strong programming experience in Go, C++, or Rust.

Experience with high-throughput data processing systems and very large datasets.

Experience building or operating databases, storage systems, data planes, or indexing pipelines.

Strong understanding of distributed systems, fault tolerance, consistency, and scalability.

Experience running production systems and handling operational incidents.

A systems-thinking mindset and the ability to reason about end-to-end data flows.

This listing is collected from the employer's public hiring board via Ashby and may be abbreviated. See the full posting and apply on the employer's site.

Apply at Perplexity

Related jobs

New Head of EMEA Policy

Scale AI London, UK via Greenhouse

At Scale, we believe that AI will dramatically improve the world, and our mission is to accelerate the development of AI. As Scale continues to grow its Government Relations and Policy Insights function, we are looking for a Head of EMEA Policy to support our engagement with governments and third pa…

Posted Today

New Curriculum Manager - AI Engineering

DataCamp London, England, United Kingdom via Arbeitnow

About DataCamp Data and AI skills are critical for thriving today, and DataCamp is the platform that empowers everyone to learn them. We help individuals and Fortune 1000 companies close the data and AI skills gap through world-class learning, hands-on training, and a global community of expert ins…

Posted Today

New AI Engineering, Growth

Hrtechx London via Arbeitnow

Growth at HRtechX The world’s most demanding investors are busy managing trillions, and ignore anything average. Are you excited by the challenge of making them act? Are you looking to work in an exceptional entrepreneurial team with ambitious, motivated and extremely driven people? And do you wan…

Posted Today

New AI Engineer, Growth

Hrtechx London via Arbeitnow

Growth at HRtechX The world’s most demanding investors are busy managing trillions, and ignore anything average. Are you excited by the challenge of making them act? Are you looking to work in an exceptional entrepreneurial team with ambitious, motivated and extremely driven people? And do you wan…

Posted Today

New AI Engineer

Hrtechx London via Arbeitnow

Growth at HRtechX The world’s most demanding investors are busy managing trillions, and ignore anything average. Are you excited by the challenge of making them act? Are you looking to work in an exceptional entrepreneurial team with ambitious, motivated and extremely driven people? And do you wan…

Posted Today