Staff Engineer, Data Lake Platform
- Salary
- Not disclosed
- Experience
- 10+ years
- Job type
- Full Time
- Work mode
- On-site
More details
- Function
- Data Platform Engineer
Job description
Stripe is a financial infrastructure platform empowering businesses worldwide, from global enterprises to ambitious startups, to accept payments, grow revenue, and accelerate opportunities. Our mission is to increase the GDP of the internet, offering an unprecedented opportunity to shape the global economy and advance your career.
The Data Lake team is responsible for building and maintaining Stripe's core data access and governance infrastructure, providing a secure, fast, and compliant pathway to critical big data assets. We serve a diverse range of users, including developers, data engineers, analysts, ML/AI teams, security professionals, and business users. Currently, the team is undergoing a significant architectural transformation to align with modern data ecosystem standards, ensuring scalability and support for diverse workloads.
This role offers a unique opportunity to work on foundational infrastructure with broad impact, influencing petabytes of data and hundreds of production pipelines. You will be instrumental in a multi-year migration to modern, open-source solutions like Apache Iceberg, making critical architectural decisions that shape the platform's engagement with the broader data infrastructure ecosystem. You will also own the object storage abstraction layer, defining how engineering teams interact with vast amounts of data securely and efficiently.
Responsibilities
- Architect the unified Iceberg platform, leading the technical design of a metastore service as the single source of truth for Iceberg table management across all compute engines (Spark, Trino, Flink, PyIceberg). Define API contracts, authorization models, per-table credential vending, and integration patterns.
- Own the metastore migration strategy, driving sequencing, backward compatibility, rollback approaches, and cross-team coordination for migrating Hive Metastore-backed workloads to the new platform, ensuring continuous operation of production data infrastructure.
- Shape the object storage abstraction layer, defining bucket provisioning, access control policy design, and developer-facing client libraries to make object storage ergonomic and secure by default.
- Lead compliance architecture in partnership with security and compliance teams, translating regulatory requirements into durable preventative technical controls such as audit logging, access review infrastructure, data segregation, and lifecycle enforcement.
- Drive cost and efficiency at petabyte scale by identifying systemic inefficiencies in storage layout, snapshot retention, and data lifecycle, and designing automated, self-service tooling.
- Set the technical bar by owning critical design reviews, establishing standards for reliability, security, and developer experience, and mentoring senior engineers through high-stakes architectural decisions.
Requirements
13 skillsWhat you’ll need
- 10+ years of professional software engineering experience.
- Demonstrated track record of designing, building, and operating large-scale distributed storage or data infrastructure systems.
- Deep experience with object storage (S3, Azure Blob, or equivalent), including IAM, access control policy design, lifecycle management, and operational practices at petabyte scale.
- Proven ability to lead complex, multi-quarter infrastructure projects end-to-end, including cross-team dependency management and coordinating migrations across many consuming teams.
- Strong background in authorization and access control design for distributed data systems.
- Deep expertise in Apache Iceberg (table format internals, REST Catalog specification, snapshot lifecycle management, compaction, and compute engine integration).
- Background in compliance-sensitive infrastructure (SOX, ICFR, or equivalent regulatory frameworks) with an understanding of how audit and access review requirements translate into preventative technical controls.
- Experience safely executing large-scale data migrations with a strong instinct for sequencing, blast radius reduction, rollback, and data integrity validation.
- A strong developer experience sensibility: the ability to build abstractions that are ergonomic, well-documented, and actively reduce toil for the engineering teams that depend on your platform.
Stripe
This job was sourced from an external listing. Applying takes you to the original posting.