Engineering Manager/Director - Spark Runtime Platform at Databricks

Databricks logo


(74 open jobs)

Databricks provides a Unified Analytics Platform that accelerates innovation by unifying data science, engineering and business.

Job title
Engineering Manager/Director - Spark Runtime Platform
Job listing last updated at
Jun 19, 2019
Job listing location
San Francisco
Job listing source
Job listing link
External link
Bay Area company location: 160 Spear St 13th floor, San Francisco CA

Job description


Databricks is a market leader in providing a big data and ML platform used by thousands of companies. We are seeking a strategic technical leader to manage the engineering teams responsible for building a reliable, cloud agnostic compute fabric powering Databricks Runtime (built on Apache Spark). This group currently consists of two fast growing teams:

  • Clusters - This team builds the compute fabric necessary for managing Spark workloads in a robust, flexible, secure and cloud agnostic way. Common use cases are stable and fast while advanced users have the primitives required to take advantage of cutting edge cloud features, services and data sources.
  • Serverless - This team builds the infrastructure and abstractions for resource management, data security, and application environment customization that enable customers to submit workloads, programmed against Spark APIs, without the need to specify or manage resources and execute them efficiently in terms of performance and cost.

In this role, you will work closely with leaders across the company, within engineering, as well as with product management, field engineering, recruiting and HR. You will be responsible for ensuring sustained hiring of the highest quality talent, organizational design and alignment with peer/partner teams within and outside engineering and robust processes for efficient execution against product strategy.


  • Deliver a high quality compute fabric and platform abstractions for resource, access control and environment management
  • Ensure high SLAs across owned services (at least 4 9’s, low/no regressions)
  • Grow a world class team of software engineers working on our compute fabric. Increase headcount by 10+ engineers in next 18 months, with continued growth beyond that according to product objectives. Hire top-notch senior talent
  • Ensure consistent delivery against milestones and strong alignment with field and customer
  • Manage technical debt, including long term technical architecture decisions and balance product roadmap


  • Technology - Can ensure the team builds high quality and reliable infrastructure services. Experience being responsible for testing, quality, and SLAs of a product. Previous experience building and leading teams in a complex technical domain, such as on distributed data systems or database internals
  • Team - Ability to attract, hire, and coach engineers who meet the Databricks hiring standards. Can uplevel existing team via hiring top-notch senior talent, growing leaders and helping struggling members. Can gain trust of team and guide their careers. Experience managing distributed teams preferred.
  • Domain - Existing experience building and running cloud platforms -or- demonstrated ability to quickly learn new concepts in the SaaS space (e.g technical background and fast learner).
  • Product - Comfort working cross functionality with product management and directly with customers; ability to deeply understand product and customer personas.

Let Bay Area companies find you

Join candidate network