Software Engineer - Hadoop

San Francisco, CA

Who we are:

Twitter is seeking engineers to help build and grow our data storage and processing systems. Our Hadoop infrastructure is one of the industry’s largest-scale deployments with over 50K machines that store and process hundreds of petabytes of data. Our team has a track record of active contributions to the Apache Hadoop code base. All Twitter features and products use our Hadoop infrastructure one way or the other.

What you’ll do:

 As a Hadoop Engineer, you will build systems that serve tens of thousands of jobs and queries per day. The services you build will integrate directly with Twitter’s products, opening the door to new and cutting-edge features. You will empower dozens of engineering teams, hundreds of co-workers, and millions of users to gain new insights and dream of new possibilities.
 

The Hadoop team is hiring in the following areas:

  • Distributed storage and compute infrastructure

  • Event log ingestion, consolidation, and replication

  • Automated self-service cross cluster & cloud data replication

  • Hybrid on-premise and in-cloud Hadoop infrastructure

  • Near real-time Hadoop usage analysis and chargeback tooling

 

Who You Are:

You are passionate about cutting-edge open-source technologies. You enjoy working with top notch engineers to build new features and take on complex problems. You have experience in distributed systems, database internals, networking fundamentals, or performance analysis. You like to present our technical challenges and your accomplishments at conferences. You want to work in a team with an enormous impact on Twitter’s business and the world at large.

Responsibilities:

  • Work with and contribute to the Apache Hadoop and related open source communities to build new features and fix issues in support of Twitter’s usage and growth.

  • Design and build a hybrid cloud solution combining on-premise and in-cloud clusters.

  • Diagnose and troubleshoot complex distributed systems problems and develop solutions with a significant impact at our massive scale.

  • Build tools to ingest and process more than 1 trillion messages per day.

  • Design and develop next-gen storage and compute platforms used by dozens of engineering teams in our product, revenue and data science organizations.

  • Communicate with a wide set of teams, including Hardware, Network, Linux kernel, JVM, SiteOps teams, and cloud vendors.

  • Build advanced tooling for testing, monitoring, administration, and operations of multiple clusters across data centers, on-premise and in-cloud clusters.

 

 

Requirements:

  • Hands-on experience in distributed systems.

  • Strong software development skills in at least one of: Java, C/C++, or Scala.

  • BS, MS or PhD degree in Computer Science or Engineering, or equivalent experience.

  • Experience building and supporting large-scale systems in a production environment.

 

A plus:

  • Running large infrastructure services on cloud.

  • Experience with OSS technologies such as HDFS, YARN, Tez, Flume, HBase, Spark, Zookeeper.

  • Solid knowledge of networking and Linux systems management.

  • Experience with operating system internals, file systems, disk/storage technologies and storage protocols.

We are committed to an inclusive and diverse Twitter. Twitter is an equal opportunity employer. We do not discriminate based on race, ethnicity, color, ancestry, national origin, religion, sex, sexual orientation, gender identity, age, disability, veteran status, genetic information, marital status or any other legally protected status.

San Francisco applicants: Pursuant to the San Francisco Fair Chance Ordinance, we will consider for employment qualified applicants with arrest and conviction records.

Engineering Hiring Process

Step 1

Once your application is received, a recruiter will reach out pending your qualifications are a match for the role.

Step 2

If your background is a match, you may have 1-2 technical phone interviews or be given the chance to provide a work sample depending on the role.

Step 3

If the phone interviews go well or your work sample is strong, the final step includes interviews with 5-6 people held onsite in our office.

Application

Personal Information

This field is required.
This field is required.
This field is required.
This field is required.
Required field. PDFs only; max file size is 1MB.
Required field. PDFs only; max file size is 1MB.

U.S. Equal Opportunity Employment Information  (Completion is Voluntary)

At Twitter, we have a bold aspiration to reach every person on the planet. We believe that goal is more attainable with a team that understands and represents different cultures and backgrounds and we are committed to an inclusive and diverse Twitter.

This is where you come in! Please take a few minutes to provide us with your information. You are not required to provide this information and you may select “Decline to Disclose”. Your decision to provide information (or not) will not affect your employment or opportunities at Twitter.

Twitter is an equal opportunity employer. We do not discriminate based on race, color, ethnicity, ancestry, national origin, religion, sex, gender, gender identity, gender expression, sexual orientation, age, disability, veteran status, genetic information, marital status or any legally protected status.

You can view the ‘EEO is the Law’ poster here.

Twitter does not accept any unsolicited resumes from recruiting agencies and will not pay fees associated with any such resumes. Agencies, please do not send resumes to any Twitter location, employee, or email address.

Twitter, Inc. is committed to working with and providing access and reasonable accommodations to applicants with physical or mental disabilities. If you need an accommodation in order to apply for open job opportunities, please submit a description of your accommodation request to RARequest-Recruiting@twitter.com. This email is only for accommodation requests related to the application process.

Twitter cares about your privacy and protecting your data.  Please click the privacy policy link and acknowledge you have read and understood how Twitter treats your privacy and your data.  

Would you like to receive email communication from Twitter about career opportunities? You may unsubscribe at any time.
Applicant Data - You have a choice. Can we keep your personal data for both the job you are applying for and any other Twitter jobs that we feel you may be a match for? If you choose yes we will retain your personal data for a period of twelve months to consider you for other job opportunities at Twitter.
Analytics - May we use personal data from your resume and application to analyze and improve the Twitter hiring experience.
Success
Thanks for applying!
Error
Submission failed. Please make sure all fields are correctly formatted.

Don't see the right fit?

Check out other opportunities at Twitter.