We are embarking on a mission to redefine how data processing is done at Twitter. The team is building an integrated platform that unifies data processing with all other parts of the development ecosystem that is needed to develop products. Our work is a vital part of an overall Twitter goal to double development speed by the end of 2022.
We are making the Terabytes of data Twitter collects every day accessible to all applications. Platforms and tooling we’ve built handle millions of events per second and hundreds of thousands of queries per second today, with plans to grow both by an order of magnitude in the next couple of years. Engineering, Data Science, and Product Management teams need that data to be broken down by segments and user attributes that cover as many as 50 dimensions. Our next-generation analytics framework makes that data easier to generate, store and query. The team is pioneering new technologies and was the first to run production services in Google Cloud. Using public cloud technologies has allowed us to focus all of our time on core large-scale problems of Twitter while offloading pager and infrastructure maintenance to Google.
The team was assembled from leading engineers in streaming and cloud development across Twitter and you’ll have the opportunity to contribute to a key part of Twitter’s future and uplevel your own career. If you have a philosophy of building tools that deal with a broad set of problems, this is the team for you. If you have a steadfast customer focus that will help us to accelerate developer and analytic productivity, this is the team for you.
What You'll Do:
Rather than speak in generalities, here is a sampling of what we’ve been building:
Unified Data Platform (UDP) - Unified platform for building, configuring, and productionizing data processing across Twitter’s control plane.
Large Data Collider (LDC) - Integrated managed platform for generating and serving aggregated datasets.
DataAPI - Metrics-driven query service that enables the reuse of metrics across use cases and backends