Matei Zaharia

Digital Library

ACM Prize in Computing

Canada - 2025

citation

For visionary development of distributed data systems and computing infrastructure, which has enabled large-scale machine learning and analytics at global scale.

Matei Zaharia’s contributions are best summarized as the visionary creation of distributed data systems and computing infrastructure that have fundamentally reshaped modern data processing and analytics in both research and industry. Zaharia’s work has identified practical bottlenecks that previously constrained entire fields and resolved them with rigorous intellectual depth, producing systems that became default building blocks rather than isolated demonstrations.

His most recognized breakthrough, Apache Spark, addressed the severe limitations of earlier big-data frameworks by introducing Resilient Distributed Datasets (RDDs), a novel shared-memory abstraction that enables efficient, fault-tolerant, in-memory computation. Spark’s design provided a flexible, unified engine that supports batch processing, streaming, interactive queries, and machine learning (ML) workloads. Spark has become the de facto standard substrate for large-scale analytics, adopted by thousands of organizations worldwide. He translated these innovations into widely deployed infrastructure through the founding of Databricks and sustained leadership in open-source ecosystems.

Building on Spark, Zaharia continued to solve core infrastructure problems created by the shift to cloud-scale data and ML. He co-developed Delta Lake, a transactional storage layer that brings strong reliability guarantees and performance-oriented data management to cloud object stores, effectively bridging the gap between data lakes and data warehouses. He also created MLflow, an open-source platform that systematizes the ML lifecycle—experiment tracking, reproducibility, model management, and deployment—helping teams operationalize machine learning across diverse frameworks and environments.

Through both conceptual contributions and durable adoption his work has advanced computer science while democratizing large-scale data and ML capabilities for organizations and practitioners around the world.

Press Release

ACM Doctoral Dissertation Award

USA - 2014

citation

For his dissertation, "An Architecture for Fast and General Data Processing on Large Clusters," nominated by the University of California at Berkeley.

2025 ACM Prize in Computing

ACM named Matei Zaharia as the recipient of the ACM Prize in Computing for his visionary development of distributed data systems and computing infrastructure, which has enabled large-scale machine learning, analytics, and AI at global scale.

The ACM Prize in Computing recognizes early-to-mid-career computer scientists whose work has had broad and lasting impact. The award carries a $250,000 prize, with financial support provided by an endowment from Infosys Ltd, a global leader in next-generation digital services and consulting.

Zaharia’s work addressed a central challenge in computing: how to work with and analyze rapidly growing volumes of data efficiently, and at a scale previously accessible only to the largest technology companies. Early distributed data systems were limited in speed and poorly suited to emerging workloads such as machine learning and interactive analysis. Through a sequence of open-source systems, each targeting a distinct bottleneck, Zaharia changed what any organization could do with massive datasets.

As a doctoral student at UC Berkeley, Zaharia started Apache Spark, a new approach to distributed computing that reliably leverages memory to accelerate computations. This design made Spark dramatically faster than existing frameworks for the kinds of iterative computations essential to machine learning, while its unified architecture allowed batch processing, streaming, graph computation, and interactive queries to run within a single system. Spark quickly moved from research into widespread use and is now the de facto standard for large-scale data analytics, deployed across tens of thousands of organizations and integrated into major cloud platforms. Zaharia’s doctoral dissertation on Spark received the ACM Doctoral Dissertation Award in 2014.

With the shift to the cloud, Zaharia turned to a different problem: the lack of reliability and consistency in sprawling cloud data lakes – or the massive, centralized, and often unmanaged repositories storing vast amounts of raw data. He co-developed Delta Lake to bring transactional guarantees and principled data management to cloud object stores, making data pipelines more dependable and enabling a new class of architecture – the data lakehouse – that combines the flexibility of data lakes with the reliability of traditional data warehouses. Delta Lake is now widely adopted across industries, handling exabytes of data daily.

The growing use of machine learning introduced additional complexity. Zaharia started MLflow, another open-source platform to address fragmentation in machine learning and AI workflows, where teams struggled to track experiments, reproduce results, and deploy models consistently. MLflow provided a structured framework for managing the machine learning lifecycle – from experiment tracking and model versioning to deployment across diverse tools and environments – and has become a leading platform for operationalizing AI at scale. Together, these systems reshaped how data is leveraged in practice.

By building tools that any organization could freely use and extend, Zaharia ensured that the benefits of scalable computing became accessible to researchers, nonprofits, and enterprises across every industry. As investment in artificial intelligence accelerates, the infrastructure he built remains key to how data is processed, managed, and used to train and deploy AI applications and agents.

Today, Zaharia is focusing research on AI development, specifically how to build and scale reliable agents. He is a co-author on recent open source research, including DSPy and GEPA, which focus on auto-optimizing prompts and models to improve agent quality for specific tasks.