Databricks Moves Delta Lake Project to Linux Foundation
Databricks transfers its Delta Lake open-source project to the Linux Foundation to support the emerging Lake House data management pattern.
Databricks officially transfers its Delta Lake open-source project to the Linux Foundation under an open governance model. The project, which launched earlier this year, already sees significant adoption and earns backing from major tech players like Intel and Alibaba. This move mirrors the company's previous decision to donate its Apache Spark SQL project to a major open-source foundation.
This transition reflects a major shift in how organizations handle massive amounts of information through a concept Databricks calls the "Lake House" pattern. Instead of simply using data lakes for machine learning and data science, companies now perform traditional data warehouse tasks like business intelligence and reporting on these same systems. Delta Lake serves as the essential data layer for this pattern by bringing ACID transactions, scalable metadata handling, and data versioning to standard data lakes.
The release of Spark 3.0 complements this strategy by significantly speeding up these Lake House use cases and introducing a pluggable data catalog feature. All information within the Delta Lake system stores efficiently in the Apache Parquet format while allowing users to easily enforce and modify schemas. Databricks selects the Linux Foundation to host the project due to its proven success in managing the world's largest cloud and open-source initiatives.