However, the biggest innovation in Delta Engine to tackle the challenges facing data teams today is the native execution engine, which we call Photon. This delivers up to 5x faster scan performance for virtually all workloads. The improved query optimizer extends the functionality already in Spark 3.0 (cost-based optimizer, adaptive query execution, and dynamic runtime filters) with more advanced statistics to deliver up to 18x increased performance in star schema workloads.ĭelta Engine’s caching layer automatically chooses which input data to cache for the user, transcoding it along the way in a more CPU-efficient format to better leverage the increased storage speeds of NVMe SSDs. We discuss several alternatives and future directions for research.ĭelta Engine由3部分组成,优化器是Spark3.0的扩展,caching层,原生执行器Photon Other designs may also be viable, however, as are other concrete technical choices in our high-level design (e.g., our stack at Databricks currently builds on the Parquet storage format,īut it is possible to design a better format). We have been building towards a Lakehouse platform based on this design at Databricks through the Delta Lake, Delta Engine and Databricks ML Runtime projects. In this section, we sketch one possible design for Lakehouse systems, based on three recent technical ideas that have appeared in various forms throughout the industry. 针对这3个方面, Spark对应的有3个project,Delta Lake, Delta Engine and Databricks ML Runtime projects We present results from a SQL engine over Parquet (the Databricks Delta Engine ) that outperforms leading cloud data warehouses on TPC-DS. Nonetheless, we show that a variety of techniques can be used to maintain auxiliary data about Parquet/ORC datasetsand to optimize data layoutwithin these existing formats to achieve competitive performance. In contrast, classic data warehouses accept SQL and are free to optimize everything under the hood, including proprietary storage formats. That have been amassed over the last decade (or in the long term, some other standard format that is exposed for direct access to applications). Lakehouses will need to provide state-of-the-art SQL performance on top of the massive Parquet/ORC datasets These APIs enable ML workloads to directly benefit from many optimizations in Lakehouses. In addition, many ML systems have adopted DataFrames as the abstraction for manipulating data,Īnd recent systems have designed declarative DataFrame APIs that enable performing query optimizations for data accesses in ML workloads. ML systems’ support for direct reads from data lake formats already places them in a good position to efficiently access a Lakehouse. Support for machine learning and data science: Second, not only were datasets growing rapidly, but more and more datasets were completely unstructured, e.g., video, audio, and text documents, which data warehouses could not store and query at all. This forced enterprises to provision and pay for the peak of user load and data under management, which became very costly as datasets grew. We refer to this as the first generation data analytics platforms.Ī decade ago, the first-generation systems started to face several challenges.įirst, they typically coupled compute and storage into an on-premises appliance. Which then could be used for decision support and business intelligence (BI).ĭata in these warehouses would be written with schema-on-write, which ensured that the data model was optimized for downstream BI consumption. The history of data warehousing started with helping business leaders get analytical insights by collecting data from operational databases into centralized warehouses,
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |