Informing you of the latest news in the lakeFS community

The State of Data Engineering 2023

The annual lakeFS State of Data Engineering is now available, and this year, as we ponder where data engineering is headed, we notice two major shifts. The first is the economic slowdown, and the second is the emergence of Generative AI. 🤖

In this year’s report, we note how established domains face greater competition over pricing, more acquisitions, more consolidation, and fewer new players. Ultimately, the current crisis will bring us great products and clear the clutter around what is really needed in the data domain versus fad trends. 

Even with the economic slowdown, Generative AI has emerged unscathed, growing its market share. 

This focus on GenAI is bound to influence the data engineering space as well. At this point, we’re seeing some preliminary LLM-specific features in MLOps platforms, developer experience tools focused on GenAI-based applications, and LLM libraries.

Access the full report in this year’s edition of The State of Data Engineering.

Data Engineering Patterns: Write-Audit-Publish (WAP)

Robin Moffatt looks at what WAP is, shows how it can be implemented (comparing various technologies), and provides a deep-dive walkthrough of WAP in action. Check out the guide in this 3-part series:

Data Engineering Patterns: Write-Audit-Publish (WAP)

This part dives into the concept of Write-Audit-Publish, a pattern in data engineering to enforce data quality in data pipelines.

How to Implement Write-Audit-Publish (WAP)

A comparison of tools and their suitability for Write-Audit-Publish including Apache Iceberg, Apache Hudi, Delta Lake, Project Nessie, and lakeFS.

Putting the Write-Audit-Publish Pattern into Practice with lakeFS

A step-by-step tutorial on implementing Write-Audit-Publish with lakeFS, including support for multiple data formats.

Community events you shouldn't miss

Technical Deep Dive: lakeFS Delta Lake Diff Functionality | Watch now

Data & AI Summit | June 26–29, 2023 | San Francisco, Moscone Center | In Person | Virtual: June 28-29, 2023 | Register 

Fun moments from around the web