Data Platform Next Step: Insights from a Microsoft Conference in Denmark

Datashift's Data Engineering team was recently delegated at the Data Platform Next Step for Data Engineers. This Microsoft conference, held at the Legoland Hotel in Billund Denmark, provided a platform for international experts to share deep technical insights. In this blog post, our consultants explore three key takeaways from the conference that have the potential to revolutionize the way businesses leverage data.

Data Activator: Driving Actions and Alerts on Your Data

The Data Activator service is not yet in public preview but will have an added value to the complete Microsoft Fabric solution for sure. The Data Activator service is a no-code way to drive different actions on your data and will be used as a “system of detection”. The Data Activator service could be enabled on PowerBI visuals to drive alerts on the underlying data. The service could also connect to and be triggered by the other services that are available in Fabric and send output alerts or actions to services like teams or outlook. It seems that Microsoft thought about a way to implement data quality checks on the complete fabric landscape. Besides that, Data Activator will also have the capability to connect with services outside Fabric, like EventHub, to create alerts and actions based on your IOT (Internet of Things) events for example.

Delta Tables: Unlocking the Power of OneLake

Fabric makes a distinction between a lakehouse and a warehouse that both access the underlying lake to rule them all, OneLake. Both the warehouse and lakehouse store their files and tables in an underlying delta format, which is an open-source data framework that enables some advantages like ACID, schema enforcement, time travel, upserts, deletes… on files and tables that are stored in OneLake. That means that no matter who is accessing data in OneLake (data engineer via notebooks, data analyst via the warehouse or a business analyst via PowerBI), they will all access it as a delta format under the hood. This means that working efficiently with delta tables will become more important for data engineers! Optimizing your delta tables for parallelism and processing less data is therefore crucial. Microsoft Fabric gives the opportunity to enable V-Ordering, which is a write time optimization to parquet file format from which Power BI and SQL engines can make use of as well to achieve in memory like data access times!

Fabric + dbt: A Powerful Combination for Analytics Engineering

Our last key takeaway from this conference is the combination of Fabric with one of the most popular analytics engineering tools out there, dbt. The combination will make sure dbt can leverage all its advantages on the separate processing power of fabric while storing views and tables in OneLake. We think that this could be an immensely powerful combination with the other Fabric and Microsoft services. Especially the use of OneLake could have a major advantage for the whole data team!

Conclusion

The Data Platform Next Step conference in Denmark provided invaluable insights into the future of data analytics. From the upcoming Data Activator service to the importance of optimizing delta tables and the potential of combining Fabric with dbt, the conference showcased impactful advancements. With these key takeaways, we believe businesses can improve harnessing the true power of their data, driving informed decision-making, and achieving remarkable outcomes in the data-driven landscape of the future. Need any help or advice on implementing these tools? We’re here to help!