Microsoft Azure Data Factory as a must-consider alternative for on-premise ETL – tools

Cloud computing at your service

When using cloud-based technology your data is processed, stored and maintained in the cloud and not on a physical server at your organization.

This means that no infrastructure is necessary for the set-up and you don’t need to worry about system maintenance. This helps in saving resources (time and money) at the start of a project which can be used to understand requirements of the business. Your cloud solution will also be more adaptable to changing situations : newer features can be added easyly and up-scaling is only one click away. By the hand of the different data security protocols and features, you can sleep soundly that your data will be secure in the cloud.

Here comes Microsoft (again)

Microsoft Azure Data Factory (ADF) is a managed service in the cloud which provides the ability to extract data from different sources, transform it with data driven pipelines, and process the data. It is built for complex hybrid extract-transform-load (ETL), extract-load-transform (ELT), and data integration projects.

ADF is equipped with a functionality to make it easy to manage and monitor, including:

• Built-in monitoring: transformations and pipelines are monitored automatically and collected into a user-friendly dashboard.
• Alerting: Alerts can be set-up very easy for every measure that is recorded by Azure. For example when you run fails, or when the % CPU is above a certain value.
• Scheduling can be event-triggered. For example when a csv-file is uploaded into the blob storage.
• Source control integration

Microsoft Azure Data Factory is a mature ETL - tool

MS ADF is a service which runs in the cloud, meaning there’s no software to install and no operating system to configure on an external server. Scaling up or down is fast and easy in ADF, a feature which can be time consuming in a classic environment.

For the techies amongst us: MS ADF uses JSON script for its orchestration (coding in the background), while a classic ETL-tool uses drag-and-drop tasks (no coding). Because it is a service rather than software, its cost is based on usage. The flexible licensing model means that you can pay-as-you-go rather than buying a license up front as in a classic case.

Comparing ETL-tools amongst each other is quite complex since multiple parameters should be taken into account. We are writing a blog post on the comparison between MS ADF and classical ETL-tools. So more to come…

So what?

As an experienced Microsoft partner, Datashift helps companies developing new data platforms in the cloud. We often start by creating a business case to understand and grasp what the rationale should be of a data platform and its underlying architecture. We see data as a means to generate business results.

If you want to discover which outcomes we can generate by using MS Azure in general or MS ADF in particular, feel free to reach out!