Skip to content

Monitoring Azure Data Factory

This post is part 13 of 13 in the series Beginner's Guide to Azure Data Factory

In the previous post, we looked at the three different trigger types, as well as how to trigger pipelines on-demand. In this post, we will look at what happens after that. How does monitoring work in Azure Data Factory?

Now, if we want to look at monitoring, we probably need something to monitor first. I mean, I could show you a blank dashboard, but I kind of already did that, and that wasn’t really interesting at all 🤔 So! In the previous post, I created a schedule trigger that runs hourly, added it to my orchestration pipeline, and published it.

Let’s take a look at what has happened since then!

Continue reading →

Triggers in Azure Data Factory

This post is part 12 of 13 in the series Beginner's Guide to Azure Data Factory

In the previous post, we looked at testing and debugging pipelines. But how do you schedule your pipelines to run automatically? In this post, we will look at the different types of triggers in Azure Data Factory.

Let’s start by looking at the user interface, and dig into the details of the different trigger types.

Creating Triggers

First, click Triggers. Then, on the linked services tab, click New:

Screenshot of Azure Data Factory user interface with Triggers open, highlighting the button for creating a new trigger

The New Trigger pane will open. The default trigger type is Schedule, but you can also choose Tumbling Window and Event:

Screenshot of Azure Data Factory user interface with the New Trigger pane open and the different trigger types highlighted

Let’s look at each of these trigger types and their properties :)

Continue reading →

Debugging Pipelines in Azure Data Factory

This post is part 11 of 13 in the series Beginner's Guide to Azure Data Factory

In the previous post, we looked at orchestrating pipelines using branching, chaining, and the execute pipeline activity. In this post, we will look at debugging pipelines. How do we test our solutions?

You debug a pipeline by clicking the debug button:

Screenshot of the Azure Data Factory interface, with a pipeline open, and the debug button highlighted

Tadaaa! Blog post done? :D

I joke, I joke, I joke. Debugging pipelines is a one-click operation, but there are a few more things to be aware of. In the rest of this post, we will look at what happens when you debug a pipeline, how to see the debugging output, and how to set breakpoints.

Debugging Pipelines

Let’s start with the most important thing:

Continue reading →

Orchestrating Pipelines in Azure Data Factory

This post is part 10 of 13 in the series Beginner's Guide to Azure Data Factory

In the previous post, we peeked at the two different data flows in Azure Data Factory, then created a basic mapping data flow. In this post, we will look at orchestrating pipelines using branching, chaining, and the execute pipeline activity.

Let’s continue where we left off in the previous post. How do we wire up our solution and make it look something like this?

Diagram showing data being copied from an on-premises data center to Azure Data Lake Storage, and then transformed from Azure Data Lake Storage to Azure Synapse Analytics (previously Azure SQL Data Warehouse)

We need to make sure that we get the data before we can transform that data.

One way to build this solution is to create a single pipeline with a copy data activity followed by a data flow activity. But! Since we have already created two separate pipelines, and this post is about orchestrating pipelines, let’s go with the second option :D

Continue reading →

Data Flows in Azure Data Factory

This post is part 9 of 13 in the series Beginner's Guide to Azure Data Factory

So far in this Azure Data Factory series, we have looked at copying data. We have created pipelines, copy data activities, datasets, and linked services. In this post, we will peek at the second part of the data integration story: using data flows for transforming data.

But first, I need to make a confession. And it’s slightly embarrassing…

Continue reading →