Skip to content

Monitoring Azure Data Factory

Woman standing next to a projector showing the Azure Data Factory logo.

In the previous post, we looked at the three different trigger types, as well as how to trigger pipelines on-demand. In this post, we will look at what happens after that. How does monitoring work in Azure Data Factory?

Now, if we want to look at monitoring, we probably need something to monitor first. I mean, I could show you a blank dashboard, but I kind of already did that, and that wasn’t really interesting at all ðŸĪ” So! In the previous post, I created a schedule trigger that runs hourly, added it to my orchestration pipeline, and published it.

Let’s take a look at what has happened since then!

Monitoring Overview

First, let’s navigate to the Monitor page, which is divided into five sub-pages:

MonitoringMenu

We’ll start at the top.

Dashboards

Under dashboards, you will see statistics of your pipeline, activity, and trigger runs. In my case…

…uh-oh 😅

MonitoringDashboards01

Turns out, if you drop a table that you’re trying to load data into, your pipelines start failing. Who could ever have predicted that? 😂 Now, your goal is obviously no failures, but I wanted my pipelines to fail to show you a few things.

If you click on the pipeline chart, you will be taken to the pipeline runs page, with a pre-defined filter of either succeeded or failed.

Pipeline Runs

I clicked on failed, and everything is red and sad, aww 😔 But! You can click the filter to change the view:

MonitoringPipelineRunsFailed

If you switch from failed to succeeded

MonitoringPipelineRunsFailedToSucceeded

…everything looks green and happy, yay 😃

MonitoringPipelineRunsSucceeded

You probably want to see everything, though, to get a full view of your environment. You can change the filter again from succeeded to all statuses, then view the results as a list…

MonitoringPipelineRunsAllStatusList

…or as a Gantt chart:

MonitoringPipelineRunsAllStatusGantt

If you are working in multiple time zones, it’s easy to see the pipeline runs in your local time zone. Select a new time zone…

MonitoringPipelineRunsAllStatusGanttTimeZoneSelect

…and the dates and times will adjust automatically, while the view itself remains the same:

MonitoringPipelineRunsAllStatusGanttTimeZoneNew

Alright, cool! But…

What do I do about failed pipeline runs?

You can click on three buttons. One, you can view activity runs. Two, you can rerun the pipeline. Three, you can view the error message.

MonitoringPipelineRunsFailedActions

I usually check the error message, then see if I have to rerun the pipeline, otherwise look at the activity runs.

Activity Runs

The activity runs view should look familiar 😊 This is almost the same as the debugging output view. You can click the actions, view the details, and so on:

MonitoringPipelineRunsFailedActivityRun

But there’s one difference! Remember how we couldn’t debug from? At least we can rerun from! That certainly helps with production workloads!

MonitoringPipelineRunsFailedActivityRunRerun

In this example, we have a bunch of failed pipelines. (Remember not to drop tables in production! ðŸĪŠ) The trigger runs, however…

Trigger Runs

…are just fine! That’s because all the triggers started successfully. One thing to note, however, is the slight offset on some of the trigger runs:

MonitoringTriggerRuns

You can click on properties to see the scheduled trigger time and the actual trigger time:

MonitoringTriggerRunsProperties

Integration Runtimes

Under integration runtimes, you can monitor… well… your integration runtimes. We haven’t created any new integration runtimes yet, so we currently only see the default AutoResolveIntegrationRuntime:

MonitoringIntegrationRuntimes

This should always be running. If this is not running, Azure is experiencing some serious issues. In that case, check the Azure Status page for more information.

Alerts and Metrics

Finally, while the dashboards and overviews are nice and handy, you may not want to open them multiple times per day to check that everything is running smoothly.

(And if you do, you can just skip this part 😅)

You can set up alerts to get notifications if something fails (or succeeds!) in your Azure Data Factory.

How do I create a new alert?

From the alerts and metrics page, click new alert rule:

Alerts01NewAlertRule

Give the alert a name and description, and choose the severity. In the Azure Data Factory interface, you get no explanations of the severity levels, but these are listed in the official documentation:

  • Sev 0 = Critical
  • Sev 1 = Error
  • Sev 2 = Warning
  • Sev 3 = Informational
  • Sev 4 = Verbose

Then, click + add criteria:

Alerts02NewAlertRulePane

You can add alerts for a whole bunch of metrics. The most common ones are probably the failed metrics, but maybe you want to notify someone when a pipeline succeeds? You can do both:

Alerts03AddCriteria

Once you select a metric, you configure the alert rules for that metric:

Alerts04ConfigureAlertLogic

In dimension name, choose the pipelines you want to monitor:

Alerts04ConfigureAlertLogicDimensionName

In failure type, choose the failure types you want to monitor:

Alerts04ConfigureAlertLogicFailureTyoe

Then, set the threshold for how many pipelines need to fail before you get an alert:

Alerts04ConfigureAlertLogicThreshold

And then, the frequency for how often you want to check for failures:

Alerts04ConfigureAlertLogicFrequency

Finally, configure which notifications you want to get. Click + configure notification:

Alerts05ConfigureNotification

Give the new action group a name and click + add notification:

Alerts05AddNotification

Choose the type of notification you want to receive, and fill in the required information. For example, to receive an email notification, fill in your email:

Alerts05AddNotificationPane

Now click save, save, save, save, save until you’re done saving 😄

If you get the MissingSubscriptionRegistration error:

Alerts06Error

…follow these steps 😊

Tadaaa! Your alerts and metrics page is now slightly less empty:

Alerts07Overview

Summary

In this post, we looked at how monitoring and alerting works. But what if we want to customize the monitoring views even further? Is there a way to do that in Azure Data Factory?

Well, I’m glad you asked! Because that just so happens to be the next topic I want to cover. Let’s look at adding annotations and user properties next!

Share or Comment?

About the Author

Professional headshot of Cathrine Wilhelmsen.Cathrine Wilhelmsen is a Microsoft Data Platform MVP, international speaker, author, blogger, organizer, and chronic volunteer. She loves data and coding, as well as teaching and sharing knowledge - oh, and sci-fi, gaming, coffee and chocolate ðŸĪ“