Skip to content

Annotations and User Properties in Azure Data Factory

This post is part 14 of 25 in the series Beginner's Guide to Azure Data Factory

In the previous post, we looked at how monitoring and alerting works. But what if we want to customize the monitoring views even further? There are a few ways to do that in Azure Data Factory. In this post, we will add both annotations and custom properties.

But before we do that, let’s look at a few more ways to customize the monitoring views.

Customizing Monitoring Views

In the previous post, we mainly looked at how to configure the monitoring and alerting features. We saw that we could change filters and switch between list and Gantt views, but it’s possible to tweak the interface even more to our liking.

However… it’s not very consistent at the moment. Under trigger runs, you can click the edit columns button:

Screenshot of the trigger runs monitoring view, highlighting the edit columns button

And then add, remove, and reorder columns in the edit columns pane:

Screenshot of the trigger runs monitoring view, after opening the edit columns pane

But under pipeline runs, you have to right-click on a column header to choose the columns you want to display:

Screenshot of the pipeline runs monitoring view, highlighting the column editor

I don’t know if this is because they are in the middle of revamping the user interface or if it’s a result of gradually adding new features that are more modern ¯\_(ツ)_/¯ What I do know is that a) I need to keep checking back to see if I have to change my post and screen shots, and b) you need to get comfortable adapting to change. Because things change often in Azure :)

Anyway! What I really want to show is that under pipeline runs, you can filter on annotations:

Screenshot of the pipeline runs monitoring view, highlighting the annotations filter menu

Right now, the list is blank, because we haven’t added any annotations. How do we do that?

Annotations

Annotations are additional, informative tags that you can add to specific factory resources: pipelines, datasets, linked services, and triggers. By adding annotations, you can easily filter and search for specific factory resources.

You need to figure out what kind of annotations make sense to you. Maybe you want to filter on the different logical steps of your solution, so you add the tags extract and transform? Perhaps ingest and prepare? Or maybe you want to tag resources with a business unit or a project name? It’s entirely up to you. All I recommend is that you’re consistent :)

How do I add annotations?

For pipelines and datasets: In the properties pane, under general settings, click + new annotation, then add the annotation value:

Screenshot of the pipeline interface, highlighting a new annotation in the properties pane

For linked services and triggers: In the configuration pane, click + add annotation, then add the annotation value:

Screenshot of the trigger interface, highlighting a new annotation

Remember to publish your changes :)

How do I use annotations?

Once you publish and rerun your pipeline, you will now see the annotation icons under pipeline runs:

Screenshot of the pipeline runs monitoring view, highlighting the latest pipeline runs with annotations

Click on one of the annotation icons to view the annotation value:

Screenshot of the pipeline runs monitoring view, highlighting how to click on the annotation icon to see the details

Click on the annotation filter button and select an annotation to filter on:

Screenshot of the pipeline runs monitoring view, highlighting how to click on the annotation filter button to select which annotations to filter on

You will now see the filtered list of pipeline runs with that annotation:

Screenshot of the pipeline runs monitoring view, with a filtered list of pipeline runs

On tha Author page, you can also use the search bar to search for factory resources with specific annotations:

Screenshot of the search results page, highlighting that the results column is showing the annotation searched for

User Properties

User properties are basically the same as annotations, except that you can only add them to pipeline activities. By adding user properties, you can view additional information about activities under activity runs.

For the copy data activity, Azure Data Factory can auto generate the user properties for us. Whaaat! :D

Open up a pipeline, click the copy data activity, and go to the user properties. Click auto generate:

Screenshot of a copy data activity, highlighting the auto generate button in the user properties pane

Azure Data Factory creates the source and destination user properties for you, based on the copy data activity settings:

Screenshot of a copy data activity, highlighting the auto generated user properties

Once you publish and rerun your pipeline and go to the activity runs, you will see the user properties button:

Screenshot of the activity runs monitoring view, highlighting the user properties button

When you click the button, you can choose to add each user property as a new column:

Screenshot of the activity runs monitoring view, highlighting the user properties column selection

The new user properties columns will show up in the monitoring view:

Screenshot of the activity runs monitoring view, highlighting the user properties columns

By adding user properties, you can easily see for example the source and destination files and tables directly in the monitoring view, without having to open the pipeline itself.

Summary

In this post, we looked at how to customize monitoring. We created annotations to easily filter and search for our factory resources. Then we created user properties so we could easily view our source and destination files and tables in monitoring without having to open the pipeline itself.

Now… At this point in the series, we have covered most of the basics when it comes to developing pipelines. But we have only worked with cloud data stores. What if we need to work with on-premises data? And what if we already have an on-premises SQL Server Integration Services (SSIS) solution and we don’t want to rebuild everything in Azure Data Factory?

In the next few posts, we will look at the different types of integration runtimes, how to work with on-premises data, and what SSIS Lift and Shift means.

🤓

About the Author

Cathrine Wilhelmsen is a Microsoft Data Platform MVP, BimlHero Certified Expert, Microsoft Certified Solutions Expert, international speaker, author, blogger, and chronic volunteer who loves teaching and sharing knowledge. She works as a Senior Business Intelligence Consultant at Inmeta, focusing on Azure Data and the Microsoft Data Platform. She loves sci-fi, chocolate, coffee, craft beers, ciders, cat gifs and smilies :)