Skip to content

Templates in Azure Data Factory

This post is part 19 of 25 in the series Beginner's Guide to Azure Data Factory

In the previous post, we looked at setting up source control. Once we did that, a new menu popped up under factory resources: templates! In this post, we will take a closer look at this feature. What is the template gallery? How can you create pipelines from templates? And how can you create your own templates?

Let’s hop straight into Azure Data Factory!

Using Templates from the Template Gallery

From the Home page, you can create pipelines from templates:

Screenshot of the Azure Data Factory Home page, highlighting the create pipeline from template option

From the Author page, you can click on the pipeline actions menu, then click pipeline from template:

Screenshot of the Azure Data Factory Author page, highlighting the pipeline from template option

Both of these open up the template gallery, with a whole bunch of pre-defined templates and patterns:

Screenshot of the template gallery

You can filter on categories

Screenshot of the template gallery, highlighting how to filter templates by categories

…or tags

Screenshot of the template gallery, highlighting how to filter templates by tags

…or services

Screenshot of the template gallery, highlighting how to filter templates by services

When you click on a template, you will see a preview of the pipeline, the description, tags, services, and user inputs:

Screenshot of the use this template pane

In the user inputs, select the datasets you want to use, or create new ones. Then, click use this template:

Screenshot of the use this template pane, highlighting the selected user inputs and use this template button

The template will create the necessary resources

Screenshot of the notification saying that resources have been created from template

…and call out the things you need to configure:

Screenshot of the validation output highlighting what needs to be configured

If you’re not sure how to configure the pipeline, you can click the documentation link:

Screenshot of the template documentation link in the pipeline properties pane

This will take you to the template documentation:

Screenshot of the official documentation for Azure Data Factory templates

Easy, huh? :D

Creating Custom Templates

You can also create custom templates and share them with your team – or share them externally with others. Custom templates are saved in your code repository and will show up in the template gallery for you and your team. If you want to share them externally, you can easily export them, so others can import them in their Azure Data Factory.

Let’s take a look!

Say that you have a pipeline that looks up some metadata, copies some data based on that metadata, logs the status of the pipeline, and scales some resources up and down. Something liiike… this! You can save that as a pipeline by clicking save as template:

Screenshot of an open pipeline, highlighting the save as template button

This will open the save as template pane, where you see an illustration of your pipeline. Here, you specify the template name, description, tags, services, and documentation link:

Screenshot of the save as template pane

Notice that you can create your own tags, but that you have to select from a pre-defined list of services:

Screenshot of the save as template pane, highlighting the configurations

You may want to create a naming convention or a set of standard tags to stick to, so it doesn’t get out of control :)

From here, you can either save or export the template.

But before we do that… do you ever need screenshots of your pipelines? You don’t actually need to take a screenshot. Just save the preview image as an SVG:

Screenshot of saving the template preview image as an SVG file

SVG files, scalable vector graphics, can be scaled up and down without loss in quality. Tadaaa! Now you have an illustration of your pipeline, instead of a screenshot:

SVG version of a pipeline template

Cool cool cool. I love SVGs 🤓

Saving Custom Templates

When you save a template, it will show up under factory resources, as well as in the template gallery:

Screenshot of custom templates

If you click on it, you can choose to edit, use, export, or delete the template:

Screenshot of a custom template, highlighting the edit, use, export, and delete buttons

Notice that the user inputs are using the names of our original datasets and linked services. You have to decide whether this is informative and useful, or if you need to rename them to something more descriptive and less specific:

Screenshot of a custom template, highlighting the user inputs

When you use a template, notice how the documentation link is now pointing to a custom URL. This is a great way of linking your Azure Data Factory to your project documentation:

Screenshot of a pipeline created from a custom template, highlighting that the documentation button points to a custom URL

Custom templates are saved in your code repository:

Screenshot of the Azure DevOps code repository, showing custom templates

Exporting and Importing Custom Templates

When you export a template, you will download a ZIP file. The contents of the ZIP file is the same as you will find in your code repository:

Screenshot of the downloaded and unzipped custom templates folder

You can import and export templates from the templates menu:

Screenshot of the import template button
Screenshot of the export template button

Now you never have to start building a pipeline from scratch again :D

Summary

In this post, we looked at pipeline templates, the template gallery, and how you can create your own templates. They are an easy and convenient way to share design patterns and best practices between team members, and across Azure Data Factories.

But this pipeline… 👇🏻 This pipeline uses a couple of fun activities… Notice the Lookup and ForEach? Yeah. They’re awesome. They can be used to create dynamic pipelines, woohoo!

SVG version of a pipeline template

In the next few posts, we will go through how to build dynamic pipelines in Azure Data Factory. We will cover things like parameters, variables, loops, and lookups. Fun!

🤓

About the Author

Cathrine Wilhelmsen is a Microsoft Data Platform MVP, BimlHero Certified Expert, Microsoft Certified Solutions Expert, international speaker, author, blogger, and chronic volunteer who loves teaching and sharing knowledge. She works as a Senior Business Intelligence Consultant at Inmeta, focusing on Azure Data and the Microsoft Data Platform. She loves sci-fi, chocolate, coffee, craft beers, ciders, cat gifs and smilies :)