Skip to content

Variables in Azure Data Factory

This post is part 22 of 26 in the series Beginner's Guide to Azure Data Factory

In the previous post, we talked about why you would want to build a dynamic solution, then looked at how to use parameters. In this post, we will look at variables, how they are different from parameters, and how to use the set variable and append variable activities.

Variables

Parameters are external values passed into pipelines. They can’t be changed inside a pipeline. Variables, on the other hand, are internal values that live inside a pipeline. They can be changed inside that pipeline.

Parameters and variables can be completely separate, or they can work together. For example, you can pass a parameter into a pipeline, and then use that parameter value in a set variable or append variable activity.

System Variables

In the previous post, I called out the syntax for dataset and pipeline parameters. Did you notice the difference? :)

@dataset().ParameterName
@pipeline().parameters.ParameterName

In a dataset, you can reference the parameter name directly. But in a pipeline, you have to first reference “parameters”. That’s because inside a pipeline, you have both parameters and system variables:

Screenshot of the system variables inside a pipeline

To use system variables, you reference them in a similar way to parameters:

@pipeline().DataFactory
@pipeline().Pipeline
@pipeline().RunId
@pipeline().TriggerId
@pipeline().TriggerName
@pipeline().TriggerTime
@pipeline().TriggerType

User Variables

User variables are slightly different, though. You reference them by their name:

@variables('VariableName')

Let’s see how this works!

In this example, we will build a new pipeline to show how a variable can be used and updated. Our goal is to create a pipeline that can be used to load some or all the files from Rebrickable.

Create Variables

Create a new pipeline, go to the variables properties, and click + new:

Screenshot of an open pipeline, showing the variables properties, highlighting the new variable button

Give the variable a name and choose the type. You can specify a default value if you want:

Screenshot of an open pipeline, showing the variables properties, highlighting a new variable

Create two variables. One array variable named Files, and one string variable named ListOfFiles:

Screenshot of two variables

Then, create a bool parameter named LoadAllFiles:

Screenshot of one parameter

Next, we will create the pipeline activities.

Set Variable

Add a new set variable activity, go to variables, and choose the Files variable:

Screenshot of an open pipeline, showing a set variable activity, highlighting the variable name property

Add the array value colors,inventories,inventory_parts,part_categories,parts,sets,themes” (without quotes and spaces):

Screenshot of an open pipeline, showing a set variable activity, highlighting the variable value property

These are the seven Rebrickable files I consider the main files.

Then, add an if condition, and use LoadAllFiles parameter as the expression. This is a bool parameter that will evaluate to true or false:

Screenshot of an open pipeline, showing an if condition activity, highlighting the expression setting

(Hee hee, I’m sneaking in new activities in this series 🙃)

Add an if true activity:

Screenshot of an open pipeline, showing an if condition activity, highlighting the add if true activity button

Append Variable

Inside the true activities, add an append variable activity. Choose the Files variable, and use the valueinventory_sets“:

Screenshot of an open pipeline, showing an append variable activity, highlighting the name and value properties

Add a second append variable activity for “part_relationships“:

Screenshot of an open pipeline, showing an append variable activity, highlighting the name and value properties

Use Variables

Finally, add another set variable activity, and click to add dynamic content:

Screenshot of an open pipeline, showing a set variable activity, highlighting the add dynamic content link under the value setting

In the add dynamic content pane, you can click to add a variable:

Screenshot of the add dynamic content pane, highlighting the list of variables

You can use a combination of variables, functions, and string interpolation:

Screenshot of the add dynamic content pane, highlighting the list of functions

Let’s test our pipeline! Click debug:

Screenshot of an open pipeline, highlighting the debug button

And set LoadAllFiles to false:

Screenshot of the pipeline run parameters, highlighting the LoadAllFiles parameter set to false

Click on the set files output. We will see our array with seven files listed:

Screenshot of the pipeline output after a debug run, highlighting the output of the set files activity

Click on the set list of Files output. We will see the result of the expression using the variable, functions, and string interpolation. It says “Load 7 files: colors, inventories, inventory_parts, part_categories, parts, sets, themes“:

Screenshot of the pipeline output after a debug run, highlighting the output of the set list of files activity

Let’s try again! debug the pipeline again, but this time, set LoadAllFiles to true:

Screenshot of the pipeline run parameters, highlighting the LoadAllFiles parameter set to true

This, time, we see that the append variable activities also ran :) We also get a different output:

Screenshot of the pipeline output after a debug run, highlighting the output of the set list of files activity

Summary

In this post, we looked at variables and how they are different from parameters. Then we built a pipeline to show how to use the set variable and append variable activities.

We took a sneak peek at working with arrays in this post, but we didn’t actually do anything with it. We just showed the output.

In the next post, however, we will take a closer look at arrays. We will see how they can control a foreach loop!

🤓

About the Author

Cathrine Wilhelmsen is a Microsoft Data Platform MVP, BimlHero Certified Expert, international speaker, author, blogger, and chronic volunteer. She loves data and coding, as well as teaching and sharing knowledge - oh, and sci-fi, chocolate, coffee, and cats :)