Comparing Mapping and Wrangling Data Flows in Azure Data Factory

In 2019, the Azure Data Factory team announced two exciting features. The first was Mapping Data Flows (currently in Public Preview), and the second was Wrangling Data Flows (currently in Limited Private Preview). Since then, I have heard many questions. One of the more common questions is “which should I use?” In this blog post, we will be comparing Mapping and Wrangling Data Flows to hopefully make it a little easier for you to answer that question.

Illustration of person Comparing Mapping and Wrangling Data Flows in Azure Data Factory.

Should you use Mapping or Wrangling Data Flows?

Now, we all know that the consultant answer to “which should I use?” is It Depends ™ 😄 But what does it depend on?

To me, it boils down to a few key questions you need to ask:

  • What is the task or problem you are trying to solve?
  • Where and how will you use the output?
  • Which tool are you most comfortable using?

Before we dig further into these questions, let’s start with comparing Mapping and Wrangling Data Flows.

Generating SELECT statements in Biml using GetColumnList

Biml Wheel.

In a previous blog post, we looked at how to generate SQL using Biml. (If you haven’t read that post, you may want to start there and then come back here.) In this post, we will go through how to generate SELECT statements using the Biml column method GetColumnList.

Using Biml column methods

Biml column methods return code fragments. These code fragments can be used as building blocks to generate custom T-SQL statements. For example, the GetColumnList method returns a list of columns, separated by commas, that you can use in a SELECT statement. You can filter the columns and customize the output by passing parameters.

Examples of GetColumnList code fragments

If you have a table with three columns, the default output will look something like this:

[PersonID], [FirstName], [LastName]

But what if you don’t want to select all three columns? Or what if you want to use an alias for your table? No problem! The customized output can look something like this instead:

p.FirstName, p.LastName

We will go through the different ways of customizing the output a little later in this post.

Handling Schema Drift in Azure Data Factory

On April 4th, 2019, I presented my Pipelines and Packages: Introduction to Azure Data Factory session at 24 Hours of PASS. I was excited to show some cool features and use cases, including how to handle schema drift in the new Mapping Data Flows feature.

Aaaaand… I failed! 🤦🏼‍♀️

Illustration of a person with a bomb as a head saying “oh no!”.

Or, more specifically, my demo failed…

…when you test your demo three times and everything is fine, then it fails in your live session, but runs perfectly again once the session is over… 😂

Bullying, triggers, reactions, and reflections

After spending a week at MVP Summit with old and new friends, I feel energized. My brain has been refilled with lots of happy memories 😊

And one not-so-happy memory:

I just left karaoke crying after being booed off stage by a stranger. Years of bullying came flooding back in seconds, killing all confidence I’ve finally built up. I’m ashamed of not handling it better, but thankful I have friends who care. Don’t be that stranger. Be kind.

One. Just one negative experience during an otherwise fantastic week. One negative experience that objectively really wasn’t that big of a deal, followed by an absolutely overwhelming number of replies full of love and support. (Thank you 💙) And yet, this one negative experience has dominated my thoughts and feelings for days.

It’s easy to say that I refuse to let one stranger and one negative experience take over and push my happy memories away. Actually doing it is a whole different situation. No matter how determined I am or how much support I get, dealing with an experience like this is never easy for me.

I have trained my brain for many years to respond in a certain way, to gnaw at the negative even when it’s surrounded by positive. That habit does not change overnight just because I want it to. It’s something I need to actively work on to improve, and that requires determination and effort over time.

Writing is my way of dealing with things like this. Structuring my thoughts, describing my feelings, and reflecting on why I reacted the way I did is all part of my process.

But why are you sharing this publicly?

I’m very aware that my social media accounts are mostly highlight reels. I choose to focus on the positive things in my life, and so what most people see is the happy, bubbly, super excited version of me. I wish life was like that all the time 😊 But it’s not. Most days are not. Some days, you get flat out told that you suck, and instead of being strong and brushing it off, you end up crying and withdrawing from everything and everyone.

And that’s ok. That’s life. We’re flawed, and if we weren’t flawed, we wouldn’t be human. But being human also means we have choices. We can choose to take responsibility for our own feelings, get back up, try to learn from our experiences, and keep going.

Writing helps me. If I can also help someone else by sharing this publicly, whether it is helping one person understand why someone can react strongly to something seemingly small, or showing another person that they’re not alone in experiencing similar feelings, I will count that as a double win 😊

So what happened?

Technical Accomplishments and Happy Memories (T-SQL Tuesday #112)

T-SQL Tuesday logo.

The March 2019 edition of T-SQL Tuesday is hosted by Shane O’Neill (@SOZDBA). This month, Shane wants us to share our cookies. Wait… what? Yes! Cookies 🍪 In this analogy, cookies are accomplishments or memories you can look back on when things get tough. Something that will give you an energy kick to keep you going when you think you’re completely done.

I’ve spent a lot of time reflecting on this since the topic was announced. My cookie jar definitely consists of two types of cookies: technical and non-technical. There are more of the latter, which I believe is a good thing 😊

In this post, I’ll highlight some of the technical accomplishments I’m proud of, and share some of my happy memories that always make me smile. (I mean pictures. I will share lots and lots of pictures.)