Category: Biml

These posts are about Biml (Business Intelligence Markup Language), BimlScript, BimlExpress, BimlStudio, BimlOnline, the BimlHero Certified Expert Program and events by the Biml community or Varigence. Older posts may mention BI Developer Extensions, BIDS Helper and Mist.

Generating SELECT statements in Biml using GetColumnList

In a previous blog post, we looked at how to generate SQL using Biml. (If you haven’t read that post, you may want to start there and then come back here.) In this post, we will go through how to generate SELECT statements using the Biml column method GetColumnList.

Using Biml column methods

Biml column methods return code fragments. These code fragments can be used as building blocks to generate custom T-SQL statements. For example, the GetColumnList method returns a list of columns, separated by commas, that you can use in a SELECT statement. You can filter the columns and customize the output by passing parameters.

Examples of GetColumnList code fragments

If you have a table with three columns, the default output will look something like this:

[PersonID], [FirstName], [LastName]

But what if you don’t want to select all three columns? Or what if you want to use an alias for your table? No problem! The customized output can look something like this instead:

p.FirstName, p.LastName

We will go through the different ways of customizing the output a little later in this post.

Continue reading →

Generating SQL using Biml (T-SQL Tuesday #110)

T-SQL Tuesday #110: Generating SQL using Biml

The first T-SQL Tuesday of 2019 is hosted by Garry Bargsley (@gbargsley), and the topic is “Automate All the Things“. Garry wants to know what this phrase means to each of us. What do we want to automate? What is our go-to technology for automation? To me, this was super easy. Surprise, surprise! It’s Biml, of course :) Since this post is part of T-SQL Tuesday, I wanted to go back to the basics and write about how you can generate SQL using Biml. But first, a little bit of background for those who are not that familiar with Biml.

Continue reading →

Don’t Repeat Your Biml – C# Extension Methods

In a previous blog post, we looked at how to use C#/VB Code Files in Biml. There are several benefits to moving custom C# code into separate files. It allows you to reuse that code across multiple projects and solutions. You can maintain the code in your editor of choice, taking advantage of intellisense and syntax highlighting. And finally, my personal favorite: you can create custom extension methods.

In this post, we will look at how to simplify our Biml projects by creating and using C# extension methods. We will build on the examples from the previous C#/VB Code Files in Biml blog post.

Continue reading →

Biml for OData Source and Connection Manager in SSIS

As of July 2018, there is no built-in Biml support for OData. To work with OData in Biml, you have to create a custom source and connection manager. This requires more Biml code than built-in functions like OleDbSource and may look a little overwhelming at first. But don’t worry! You don’t have to start from scratch.

In this blog post, we will first look at the properties of the OData Connection Manager and how to script it in Biml. Then, we will do the same for the OData Source. Finally, we will tie it all together and create a complete SSIS package that you can use a starting point for your own projects.

The Quick and Easy Solution

But before we dig into any code, let’s skip to the quick, easy, and timesaving solution. That’s what we all really want, right? :)

Install or upgrade to BimlExpress 2018
Create an example SSIS package using an OData Source and Connection Manager
Convert the SSIS package to Biml
Done! :)

As promised: quick, easy, and timesaving! The new Convert SSIS Packages to Biml feature was released in BimlExpress 2018, and it really is a lifesaver. After converting to Biml, you can simply copy and paste the code into your projects.

However!

You may run into some bugs when you convert your SSIS packages to Biml. I ran into two issues while writing this blog post. The first was that I had to add UsesDispositions=”true” to the Source component. The second was that the data types in the Source component were prefixed with System. I have fixed both of these issues in my examples below. In addition to these issues, the converted Biml also contained some unnecessary code. Unnecessary code does not break anything, but it can make your code harder to read and maintain. Personally, I prefer my code to be as clean and simple as possible.

Ok, let’s dig into the actual Biml code!

Continue reading →

Biml Annotations and ObjectTags

Biml objects have many built-in attributes. For example, all Tables have SchemaName and all Packages have ProtectionLevel. When your Biml solution starts to grow, you will quickly see the need for adding additional metadata that can be used in other Biml files. A common use case in Data Warehouse Staging projects is to store the source schema and source table name on your staging table objects. This allows you to use the source metadata in a higher tier Biml file that generates the SSIS packages to load the tables. To store and use this additional metadata, you can use Biml Annotations or ObjectTags.

Biml Annotations and ObjectTags are both Key/Value pairs. Annotations are String/String pairs intended for storing simple text metadata, while ObjectTags are String/Object pairs that can also store more complex metadata in .NET objects.

Continue reading →

Don’t Repeat Your Biml – Tiered Biml Files

Biml (Business Intelligence Markup Language) - Tiered Biml Files

Many Biml solutions start off very simple, with just a single Biml file that generates a few SSIS packages. Most developers quickly see the need for a more complex solution for multiple sources. One way to reuse code and apply the Don’t Repeat Yourself software engineering principle in Biml is to use Tiered Biml Files.

In addition to using Tiered Biml Files, there are four other main ways you can avoid repeating your Biml code:

Include Files
CallBimlScript
C#/VB Code Files
Transformers (BimlStudio feature only)

In this post, we will look at how to use Tiered Biml Files.

Continue reading →

The Biml Book is here!

Woohoo! The Biml Book: Business Intelligence and Data Warehouse Automation is now available for pre-order from Amazon and Apress! :D

This is the first book I’ve co-authored, and I have to admit it’s a very strange feeling to see my name on the cover of a book. Am I allowed to say I’m quite proud? Oh, I’ll say it anyway. I’m proud and very honored to have written this book with such a talented group of people: Andy Leonard (@AndyLeonard), Scott Currie (@scottcurrie), Ben Weissman (@bweissman), Bill Fellows (@billinkc), Martin Andersson (@frysdisken), Peter Avenant (@PeterAvenant), Simon Peck (@biguynz), Reeves Smith (@SQLReeves), Raymond Sondak (@raymondsondak) and Jacob Alley.

What’s in The Biml Book?

The first part of the book starts with the basics: getting your development environment configured, Biml syntax, and scripting essentials.

The next part of the book guides you through the process of using Biml to build a framework that captures both your design patterns and execution management. In addition to leveraging design patterns in your framework, you will learn how to build a robust metadata store and how to package your framework into Biml bundles for deployment within your enterprise.

In the last part of the book, you will learn more advanced Biml features and capabilities, such as SSAS development, T-SQL recipes, automated documentation, and Biml troubleshooting.

When can I get The Biml Book?

Amazon says early December, but it might be available sooner. If you don’t want to wait, you can pre-order The Biml Book from Amazon or Apress right now.

Yay! This has been a long journey, and I’m so happy the other guys let me be a part of it :)

Create your own BimlExpress Keyboard Shortcuts

Are you tired of right-clicking on your Biml files to Check Biml for Errors or to Generate SSIS Packages? Did you know that you can create your own BimlExpress Keyboard Shortcuts? :)

Go to Tools → Options:

Select Environment → Keyboard, then type Biml in the Show commands containing box:

Select a Biml command, click in the Press shortcut keys box, click the keyboard shortcut combination of your choice, and click the Assign button. In this example, I have used Ctrl+Shift+C, Ctrl+Shift+B (I chose C then B for “Check Biml”):

Click OK, and that’s it! You can now use your keyboard shortcuts while having one or more Biml files selected. The shortcuts will appear in your BimlExpress menus in the toolbar and when you right-click on a file :)

Why can’t I create SSIS Project Parameters from Biml?

BIDS Helper and BimlExpress does not support creating SSIS project parameters from Biml out of the box. There are workarounds (and I have previously blogged about my solution for creating project parameters from Biml), but why is this not a standard feature in BIDS Helper or BimlExpress? Many people have asked about this, so I sat down with Biml creator Scott Currie (@ScottCurrie) to get the full story.

Why doesn’t BIDS Helper or BimlExpress emit SSIS project parameters from Biml?

Well, technically it could, but it shouldn’t. The user experience would have serious issues, leading to confusion, frequent errors, and the potential for data loss. How can that be?

Continue reading →

Get Started with BimlExpress

Say welcome to BimlExpress – the newest, shiniest and completely free Biml toy! :) I first mentioned this at SQLSaturday Vienna 2016 and have been waiting for the official release since then. I’m very happy that I can now send you all over to Varigence’s website to download the Visual Studio Extension!

What is BimlExpress?

BimlExpress is a free Visual Studio add-in for working with Biml in your SSIS projects. You can add and edit Biml files, and generate SSIS packages from Biml. The code editor has syntax coloring, error highlighting, Intellisense and a preview pane.

If you are already using BI Developer Extensions (previously known as BIDS Helper), you will see that BimlExpress is similar. You will find all the same Biml features as in BI Developer Extensions – just with a new and improved code editor. No more squiggly red lines, yay!

Which versions of Visual Studio does BimlExpress work with?

BimlExpress 2019 was released in May 2019. It works with Visual Studio 2010 – 2019.
BimlExpress 2018 was released in June 2018. It worked with Visual Studio 2010 – 2017.
BimlExpress 2017 was released in July 2017. It worked with Visual Studio 2010 – 2015.

Continue reading →