New Release: BimlExpress 2017!

BimlExpress 2017What day is it today? It’s the BimlExpress 2017 release day! Yay! :) I’ve waited for this release for a long time, and I know I’m not the only one. People have asked me many times if Varigence is still around. The answer is most definitely yes! It has been rather quiet from Varigence this past year, and I know we’ve all been waiting for news and updates. But don’t forget that they’re a small company, and that they’re actually giving us BimlExpress for free. They’ve focused on development this past year, and have been working like crazy to rewrite the Biml engine, add new features to BimlExpress, rebrand Mist to BimlStudio and make it even more powerful, and finish their BimlFlex framework. And today? Today we all get to enjoy the new releases! More yay! :)

Download and install BimlExpress 2017

You can download BimlExpress 2017 from the Varigence BimlExpress page:
BimlExpress Download VSIX

The new installer is a lightweight VSIX (Visual Studio Extension) Installer:
Install BimlExpress 2017 - License Agreement Install BimlExpress 2017 - In Progress Install BimlExpress 2017 - Finished

After installing, you will find BimlExpress in the list of Visual Studio Tools and Extensions:
BimlExpress 2017 Visual Studio Extension

What’s New in BimlExpress 2017?

I will show you some of the exciting new features below. Spoiler alert: there are some gamechangers in this release! Let’s take a look at some of the exciting new features in BimlExpress 2017 :)

Continue reading →

Create your own BimlExpress Keyboard Shortcuts

Are you tired of right-clicking on your Biml files to Check Biml for Errors or to Generate SSIS Packages? Did you know that you can create your own BimlExpress Keyboard Shortcuts? :)

Go to ToolsOptions:
BimlExpress Keyboard ShortcutsToolsOptions - Tools -> Options

Select EnvironmentKeyboard, then type Biml in the Show commands containing box:
BimlExpress Keyboard Shortcuts

Select a Biml command, click in the Press shortcut keys box, click the keyboard shortcut combination of your choice, and click the Assign button. In this example, I have used Ctrl+Shift+C, Ctrl+Shift+B (I chose C then B for “Check Biml”):
BimlExpress Keyboard Shortcuts - Assign Shortcut

Click OK, and that’s it! You can now use your keyboard shortcuts while having one or more Biml files selected. The shortcuts will appear in your BimlExpress menus in the toolbar and when you right-click on a file :)
BimlExpress Keyboard Shortcuts Menu

Presenting remotely for Malta Microsoft Data Platform User Group

Cathrine Wilhelmsen Presenting for Malta Microsoft Data Platform User GroupWoohoo! On Tuesday, March 28th, I will be doing a remote session for the Malta Microsoft Data Platform User Group. I will be presenting my Biml for Beginners: Speed up Your SSIS Development session, one of my favorites :)

The meeting starts with my remote session, then there’s a short break before moving on to Gossip Geek – “… your one and only source into the fabulous lives of Geek’s elite…”. I have no clue what that is, but it sounds fun, so make sure to attend if you’re in the area :D And if Business Intelligence, SSIS and Biml is not your thing, maybe Columnstore Indexes are? The meeting finishes with Slawomir Drzymala (@sdryzmala) presenting Columnstore Index – do we need another one?!.

The Malta Microsoft Data Platform User Group is brand new. It was started in January 2017, and this will be their third meeting. I’m very happy I get to help support a new local group! If you’re a speaker and want to do a remote (or on-site!) session, you can find Malta Microsoft Data Platform User Group on Meetup, Facebook and Twitter (@MMDPUG).

And of course, if you’re from Malta, make sure you join and attend the meetings! If the sessions that day aren’t super interesting, attend anyway for the networking :)

Wow, I haven’t written one of these posts for a long time! It feels great to slowly move back into the world of speaking again. Thanks for inviting me to speak, MMDPUG! I wish I could be there in person, but I’m really looking forward to presenting for you :)

Speaking at Microsoft Ignite and SQLSaturday Minnesota

Hello, cold weather! I can’t believe it’s the middle of September already. Since I last wrote here I’ve been on vacation in Colorado, organized the third SQLSaturday in Oslo, road tripped around the fjords and mountains on Norway’s west coast, and worked three days a week for my client in Stockholm. But those are stories for later blog posts, Instagram, Facebook and Snapchat :) Now we’re back to speaking season!

Microsoft Ignite

Cathrine Wilhelmsen Speaking at Microsoft Ignite 2016I’m speaking at Microsoft Ignite! Whoa! I decided to give it a try and submit a session, but I never thought I had a chance since I’m speaking about Biml, a third-part product. But whaddayaknow, they selected my session! And my hubtalk! I was shocked, now I’m just nervous :) I’ve never been to a conference this large before, so I don’t really know what to expect.

On Tuesday, September 27th, I will present Avoid repeating yourself – an introduction to Agile SSIS development. It’s a 20-minute version of my Biml for Beginners session. Since I don’t have a lot of time I will focus more on the why than the how. Exciting!

On Thursday, September 29th, I will host an Automate Data Warehouse and Business Intelligence development hubtalk. A hubtalk is more like a discussion than a presentation, and I will be drawing on a Surface Hub. (So nervous!) I will be talking about Biml and metadata-driven development, and I hope to get a good discussion going with the attendees and learn from their experiences as well.

SQLSaturday Minnesota

Cathrine Wilhelmsen Speaking at SQLSaturday Minnesota 2016Since I was already going to the US, I wanted to try to squeeze in a SQLSaturday as well. I may be slightly addicted :D I really wanted to go to Kansas City since they have a Biml track, but I just couldn’t get it to work this time. However, I managed to get good tickets to Minnesota the weekend after! I originally registered as an attendee, but offered to step in as a speaker when someone had to cancel.

So – I’m speaking at SQLSaturday Minnesota as well! :) On Saturday, October 1st, I will present my Level Up Your Biml: Best Practices and Coding Techniques session. Dave DuVarney (@daveduvarney) will present Building Metadata Driven ETL Solutions with BIML before me, so I hope it will be a great day for those interested in learning Biml.

Are you going to Microsoft Ignite or SQLSaturday Minnesota? See you there! :)

Why can’t I create SSIS Project Parameters from Biml?

Why can't I create SSIS Project Parameters from Biml in BimlExpress or BIDS Helper?BIDS Helper and BimlExpress does not support creating SSIS project parameters from Biml out of the box. There are workarounds (and I have previously blogged about my solution for creating project parameters from Biml), but why is this not a standard feature in BIDS Helper or BimlExpress? Many people have asked about this, so I sat down with Biml creator Scott Currie (@ScottCurrie) to get the full story.

Why doesn’t BIDS Helper or BimlExpress emit SSIS project parameters from Biml?

Well, technically it could, but it shouldn’t. The user experience would have serious issues, leading to confusion, frequent errors, and the potential for data loss. How can that be?

First a comparison to SSIS packages
When the BimlEngine emits a package, it knows that it is overwriting the entire .dtsx file. In general, this is safe, because each .dtsx file contains exactly one package. There is no risk that by overwriting one .dtsx file, that it might overwrite both the desired package and some other unrelated package. One package per file – good.

Furthermore, since it is one package per file, BIDS Helper or BimlExpress can present you with a convenient checklist of all the packages that are about to be overwritten. Maybe you made a mistake in your BimlScript code and accidentally generated a package that has the same name as a painstakingly created manual package. Maybe you noticed it in the overwrite list. Maybe not. Either way, you had the opportunity to prevent that bad thing from happening. Also, if you are using source control and checked in recently, you can revert the changes and restore your manual package.

Finally, if you have a package open with unsaved edits, not much changes from the above scenarios. The very uncommon worst case is that you lose a small number of changes that you made to a manual package since you last saved – and only when you also made the mistake of generating a package with the same name as your manually created package. The common case is that you lose some manual changes (e.g. moving boxes around on the design surface) that you never intended to preserve in the first place.

How are SSIS project parameters different?
SSIS project parameters do not work the same way as SSIS packages. All project parameters are stored as XML elements in a single XML document for the entire project called Project.params. This is the core reason why packages have a good overwrite story while parameters have a poor overwrite story.

It should be obvious that BimlExpress can’t just overwrite your Project.params file. Of course, BimlExpress would be creating the parameters you specified in your BimlScripts, but it would also be overwriting any parameters you might have created manually. If you are a Biml purist, you might not care about this, because you would be fine with creating all of your project parameters through Biml. Unfortunately, most Biml users are not Biml purists – and even fewer development teams are Biml purists.

The next logical thought is to avoid the overwrite problem through merging. Certainly BimlExpress could insert the generated project parameter elements into the existing Project.params file. Duplicate values could be ignored. Unfortunately, this only solves the problem for the very first version of your generated Project.params file. What happens when you change the names of the parameters in your Biml code or delete some of them? The BimlEngine has no way of knowing that a given parameter in your Project.params file was deleted or renamed and not just created by hand. This would lead to a potentially large number of orphaned project parameters that you would need to manually manage.

Perhaps, you might suggest, we could add some form of XML annotation to the Biml-generated elements in the Project.params file to solve or at least improve the merging capabilities? That could be a great solution, but Visual Studio / SSDT strips any additional properties you add whenever it saves the Project.params file. Even if Visual Studio / SSDT preserved those additional annotations, this could still be a risky strategy, since other Microsoft and 3rd party tools have the potential to fail or otherwise misbehave if the BimlEngine diverges from the standard accepted encoding for the Project.params file.

This gets even worse for project parameters in the scenario where the user has unsaved manual changes to the Project.params file at the time when generation is performed. These unsaved changes are impossible to detect and merge because the SSIS project system does not expose those changes to add-ins until they have been saved in the Project.params file. This means that parameter changes would have to be force-saved prior to any Biml generation, but would then still suffer from all of the above issues.

Can’t you do anything? Even with an optional setting?
Based on the above analysis, the only scenario that doesn’t create more user confusion and frustration is the Biml Purist scenario. For this case, we might be able to offer an option to always overwrite the Project.params file, but that would satisfy a minority of users and would also be very frustrating in cases where you forgot to turn it off for non-purist projects. Our thinking is that it is not worth the trouble it might cause to add this option.

Thank you, Scott!

Scott CurrieThank you to Scott Currie (@ScottCurrie) for taking the time to give us such a detailed explanation! I really appreciate it, and I hope this helps all of you Biml users who have been wondering why you can’t create SSIS project parameters from Biml. (Well, you can, but as you have probably realized by now – it can be risky.)

Finally, both Scott and I would love to hear your thoughts on this. Would you like to see an optional setting? Is it worth the risk, especially when working with Biml-generated packages and manually created packages? Is it worth the risk when working in mixed development teams?

What do you think?

Page 1 of 7 1 2 3 4 5 ...Last »