Working on your 20th staging package? Do you need to import 30 different flat files? Don’t have the time to create all those packages? By using Biml – Business Intelligence Markup Language – you can describe your BI solution in a declarative fashion. You can turn tedious, repetitive work into reusable scripts which generate SSIS packages for you, cutting back drastically on development time. This session will introduce you to Biml and get you started on building reusable patterns for common BI tasks.

1. CREATE ETL SOLUTIONS FASTER
WITH METADATA DRIVEN DEVELOPMENT
KOEN VERBEECK
SQL SERVER DAYS 2013

2. WHO AM I?

3. OUTLINE
Introduction
Hello World
Read Flat
File
Read Flat
Files While
Looping
Metadata
driven
development
Conclusion

4. OUTLINE
Introduction
Hello World
Read Flat
File
Read Flat
Files While
Looping
Metadata
driven
development
Conclusion

5. INTRODUCTION

6. INTRODUCTION
• large percentage of BI projects fail
• Gartner - http://www.gartner.com/newsroom/id/492112
• one of the reasons is underestimating development effort ETL
• Kimball: 70% time of building a DWH goes into ETL
http://www.informationweek.com/the-38-subsystems-of-etl/55300422

7. INTRODUCTION
“I choose a lazy person to do a hard
job. Because a lazy person will find
an easy way to do it”
Bill Gates

8. INTRODUCTION
• a lot of SSIS packages are very similar
•
•
•
•
•
packages importing flat files
packages writing change data to staging tables
packages exporting data to excel (for some reason)
packages updating dimensions
…
• … but they take a lot of time to create

9. INTRODUCTION
• solution?
• code reuse
o SSIS basically only supports copy-paste
o copy-paste has improved in SSIS 2012
• design patterns
o for example: incremental load package
o SQL Server 2012 Integration Services Design Patterns
• enable through templates
o build a template package
o save it to C:Program Files (x86)Microsoft Visual Studio
11.0Common7IDEPrivateAssembliesProjectItemsDataTransformationProjectDataTransformationIt
ems
• but still requires you to edit each package!
• (and what if you forget to edit a crucial piece?)

10. INTRODUCTION
• metadata driven development to the rescue!
• (aka code generating code)
• automate generation of common logic in SSIS packages
• first option is the “dynamic SSIS package”
1.
2.
3.
4.
reads metadata from tables
generates code
o usually outputs T-SQL or bcp commands
o uses T-SQL or C#
o for example: SELECT … INTO statements
loops over the generated code
executes each statement
• disadvantages
•
•
•
•
complex project
no parallelism
difficult row based error handling
difficult to incorporate “business logic”

11. INTRODUCTION
• second option: BIML
• started as a project at MS: http://vulcan.codeplex.com/
• developer left to found company Varigence http://www.varigence.com/
o took the idea (not the code) and developed BIML
• BIML is a markup language and compiler
o translates metadata into business intelligence solutions for SQL Server
o supports SSIS and SSAS
o Varigence made part of BIML available as open source

12. INTRODUCTION
• BIDS Helper has open source implementation of BIML
• it’s free!
• it’s already in the add-on you love!
• it is available for SSIS 2005, 2008, 2008R2, 2012 (and 2014?)
• BIML offers
• powerful code generation
o only some parts of the project deployment model are not supported
• reuse BI patterns and components
o create your pattern in BIML and generate all your packages with the same structure
o BIML files can reference each other
• .NET based script language
o C# code can be incorporated into BIML to generate objects based on metadata
o Intellisense (sometimes) available
• don’t like BIML?
o generated packages are just SSIS packages, you can edit them using BIDS/SSDT/SSDTBI
o no vendor lock-in

13. INTRODUCTION
• scenario for our demos
• import different flat files
o exports from ERP systems, other database vendors, 3rd party providers, …
• each type of flat file has a different structure
o no single SSIS package for all flat files
• the name of the flat files can change
o for example the name includes a timestamp
• this would normally require 1 SSIS package per flat file type
• couple of hours/days work?
• let’s solve it with BIML!

14. OUTLINE
Introduction
Hello World
Read Flat
File
Read Flat
Files While
Looping
Metadata
driven
development
Conclusion

15. HELLO WORLD
• basic BML script structure
Tasks
BIML
Dataflow
Connections
FileFormats
Packages
Tasks
Containers
Precedence constraints
Transformations
You can also specify
• events
• log handlers
• variables
• parameters
• custom tasks
• script tasks/components
• …

16. HELLO WORLD
• let’s take a look at a simple BIML script

17. HELLO WORLD
• BIML root node
• add connections
• add packages

18. HELLO WORLD
• specify Tasks
• specify specific properties

19. HELLO WORLD
• check for errors & generate package
• result

20. DEMO
show Hello World BIML

21. OUTLINE
Introduction
Hello World
Read Flat
File
Read Flat
Files While
Looping
Metadata
driven
development
Conclusion

22. READ FLAT FILE
• specify FlatFileFormat
• columns: name, data type, size, delimiter (, code page)
• what you’d normally specify in the flat file connection manager
• specify connection

23. READ FLAT FILE
• specify data flow with transformations
• if no input/output connectors are specified, transformations are connected in
the order specified in the BIML file
• result

24. DEMO
import flat file with BIML

25. OUTLINE
Introduction
Hello World
Read Flat
File
Read Flat
Files While
Looping
Metadata
driven
development
Conclusion

26. READ FLAT FILE IN LOOP
• now let’s loop over a bunch of flat files
• specify variables to hold path to current file and source folder
• add an expression on the flat file connection manager

27. READ FLAT FILE IN LOOP
• add a for each loop
• which has its own tasks child element

28. READ FLAT FILE IN LOOP
• result

29. DEMO
import flat file using for each loop with BIML

30. OUTLINE
Introduction
Hello World
Read Flat
File
Read Flat
Files While
Looping
Metadata
driven
development
Conclusion

31. METADATA DRIVEN DEVELOPMENT
• BIML is nice
• … but isn’t the GUI much faster to developer packages?
• time to enhance BIML with some C# goodness!
called BIMLScript
use C# to read metadata
loop over metadata and create multiple objects
entire website dedicated with tutorials and code snippets
http://bimlscript.com/
• also has an online editor
•
•
•
•

32. METADATA DRIVEN DEVELOPMENT
• Add namespaces
• Declare variables

33. METADATA DRIVEN DEVELOPMENT
• Retrieve metadata (stored in a SQL Server table)
• Loop over metadata and create corresponding objects

34. METADATA DRIVEN DEVELOPMENT
• result

35. METADATA DRIVEN DEVELOPMENT
• remarks
• make sure the code or the metadata doesn’t contain invalid XML characters
o <>“&
• using C# can mess with the Intellisense
o Visual Studio thinks it’s not valid XML anymore
o color coding can disappear > right click file and choose Open With…
o Intellisense can stop working in Visual Studio > use online editor
• beware of the protection levels
• some elements can only appear once
o do not put those in a loop
o e.g. Connections, Packages

36. DEMO
generate multiple packages using BIMLScript

37. OUTLINE
Introduction
Hello World
Read Flat
File
Read Flat
Files While
Looping
Metadata
driven
development
Conclusion

38. CONCLUSION
• BIML can radically reduce SSIS development time
• for frequently used package patterns
• when combined with BIMLScript
• BIML supports all versions of SSIS
• but some project deployment functionality is missing
• bit of a learning curve
• good understanding of SSIS is necessary
• basic C# skills needed
• return of investment is in next projects

39. RESOURCES
• Official BIML
• Varigence BIML product page
http://www.varigence.com/Products/Biml/Capabilities
• BIMLScript resource hub
http://bimlscript.com/
• BIDS Helper on Codeplex
http://bidshelper.codeplex.com/
• Blogs
• Stairway to BIML by Andy Leonard
http://www.sqlservercentral.com/stairway/100550/
• BIML articles by Joost van Rossum
http://microsoft-ssis.blogspot.be/search/label/BIML
• BIML articles by Marco Schreuder
http://blog.in2bi.eu/tags/biml/
• BIML articles by John Welch
http://agilebi.com/jwelch/tag/biml/
• Introduction to BIML part I by Koen Verbeeck
http://www.mssqltips.com/sqlservertip/3094/introduction-to-business-intelligence-markup-languagebiml-for-ssis/

40. Q&A
SQL SERVER DAYS 2013

41. THANKS FOR LISTENING!
koen.verbeeck@element61.be
@Ko_Ver
http://www.linkedin.com/in/kverbeeck
SQL SERVER DAYS 2013

42. © 2011 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market
conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation.
MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.