SQL Server Integration Services – Enterprise Manageability

SQL Server Integration Services –
Enterprise Manageability

David Pendleton
Dan English
Senior Consultant
Principal Consultant
DavidPe@magenic.com
DanE@magenic.com

Who are we? – Dan and David
Dan English David Pendleton
http://denglishbi.spaces.live.com/

• Developing with Microsoft technologies for over
10 years • 20+ years experience as a systems analyst,
• database developer, designer and software
Over 5 years experience with Data Warehousing
and Business Intelligence engineer
• •
Experienced in ETL and Analysis Services Experienced technician with strengths in
development, requirements gathering and data database design and development, systems
modeling architecture and software development
• •
Microsoft Certified IT Professional (MCITP) and Specialized in ETL, database modeling and
Microsoft Certified Technology Specialist (MCTS) design, SQL development, and change
management

Who are we? – Magenic
 Founded in 1995, Magenic is a technical consulting firm
focused exclusively on Microsoft technologies and has
designed and delivered more than 500 Microsoft-based
applications
 Headquartered in Minneapolis, with offices in Chicago,
Boston, Atlanta and San Francisco
 2005 Microsoft Partner of the Year, Custom
Development Solutions – Technical Innovation
 2007 Microsoft Partner of the Year Finalist, Data
Management
 Microsoft Gold Certified Partner and National Systems
Integrator
 40 Enterprise Data Services (EDS) consultants

Today‟s Agenda
• SQL Server Integration Services Introduction and Architecture
•8:30 to 9 (Dan English)

• Team Development, Deployment and Configuration
•9:15 to 10:30 (David Pendleton)

• Security and Manageability
•10:45 to 11:30 (Dan English)

• Summary and Q&A

Quick Audience Poll
•How many DBAs? SQL/BI Developers? Data Architects? Others?

•How many are currently evaluating SSIS?

•How many are currently using SSIS in production?

•How many are currently using SSIS 2008?

•Is SSIS being used for something besides DB Maintenance or ETL?
•If so what?

•Anyone still using DTS?

SSIS Introduction

SQL Server 6.5
SQL Server SQL Server
• Transfer
2000 2008
Manager
• DTS • SSIS
• Bulk Copy
Enhancements Enhancements
Program

7.0 2005
• Data • Integration
Transformation Services
Services

Microsoft BI Tool Offerings
DELIVERY

SharePoint Server

Analytic
Excel Scorecards Plans
Reports Dashboards
Views
Workbooks

END USER TOOLS & PERFORMANCE MANAGEMENT APPS
Excel PerformancePointServer

BI PLATFORM
Reporting Services Analysis Services
SQL Server DBMS

SQL Server Integration Services

SSIS Introduction
What can you use SSIS for?

1. Extract, Transform, Load data from heterogeneous sources
2. Combine, sanitize, standardize, split, and summarize data
3. Convert data to different data types and put into different formats and sources
4. Database maintenance and transfer operations
• Backup, check integrity, execute jobs, cleanup, rebuild indexes, update statistics, etc.
5. Transfer SQL Server database objects
6. Workflow engine supporting complex logic and precedence constraints
7. Perform Analysis Services processing and Data Mining querying
8. Use as a data source for Reporting Services report
9. Perform FTP operations
10. Extensible to create your own components (sources, transforms, control flow, etc.)
11. Utilize scripting tasks and access .NET code with VSTA (Visual Basic .NET or C#)
12. Send email notifications
13. Access Web Services
14. File archiving and maintenance tasks
15. And much, much more…

SSIS Architecture
SSIS is made up of two core components:

1. Data Transformation Run-time engine
• Handles the control flow of a package

2. Data Flow engine
• Data Transformation Pipeline engine
• Data Sources
• Transformations
• Destination Targets

SSIS Architecture
Data Transformation Run-time engine

• Provides services to the packages and the
components

• Handle the storage
• Execution
• Logging
• Debugging
• Event handling
• Deployment
• Variable management
• Transactions
• Connections

SSIS Architecture

•The „package‟ is the core component of SSIS. SSIS
packages are the basic unit of deployment and execution.

•An SSIS package is XML.

•Packages can be created using BIDS or
programmatically.

SSIS Architecture

What makes up a
package?

1.Control Flow
2.Precedence constraints
3.Data Flow

• Control Flow
• Containers
• Tasks

SSIS Architecture
• Precedence constraints
• Controls order of execution
• Success, Failure,
Completion
• Expressions
• AND/OR logic

SSIS Architecture

• Data Flow
• Source adapters
• Transformations
• Destination adapters

SSIS Architecture Enhancements
• What‟s new in SSIS 2008?
– Improved Scripting with VSTA including support for C#
– Data Flow Task performance improvements with smarter thread
allocation and processing of execution trees
– ADO NET source and destination adapters
– Import/Export Wizard enhancements
• ADO.Net support, data conversions (XML file), performance improvements
– Cache Transform and Cache Connection Manager for Lookup
Transform (Lookup Transform improved also – no match output)
– Utilize CDC and Merge for incremental updates
– Data Profiling Task (and Viewer)

SSIS Architecture Enhancements

Data Profiling Demo

SSIS Enterprise Manageability
• SSIS Package Deployment
– The SSIS runtime environment
– Package deployment
– Securing packages
• SSIS Configurations
– Understanding and using SSIS package configurations
• Team Development using SSIS
– SSIS package development in a team environment
• Demo: Tying it all together

The SSIS runtime environment

• Does not require SQL Server to run*
– Although SQL Server is probably involved somewhere
• Does not require the SSIS Service to run
– The SSIS service enables package management and
monitoring via SSMS
• Any server can host SSIS packages
– Provided the minimum requirements are met
• .NET Framework, SQL Native Client, Setup support files

• Installed using SQL Server Setup

Package Deployment

• In general, there are two ways to deploy a
package to a server
– File System
– SQL Server (msdb..sysdtspackages90)
• Package Deployment Utility
– Configured in BIDS, created by the build process
– Uses the .SSISDeploymentManifest file extension
• Deployment can also be done “manually”, e.g.
Xcopy, SMSS import

Package Deployment

• Deployment option advantages compared
– File System
• OS-based file encryption and ACLs
• Not subject to SQL Server availability
• Easier direct access for viewing or editing
• Generally easier on the developer
– SQL Server
• Easier access by multiple individuals
• Benefits from database security, roles and Agent interaction
• Packages get backed up with normal database backup
processes
• Generally easier on the DBA

Package Deployment
• Creating the Package Deployment Utility
– Accessed via Project Properties in BIDS

Securing your packages
• SSIS offers these options for securing packages
– Setting the ProtectionLevel property of a package
– Setting the ProtectionLevel and PackagePassword properties of
a package
– Controlling access by using SQL Server database-level roles
– Securing the operational environment by controlling access to
file locations
– Guaranteeing the integrity of a package by signing with a
certificate.

• The package ProtectionLevel property
– Determines the way package content is secured
• DontSaveSensitive
• EncryptSensitiveWithUserKey
• EncryptSensitiveWithPassword
• EncryptAllWithPassword
• EncryptAllWithUserKey
• ServerStorage

• DontSaveSensitive
– Removes sensitive information from the package during the
save, e.g. passwords
– When the package is opened, the removed items will have to be
provided
– Scheduled or unattended execution will likely fail
– Actually your best option for most development

• EncryptAllWithUserKey
– Encrypts the entire package by applying a user-specific key
securely stored in the user‟s profile
– The package can only be opened by the same user with access
to the same user profile.
– Guaranteed to make you unpopular with your team members

• EncryptSensitiveWithUserKey
– Encrypts sensitive information in the package using a user-
specific key securely stored in the user‟s profile
– The package may be edited by other users, but the sensitive
information will be automatically removed
– Execution without access to the encryption key will fail
– This is the default level

• EncryptAllWithPassword
– Encrypts the entire package with a user-supplied password
– Password is required to open or edit
– Requires the /De[crypt] password option to run with DTEXEC

• EncryptSensitiveWithPassword
– Encrypts sensitive in formation in the package using a user-
supplied password
– Package may be edited without the password, but sensitive
information is automatically removed
– Requires the /De[crypt] password option to run with DTEXEC

SSIS package Configurations
• Configurations allow you to modify package
property and variable values at load time
• Available in several types
– SQL
– XML
– Registry
– Environment Variables
– Parent Package
• You can mix-and-match configurations in a
package

SSIS Package Configurations
• SQL configuration
– Configurations are stored in a SQL Server table
– You may call the table anything you like
– May reside in any database, on any SQL Server
– Contains one or more configurations
• XML configuration
– Plain-text XML file
– Contains one or more configurations
– Uses the file extension .dtsConfig by default
– Easily deployed, easily shared

• Registry configuration
– Configurations are stored in a registry key
– Restricted to the HKEY_CURRENT_USER hive
– Property path is stored in the package
• Environment Variable configuration
– Package property value is stored in an environment variable
– Property path is stored in the package
– Not the same as an Indirect Configuration*
• Parent Package configuration
– Used to fetch a value from a variable in a calling package

• Understanding how configurations are applied
– SSIS 2005
1. The package is loaded
2. Configurations specified at design time are applied in the order
they appear in the package
3. Command-line configuration values are applied
4. Parent Package variable configuration values are applied
5. The package is run

• Understanding how configurations are applied
– SSIS 2008
1. The package is loaded
2. The configurations specified at design time are applied in the
order they appear in the package
3. Command-line configuration values are applied
4. The configurations specified at design time are reloaded in the
order specified in the package
5. Parent Package variable configuration values are applied
6. The package is run

Team Development and SSIS
• Use a source control system
– SSIS packages are stored as XML, but are effectively binary files
– Merging packages is difficult, if not impossible
– Only one developer can work on a package at a time
• Require exclusive check-outs
– Both Visual SourceSafe and Team Foundation Server will permit
shared check-outs; this should be disabled
• Perform regular source control operations
– Ensure you are always working with the latest version of a
package, check-in incrementally

• Design your ETL process with team
development in mind.
– One package; one table
– Package functionality should be as granular as is practical
– Use Parent packages and the Execute Package Task to group
packages together for end-to-end execution
– Spread the work around
• Your packages
• Your team

• Create and use package templates
– A template is simply an SSIS package
– Any package can be used as a template
– Templates can be directly visible to BIDS, or not
• A template in the DataTransformationItems template folder
will appear in the Add|New Item dialog
– Implement and reinforce standard practices in your templates
• Logging
• Checkpointing
• Configurations
• Annotations

• Security considerations
– Use Windows Authentication whenever possible, both in
development and production
– Manage the package ProtectionLevel property
• ProtectionLevel = DontSaveSensitive
– Maximum flexibility
– Sensitive information can be managed in SQL configurations

• Deploy your packages to the File System
• Use a common folder structure
– Provides a consistent, known file location regardless of the
server or environment
– All files of a certain type are stored in the same place
– Allows you to make all locations dynamic within your package
– Easier to manage

• Common folder structure: the SSIS Root Folder
..SSIS
..SSISCheckpointFiles
..SSISConfigurations
..SSISErrorFiles
..SSISLogFiles
..SSISPackages
..SSISRawFiles

• Use project-specific sub-folders to group related
files below the root folders
• Edit MsDtsSrvr.ini.xml to use Packages as the
SSIS Package Store

• Machine-specific Configuration file
– XML configuration
• Accessed via indirect configuration
– Only one configuration file per machine
– Contains two (and only two) configurations
• RootFolder
– Contains the path to the root level of the custom folder structure
• ConfigurationServer
– Contains the connection string to the SQL configuration server

• SQL Server Configuration
– Easily managed using DDL
– Adding an identity column helps with DML tasks
• ConfigurationID int IDENTITY(1,1) NOT NULL
– Store multiple configurations in the same table
– Allows multiple packages to access the same configurations
– All the benefits of SQL Server security
• Not only sensitive information, but all information can be
protected
– Offers the maximum flexibility

Demo: Tying it all together
• Creating the common folder structure
• Redirecting the SSIS Package Store
• Implementing the machine-specific configuration
file
• Implementing the SQL configuration database
and table(s)
• Leverage the common folder structure in a
template using variables and expressions
• Making a template available to BIDS

SSIS Security
• Security in SQL Server Integration Services consists of several layers that provide a
rich and flexible security environment.

– These security layers include the use of:
• digital signatures
• package properties
• SQL Server database roles
• and operating system permissions

• Security categories - Identity or Access control
– Identity control goal: Ensure that you only open and run packages from
trusted sources. (digital signatures)

– Access control goal: Ensure that only authorized users open and run
packages. (protection levels, encryption, msdb storage, db roles, file
permissions)

SSIS Security – database roles
• SQL Server Integration Services includes the three fixed database-
level roles for controlling access to packages
db_ssisadmin (sysadmin) – can do anything related to all packages
1.
db_ssisltduser – can import packages; view, delete, execute, export, and
2.
change own package roles
db_ssisoperator – can view, execute, export all packages (including SQL
3.
Server Agent) and they cannot import packages
And Windows administrators – can view execution details and stop all
4.
running packages

• Roles can be implemented only on packages that are saved to the
msdb database in SQL Server. You assign roles to a package using
SQL Server Management Studio. The role assignments are saved to
the msdb database.

SSIS Manageability - SSMS

• The Integration Services server
node presents two folders for
working with SSIS packages:

– the Running Packages folder
• The Running Packages folder displays
the SSIS packages that are currently
executing on the local server

– and the Stored Packages Folder.


• The Stored Packages folder lists
the saved SSIS packages that
have been registered on the local
server. By default this folder
contains two subfolders:

– the File System Folder

– and the MSDB folder

• Right-clicking a package displays a shortcut menu
that enables you to perform a number of task
including:

• New Folder. Creates a new folder in Object Explorer
for displaying packages saved in the file system or
in the sysdtapackages90 table.

• Import Package. Imports the package from the file
system to the msdb database

• Export Package. Exports the package from the
msdb database to the file system.

• Run Package. Executes the package using
dtexecui.

• Delete. Deletes the package.

• Rename. Renames the package.

SSIS Manageability - dtutil
• SSIS Package Management
Utility (dtutil)

SSIS contains the Package
Management Utility (dtutil) that
you can run from a command line.

The dtutil command prompt
allows you to manage SSIS
packages that are stored in the
database or the file system.

You may specify to copy, move,
delete, or verify the existence of a
package using the appropriate
command prompt options.

• To copy a package that is stored in the msdb database on a local instance of SQL Server using Windows
Authentication to the SSIS Package Store, use the following syntax:

dtutil /SQL srcPackage /COPY DTS;destFolderdestPackage

• To copy a package from a location on the File system to another location and give the copy a different name, use
the following syntax:

dtutil /FILE c:myPackagesmypackage.dtsx /COPY FILE;c:myTestPackagesmynewpackage.dtsx

• To copy a package on the local file system to an instance of SQL Server hosted on another computer, use the
following syntax:

dtutil /FILE c:sourcepkg.dtsx /DestServer <servername> /COPY SQL;destpkgname

• To create a new ID for a package after it is copied, use the following syntax:

dtutil /I /FILE copiedpkg.dtsx

• To create a new ID for all the packages in a specific folder, use the following syntax:

for %%f in (C:testSSISPackages*.dtsx) do dtutil.exe /I /FILE %%f

• To determine whether a package exists in the msdb database on a local instance of SQL Server that uses SQL
Server Authentication, use the following syntax:

dtutil SQL srcPackage /SOURCEUSER srcUserName /SOURCEPASSWORD *hY$d56b /EXISTS

• To move a package that is stored in the msdb database on a local instance of SQL Server that uses SQL Server
Authentication to the msdb database on another local instance of SQL Server that uses SQL Server
Authentication, use the following syntax:

dtutil /SQL srcPackage /SOURCEUSER srcUserName /SOURCEPASSWORD $Hj45jhd@X /MOVE
SQL;destPackage /DESTUSER destUserName /DESTPASSWORD !38dsFH@v

• To sign a package that is stored in a SQL Server database on a local instance of SQL Server that uses Windows
Authentication, use the following syntax:

dtutil /FILE srcPackage.dtsx /SIGN FILE;destpkg.dtsx;1767832648918a9d989fdac9819873a91f919

• The following sample encrypts the file-based PackageToEncrypt.dtsx to the file-based EncryptedPackage.dtsx
using full package encryption, with a password. The password that is used for the encryption is EncPswd.

• dtutil /FILE PackageToEncrypt.dtsx /ENCRYPT file;EncryptedPackage.dtsx;3;EncPswd

SSIS Manageability

Stored Package
Configuration Demo

SSIS Manageability - package
• Checkpoint Restart – recover from a package failure; information about the packages execution
and status is written to a checkpoint file. The Data Transformation Run-time engine can process
the checkpoint file and the package can be restarted from the container that was being processed
at the point of failure

• Transaction Support – SSIS provides full support for database transactions. If enabled and the
package fails that contains data flow tasks, all of the database update actions performed will be
rolled back.

• Transformation Error Handling – you can choose to ignore bad data or you can choose to
redirect the row to a file or table for review and later processing; setup a central database for a
central point of auditing and review.

• Event Handlers – provide the ability to raise and handle events raised by containers and tasks.
These could be used to send e-mail notifications, create logging information, or remove temporary
files or database objects.

• Execute Package Tasks – provide ability to run one package from another package; pass
information from parent package to child package and setup a master package to control the flow
of other packages.


Event Handlers Demo


Create smaller packages
• Pros
– less complex child
packages
– call them from multiple
parent packages
– simplifies development
process
– promotes reusability
and manageability

SSIS Manageability – package execution

Scheduling Package Execution
– SQL Server Agent
• Requires SQL Server installation
• Setup credentials
• Setup proxies
• Create SQL job
– Windows Task Scheduler
• Setup user account for task
– 3rd party scheduler


SQL Server Agent Job
Demo


Remote Package Execution
– SQL Server Agent job (sp_start_job)
– Execute SQL Server Agent Job task (SSIS task)
– .NET Code (SSIS Object Model)
– Web Service (IIS – ASP.NET)
– HTTP Endpoint
– WCF application


Remote Package
Execution Demo

SSIS Manageability – package logging
• The SQL Profiler Log Provider - The SQL Profiler log provider
enables you to create logs that can be opened using the SQL
Profiler.

• The SQL Server Log Provider - Using the SQL Server log provider
enables you to capture SSIS package run-time data in the
sysdtslog90 or sysssislog table of the database.

• The Windows Event Log Provider - writes entries to the
Application log in the Windows Event log on the local computer.

• The XML File Log Provider - writes log files to an XML file and the
default file extension is .xml.

• The Text File Log Provider - writes log entries to ASCII text files in
a comma-separated value (CSV) format. The default file name
extension for this provider is .log.

• Custom Log Provider - You can create your own custom log
provider to obtain information to accommodate your business
needs. Using a custom log provider allows you to integrate SSIS
logs within your applications.

Events Description

OnError Writes a log entry when an error occurs.

OnExecStatusChanged Writes a log entry when the execution status of the executable changes.

OnInformation Writes a log entry during the validation and execution of an executable to report information.

OnPostExecute Writes a log entry immediately after the executable has finished running.

OnPostValidate Writes a log entry when the validation of the executable finishes.

OnPreExecute Writes a log entry immediately before the executable runs.

OnPreValidate Writes a log entry when the validation of the executable starts.

OnProgress Writes a log entry when measurable progress is made by the executable.

OnQueryCancel Writes a log entry at any juncture in the task processing where it is feasible to cancel execution.

OnTaskFailed Writes a log entry when a task fails.

OnVariableValueChanged Writes a log entry when the value of a variable changes.

OnWarning Writes a log entry when a warning occurs.
For each data flow component, writes a log entry for each phase of validation and execution. The log entry specifies
PipelineComponentTime
the processing time for each phase.
Writes a log entry that provides diagnostic information.
Diagnostic
For example, you can log a message before and after every call to an external data provider.

SSIS Monitoring

•SQL Server Management Studio
•Logging
•Notifications
•Performance Monitor
•SSIS Service
•SSIS Pipeline
•PowerShell
•WMI

Today‟s Agenda
• SQL Server Integration Services Introduction and Architecture

• Team Development, Deployment and Configuration

• Security and Manageability

• Summary and Q&A

Resources
SQL Server 2008 – Integration Services Portal Page
http://msdn.microsoft.com/en-us/sqlserver/cc511477.aspx

SQLIS.com - SQL Server Integration Services - SSIS
http://sqlis.com

Jamie Thomson (UK) – SQL Server MVP – SSIS and forum moderator
http://blogs.conchango.com/jamiethomson/

SQL Server 2008 Upgrade Technical Reference Guide
http://www.microsoft.com/downloads/details.aspx?FamilyID=66d3e6f5-6902-4fdd-af75-9975aea5bea7&displaylang=en

Channel9 MSDN BI Screencasts
http://channel9.msdn.com/Showforum.aspx?forumid=38&tagid=277

SQL Server Best Practices
http://msdn.microsoft.com/en-us/sqlserver/bb671432.aspx

Microsoft Virtual Labs (TechNet and MSDN)
http://www.microsoft.com/events/vlabs/default.mspx

Microsoft BI Site
http://www.microsoft.com/bi/

Magenic Blogs
http://blog.magenic.com/blogs

Contact Information – Thank You!

Contact us to find out how your business can benefit
from a complimentary strategy session with one of our
consultants and look into one of our BI quickstart
engagements.

Dan - http://denglishbi.spaces.live.com
David -
Magenic - info@magenic.com

SQL Server Integration Services – Enterprise Manageability

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (16)

Similar to SQL Server Integration Services – Enterprise Manageability

Similar to SQL Server Integration Services – Enterprise Manageability (20)

More from Dan English

More from Dan English (16)

Recently uploaded

Recently uploaded (20)

SQL Server Integration Services – Enterprise Manageability