Automate your SSIS deployment process Bartosz Ratajczyk Automate your SSIS deployment process

BIG Thanks to SQLSat Denmark sponsors

What is he talking about? About making deployments easier removing manual tasks Things tested in practice starting with small steps From multiple scripts to one script deployment or using one/two steps in TFS How to build your own solution About different options and hurdles

I will not cover Single package deployments Best practices DevOps

Bartosz Ratajczyk SQL Server Consultant @ 7N Speaker SQL Saturday volunteer Advisory Board Member in Data Community Poland MCSE, MCT http://bartekr.net b.ratajczyk@gmail.com

All deployment steps Client side Server side Compile project Deploy project Server side Create folder Create environment Reference environment Create environment variables Configure project Validate project Verify validation Test run Testing (NBi, LegiTest) Naming convention compliance (ie. Jamie Thomson)

Raising the bar Version control DEV, TEST, PROD branches Different servers DEV, TEST, QA, PREPROD, PROD Different configurations Drive letters, folder names Connection strings We have more than one project

Let’s start Making THE worK EASIER SQLDay 2017

Compile project – first steps

Compilation problems EncryptAllWithPassword, EncryptSensitiveWithPassword Manual password entry The devenv.com process can hang EncryptAllWithPassword, EncryptSensitiveWithPassword requires manual password entry with devenv.com

Handcrafting the package .ispac file is just .zip with another extension name. Contains lackages, connection managers, project parameters – the same as in the SSDT project. Two additional files – manifest and data types. Does not contain the files from „Miscellaneous” folder in SSDT project Nie zawiera plików, które mamy w sekcji Miscellaneous, bo one nie są serwerowi potrzebne do szczęścia

MSBuild By default, MSBuild does not know the project type, so we have to help him a bit

SSISMSBuild SSIS compilation and deployment tasks for MSBuild DeploymentFileCompilerTask DeployProjectToCatalogTask Source code, build yourself the .dll https://www.simple-talk.com/sql/ssis/deployment-automation-for-sql-server-integration-services-ssis/ https://seddryck.wordpress.com/2016/12/27/building-the-msbuild-tasks-for-ssis-2016/ http://sqlsrvintegrationsrv.codeplex.com How to compile SSISMSBuild: https://www.simple-talk.com/sql/ssis/deployment-automation-for-sql-server-integration-services-ssis/ https://seddryck.wordpress.com/2016/12/27/building-the-msbuild-tasks-for-ssis-2016/

Compilation: Build.proj Three / four elements to write: Project, UsingTask, Target and optional parametrization– ItemGroup, ParamGroup. I’m putting the Build.proj in the same folder as my SSIS project andI don’t provide the full path – relative path will do. Tutaj wybrano ItemGroup, ale PropertyGroup też będzie dobre, może nawet lepsze, bo Property w ramach PropertyGroup ma jedną wartość, a Item w ramach ItemGroup może mieć kilka (jak tablica)

Deployment – IsDeploymentWizard IsDeploymentWizard.exe /Silent

IsDeploymentWizard /Silent /Silent == fire and forget There might be a situation, where /Silent has problems e.g. external library in Script Task Found an example with script task - /Silent had a problem, without /Silent all went OK

MSBuild, again This time I use PropertyGroup instead of ItemGroup; it doesn’t matter if we don’t want to overwrite them from command line (if we do – use PropertyGroup)

DEMO Compilation and deployment Demo 01: Compile using Build.proj (command line) Demo 02: Deploy using Deploy.proj (cmmand line) SQLDay 2017

Other methods PowerShell script using SMO SSISBuild http://muxtonmumbles.blogspot.dk/2013/11/ssis-project-deployment-and-visual.html SSISBuild http://roman.tumaykin.com/tools/2017/01/10/Continuous-integration-of-SSIS-projects-Part-1-build.html PowerShell Continuous Integration (PSCI) https://github.com/ObjectivityLtd/PSCI SSIS Build & Deploy https://marketplace.visualstudio.com/items?itemName=TG.VSTS-SSIS SMO http://muxtonmumbles.blogspot.com/2013/11/ssis-project-deployment-and-visual.html SSISBuild https://www.nuget.org/packages/SSISBuild/ https://github.com/rtumaykin/ssis-build/ PowerShell cmdlets + cmd tools Nie wymaga żadnych dodatkowych bibliotek czy instalacji, nie zależy od VS PSCI https://github.com/ObjectivityLtd/PSCI Zależy od VS IsDeploymentCmd https://github.com/tkwj/ISDeploymentCmd

What’s with the name? Jeśli nazwa projektu zawiera kropki, to podczas tworzenia pliku ispac ostatni człon nazwy po kropce jest ignorowany – zarówno w kompilacji z poziomu Visual Studio jak i przez SSISMSBuild Podejście z SMO i PowerShell jest wrażliwe na wersję assembly, np. w moim przypadku dla SSIS 2012 bibliotekę ManagedDTS 11.0 miałem tylko w wersji 32 bit i kompilacja musiała być wykonywana w PowerShell 32bit

Compilation: Build.proj (again) Żeby obejść problem z kropkami w nazwie bez konieczności dostosowywania kodu SSISMSBuild możemy dopisać krok, który zmieni nazwę wygenerowanego pliku .ispac

Handling the Environments Visual Studio Configuration Manager PowerShell + SMO Array CSV Database JSON Skrypty Powershell + SSIS – Joost van Rossum http://microsoft-ssis.blogspot.com/p/powershell.html JSON: http://microsoft-ssis.blogspot.com/2017/05/import-and-export-ssis-catalog.html

Configuration Manager

Configuration Manager - usage Different parameter values for different configurations (think: local environments) Contained in .dtproj files No single window to configure them all

Create the Environment Environment-<NAME>.csv

Reference the Environment

Configure the project Environment-<NAME>-Configuration.csv Przykłady – Koen Veerbeck: http://blogs.lessthandot.com/index.php/datamgmt/dbadmin/mssqlserveradmin/ssis-deployment-with-powershell-adding/ Hans Michiels – wszystko w T-SQL: https://www.hansmichiels.com/2016/11/04/how-to-automate-your-ssis-package-deployment-and-configuration-ssis-series/#step4 Environment-<NAME>-Configuration.csv

Data sensitivity Encryption Certificates http://www.codewrecks.com/blog/index.php/2014/07/12/store-secure-password-in-tfs-build-definition/ Certificates http://www.codewrecks.com/blog/index.php/2014/08/09/encrypt-your-password-in-tfs-build-using-certificates/ https://twitter.com/scottcurrie/status/779033445817278464

Create the environment and configure the project DEMO SQLDay 2017

Validation Validation is an asynchronous process We start the process and track the progress

Validation from code ::UseAllReferences ::SpecifyReference, $reference $project.Validate() [Microsoft.SqlServer.Management.IntegrationServices.ProjectInfo+ReferenceUsage]::UseAllReferences [Microsoft.SqlServer.Management.IntegrationServices.ProjectInfo+ReferenceUsage]::SpecifyReference, $reference)

Test run – the job

Validation and test DEMO SQLDay 2017

What else? NBi tests Jamie Thomson’s naming convention http://microsoft-ssis.blogspot.com/2016/12/ssis-naming-conventions.html http://www.nbi.io/ http://microsoft-ssis.blogspot.dk/2016/12/ssis-naming-conventions.html

And how about TFS? Custom proces for building and deployment You can expand it with PowerShell steps to control environments, validation, …

DEMO Automatic proces using TFS CI/CD in TFS demo – single checkin to TFS repository runs automatic build and deployment SQLDay 2017

So simple! Use additional software for CI/CD process The work is still boring, but this time the computer does it For automation use MSBuild Powershell SMO Use additional software for CI/CD process

BIG Thanks to SQLSat Denmark sponsors