Automate your SSIS deployment process

Prezentacja na temat: "Automate your SSIS deployment process"— Zapis prezentacji:

1 Automate your SSIS deployment process
Bartosz Ratajczyk Automate your SSIS deployment process

2 BIG Thanks to SQLSat Denmark sponsors

3 What is he talking about?
About making deployments easier removing manual tasks Things tested in practice starting with small steps From multiple scripts to one script deployment or using one/two steps in TFS How to build your own solution About different options and hurdles

4 I will not cover Single package deployments Best practices DevOps

5 Bartosz Ratajczyk SQL Server 7N Speaker SQL Saturday volunteer Advisory Board Member in Data Community Poland MCSE, MCT

6 All deployment steps Client side Server side Compile project
Deploy project Server side Create folder Create environment Reference environment Create environment variables Configure project Validate project Verify validation Test run Testing (NBi, LegiTest) Naming convention compliance (ie. Jamie Thomson)

7 Raising the bar Version control DEV, TEST, PROD branches Different servers DEV, TEST, QA, PREPROD, PROD Different configurations Drive letters, folder names Connection strings We have more than one project

8 Let’s start Making THE worK EASIER SQLDay 2017

9 Compile project – first steps

10 Compilation problems EncryptAllWithPassword, EncryptSensitiveWithPassword Manual password entry The devenv.com process can hang EncryptAllWithPassword, EncryptSensitiveWithPassword requires manual password entry with devenv.com

11 Handcrafting the package
.ispac file is just .zip with another extension name. Contains lackages, connection managers, project parameters – the same as in the SSDT project. Two additional files – manifest and data types. Does not contain the files from „Miscellaneous” folder in SSDT project Nie zawiera plików, które mamy w sekcji Miscellaneous, bo one nie są serwerowi potrzebne do szczęścia

12 MSBuild By default, MSBuild does not know the project type, so we have to help him a bit

13 SSISMSBuild SSIS compilation and deployment tasks for MSBuild
DeploymentFileCompilerTask DeployProjectToCatalogTask Source code, build yourself the .dll How to compile SSISMSBuild:

14 Compilation: Build.proj
Three / four elements to write: Project, UsingTask, Target and optional parametrization– ItemGroup, ParamGroup. I’m putting the Build.proj in the same folder as my SSIS project andI don’t provide the full path – relative path will do. Tutaj wybrano ItemGroup, ale PropertyGroup też będzie dobre, może nawet lepsze, bo Property w ramach PropertyGroup ma jedną wartość, a Item w ramach ItemGroup może mieć kilka (jak tablica)

15 Deployment – IsDeploymentWizard
IsDeploymentWizard.exe /Silent

16 IsDeploymentWizard /Silent
/Silent == fire and forget There might be a situation, where /Silent has problems e.g. external library in Script Task Found an example with script task - /Silent had a problem, without /Silent all went OK

17 MSBuild, again This time I use PropertyGroup instead of ItemGroup; it doesn’t matter if we don’t want to overwrite them from command line (if we do – use PropertyGroup)

18 DEMO Compilation and deployment
Demo 01: Compile using Build.proj (command line) Demo 02: Deploy using Deploy.proj (cmmand line) SQLDay 2017

19 Other methods PowerShell script using SMO SSISBuild
SSISBuild PowerShell Continuous Integration (PSCI) SSIS Build & Deploy SMO SSISBuild PowerShell cmdlets + cmd tools Nie wymaga żadnych dodatkowych bibliotek czy instalacji, nie zależy od VS PSCI Zależy od VS IsDeploymentCmd

20 What’s with the name? Jeśli nazwa projektu zawiera kropki, to podczas tworzenia pliku ispac ostatni człon nazwy po kropce jest ignorowany – zarówno w kompilacji z poziomu Visual Studio jak i przez SSISMSBuild Podejście z SMO i PowerShell jest wrażliwe na wersję assembly, np. w moim przypadku dla SSIS 2012 bibliotekę ManagedDTS 11.0 miałem tylko w wersji 32 bit i kompilacja musiała być wykonywana w PowerShell 32bit

21 Compilation: Build.proj (again)
Żeby obejść problem z kropkami w nazwie bez konieczności dostosowywania kodu SSISMSBuild możemy dopisać krok, który zmieni nazwę wygenerowanego pliku .ispac

22 Handling the Environments
Visual Studio Configuration Manager PowerShell + SMO Array CSV Database JSON Skrypty Powershell + SSIS – Joost van Rossum JSON:

23 Configuration Manager

24 Configuration Manager - usage
Different parameter values for different configurations (think: local environments) Contained in .dtproj files No single window to configure them all

25

26 Create the Environment
Environment-<NAME>.csv

27 Reference the Environment

28 Configure the project Environment-<NAME>-Configuration.csv
Przykłady – Koen Veerbeck: Hans Michiels – wszystko w T-SQL: Environment-<NAME>-Configuration.csv

29 Data sensitivity Encryption Certificates
Certificates

30 Create the environment and configure the project
DEMO SQLDay 2017

31 Validation Validation is an asynchronous process We start the process and track the progress

32 Validation from code ::UseAllReferences ::SpecifyReference, $reference
$project.Validate() [Microsoft.SqlServer.Management.IntegrationServices.ProjectInfo+ReferenceUsage]::UseAllReferences [Microsoft.SqlServer.Management.IntegrationServices.ProjectInfo+ReferenceUsage]::SpecifyReference, $reference)

33 Test run – the job

34 Validation and test DEMO SQLDay 2017

35 What else? NBi tests Jamie Thomson’s naming convention

36 And how about TFS? Custom proces for building and deployment You can expand it with PowerShell steps to control environments, validation, …

37 DEMO Automatic proces using TFS
CI/CD in TFS demo – single checkin to TFS repository runs automatic build and deployment SQLDay 2017

38 So simple! Use additional software for CI/CD process
The work is still boring, but this time the computer does it For automation use MSBuild Powershell SMO Use additional software for CI/CD process

39 BIG Thanks to SQLSat Denmark sponsors