Can we store our Connectionstrings or BlobStorageKeys or other Secretvalues somewhere else then in Azure Synapse Pipelines? Yes you can! You can store these valuable secrets in Azure Key Vault(AKV). • But how can we achieve this in Azure Synapse Analytics? • How do we deploy our Synapse Pipelines in Azure Dev Ops to Test, Acceptance and Production environments with these Secrets ? • Can this be setup dynamically? During this session I will give answers on all these questions. You will learn how to setup your Azure Key Vault, connect these secrets in Azure Synapse Analytics and finally deploy these secrets dynamically in Azure Dev Ops. As you can see a lot to talk about during this session.

1. InSpark
Azure Key Vault, Azure DevOps and
Azure Synapse Analytics
how do these Azure Services work perfectly together!
Data Saturday Stockholm
May 21 th
@erwindekreuk
https://erwindekreuk.com
Erwin de Kreuk

2. InSpark
We help organizations
accelerating their digital
transformation with impactful
Microsoft solutions & expertise
We Are InSpark

3. InSpark
Attollo B3 Consulting Group Quest Software SQL Service
Transmokopter SQL
dbWatch
Catman Solution DB24 Redgate Software

4. InSpark
Agenda
Azure Synapse Analytics
Azure DevOps
Azure KeyVault
How they work together

5. InSpark
The first unified, cloud native platform for
converged analytics
Azure Synapse is the only unified platform for analytics, blending big data,
data warehousing, and data integration into a single cloud native service
for end-to-end analytics at cloud scale.
The first unified cloud native platform
Azure Synapse Analytics
Data integration
Data warehousing
Big data analytics

6. InSpark
Powered by a new cloud native distributed SQL engine
Cloud-native analytics service engine
Azure Synapse Analytics

7. InSpark
Flexible consumption models
Serverless pay-per-query ideal for ad-hoc data lake exploration and
transformation
Dedicated clusters optimized mission-critical data warehouse workloads
Serverless + dedicated SQL
Azure Synapse Analytics
Serverless Dedicated

8. InSpark
“DevOps is the union of people,
process, and products to enable
continuous delivery of value to
your end users”
Donovan Brown

9. InSpark
Azure DevOps

10. InSpark

11. InSpark
The safehouse to safeguard the cryptographic keys and secrets that are used by your applications, servers and cloud
applications
Fully integrated with Azure Active Directory
Secret management: Securely store and tightly control access to tokens, passwords, certificates, API keys, and
other secrets.
Key management: Create and control encryption keys that encrypt your data.
Certificate management: Provision, manage, and deploy public and private Secure Sockets Layer/Transport Layer
Security (SSL/TLS) certificates for use with Azure and your internal connected resources.
What is Azure Key Vault?
Azure Key Vault

12. InSpark
Roles
Azure Key Vault
Auditor
Developer
Security Operations
Manages Keys
Creates a Key Vault in
Azure
Add keys / secrets to the
Key Vault
Grants Permission to
specific applications to
perform specific
operations using keys
Enables usage logs
Monitorsaccess to keys
Reviews usage logs to
confirm proper key use
and compliance with data
security standards
Deploys Application
Tells de application the
URI of the key / secret
Configures the
application to use key /
secret (and may abuse),
but never sees the keys

13. InSpark
ededeuwdvlmsynoxgn01 ededeuwprodsynoxgn01

14. InSpark
An even more secure approach
ededeuwdvlmsynoxgn01 ededeuwprodsynoxgn01

15. InSpark
Create Workspace
ededeuwdvlmsynoxgn01 ededeuwprodsynoxgn01
Create a Synapse Workspace for every environment
Define naming conventions
customername-region-environment-azureservice-department
Azure Synapse Analytics

16. InSpark
Set up Code Repository
Azure Synapse Analytics

17. InSpark
Set up Code Repository
Azure Synapse Analytics

18. InSpark
Azure SQL DB
LS_ASQL_ENV01_WWI
Azure Blob Storage
LS_ABLB_ENV01
Directly saved in Azure DevOps !!
Create Linked Services
Azure Synapse Analytics

19. InSpark
Create Linked Services
Azure Synapse Analytics
More naming conventions check:
https://erwindekreuk.com/2019/04/azure-data-factory-naming-conventions/
Azure SQL DB
LS_ASQL_ENV01_WWI
Azure Blob Storage
LS_ABLB_ENV01

20. InSpark
DS_ABLB_ENV01
DS_ASQL_ENV01_WWI
Create Datasets
Azure Synapse Analytics

21. InSpark
PL_COPY_WWI_ABLB
Create Pipeline
Azure Synapse Analytics
DS_AQSL_
ENV01_W
WI
DS_ABLB_
ENV01

22. InSpark

23. InSpark
DEMO

24. InSpark
Innovate
to
accelerate

25. InSpark
ededeuwdvlmsynoxgn01 ededeuwprodsynoxgn01

26. InSpark
ededeuwdvlmvaultoxgn01
Create Key Vault
Azure Key Vault
$SubscriptionName = “XXXXXXXXXXXXXXXXX"
$ResourceGroupName = "EDED-EUW-DEVELOPMENT-OXYGEN-RG"
$KeyvaultName="ededeuwdvlmvaultoxgn01"
$Location="West Europe"
Connect-AzAccount
Select-AzSubscription -SubscriptionName $SubscriptionName
New-AzKeyVault -Name $KeyvaultName -ResourceGroupName $ResourceGroupName -location $Location

27. InSpark
Grant the Azure Synapse access to your Azure KeyVault.
Grant Access
Azure Key Vault

28. InSpark
Create a new Linked Service
• Authentication Method
• Subscription
• Key Vault Name
Be aware that you need to publish the Azure Key Vault before you start using secrets !
Connecting in Azure Synapse
Azure Key Vault

29. InSpark
ABLB-ENV01
ASQL-ENV01-WWI
The use of naming conventions ensures that we can
configure everything more dynamically in the release process later on !
Creating secrets
Azure Key Vault
ENVIRONMENT-CONNECTION-REFERENCE
Environment
Development DVLM
Test TEST
Acceptance ACPT
Production PROD
Linked Service
Azure Blob Storage ABLB
Azure Cosmos DB SQL API ACSA
Azure Cosmos DB MongDB API ACMA
Azure Data Explorer ADEX
Azure Data Lake Storage Gen1 ADLS
Azure Data Lake Storage Gen2 ADLS
Azure Database for MariaDB AMDB
Azure Database for MySQL AMYS
Azure Database for PostgreSQL APOS
Azure File Storage AFIL
Azure Search ASER
Azure SQL Database ASQL
Azure SQL Database Managed Instance ASQM
Azure SQL Data Warehouse ASDW
Azure Table Storage ATBL
SQL Server MSQL
Reference
DatabaseConnection LogicalServer(Short)-DatabaseName
Storage
StorageName like
Audit/SSIS/Staging/DataLake

30. InSpark

31. InSpark
DEMO

32. InSpark
Innovate
to
accelerate

33. InSpark
ededeuwdvlmsynoxgn01 ededeuwprodsynoxgn01

34. InSpark
Configure the correct values
Create library
Azure DevOps

35. InSpark
Add Synapse Workspace Deployment Extension
Extension
Azure DevOps

36. InSpark
Create release in Azure DevOps

37. InSpark
Create a Release based on an Empty Job
Create release
Azure DevOps

38. InSpark
Create a Release based on an Empty Job
Create a Stage for each Environment
Create release
Azure DevOps
Test
Acceptance
Production

39. InSpark
Create a Release based on an Empty Job
Create a Stage for each Environment
Add an Artifact
Create release
Azure DevOps

40. InSpark
Create a Release based on an Empty Job
Create a Stage for each Environment
Add an Artifact
Enable Continuous Deployment
Create release
Azure DevOps

41. InSpark
Create a Release based on an Empty Job
Create a Stage for each Environment
Add an Artifact
Enable Continuous Deployment
Create release
Azure DevOps

42. InSpark
Create task
Azure DevOps
Define an Agent

43. InSpark
Define an Agent
Add the task
Create task
Azure DevOps

44. InSpark
Define an Agent
Add the task
Configure the task
Create task
Azure DevOps

45. InSpark
Define an Agent
Add the task
Configure the task
Override the Template Parameters
Create task
Azure DevOps

46. InSpark
Connecting Library
Release name format
Release
Azure DevOps

47. InSpark

48. InSpark

49. InSpark

50. InSpark

51. InSpark
DEMO

52. InSpark
Innovate
to
accelerate

53. InSpark
ededeuwdvlmsynoxgn01 ededeuwprodsynoxgn01

54. InSpark
Not all the properties have parameters by default.
template-parameters-definition.json
Adding accountkey for DataLake
with secrets from Azure Key Vault “accountkey": “|",
Custom parameters
Azure Synapse Analytics
More info can be found here:
https://docs.microsoft.com/en-us/azure/synapse-analytics/cicd/continuous-integration-delivery#create-
custom-parameters-in-the-workspace-template
= means keep the current value as the default value for the parameter.
- means don't keep the default value for the parameter.
| is a special case for secrets from Azure Key Vault for connection strings or keys.

55. InSpark
You cannot publish while SSIS Integration Runtime is running
Triggers needs to be disabled and enabled afterwards (Synapse Extension)
Every time you change something to your connection you need to update the Template parameter
Deletes are handled manually or through (Synapse Extension)
Remarks
Azure Synapse Analytics

56. InSpark

57. InSpark
DEMO

58. InSpark
Innovate
to
accelerate

59. InSpark
Attollo B3 Consulting Group Quest Software SQL Service
Transmokopter SQL
dbWatch
Catman Solution DB24 Redgate Software

60. InSpark
@erwindekreuk https://erwindekreuk.com

61. InSpark