I have a data factory pipeline which takes the following parameters
Param1
Database Server 1 Name
Database Server 2 Name
Database Server 1 Username
Database Server 2 Username
etc
My pipeline decides via some logic which database server to do an import from.
Essentially I want to deploy 2 versions of my pipeline. 1 Runs in dev and the other in prod.
I want to release a dev and prod version of my pipeline via Azure Devops. Each environment release should provide (via key vault) the values of:
Database Server 1 Name
Database Server 2 Name
Database Server 1 Username
Database Server 2 Username
First prize would be if those values did not even show up any more as parameters in the pipeline. So that triggers would just have to provide Param1. In addition if you manually run the pipeline I also just want to provide Param1.
EDIT: Note that I use the parameters eventually in a paramaterized linked service if that makes a difference (https://learn.microsoft.com/en-us/azure/data-factory/parameterize-linked-services).
I think the key idea to resolve your problem is to use two separate instances of data factory.
In the DEV enironment you have your parameterized connection as you stated above. When taking the code to PROD, you export the template and import it again into the other instance. There you have an additional config file that fills up the values needed to set up the connection properly.
If you want to avoid having the credentials stored in the config file then just add an azure key vault linked service and parameterize the secret identifier accordingly. When you import the template into PROD you even do not need to provide any parameter but the identifier for which secret to grab from key vault.
See here for more info:
devops integration
key vault integration
Related
we are facing a challenge as to how to handle the User-Assigned-Managed-Identity with the CICD pipelines in Azure DevOps when the UAMI are not the same across environments?
Dev, Test and Prod have different subscriptions therefore each synapse implementation can have its own Azure subscription and possibly different UAMI configured.
When we publish the templates from synapse it generates a separate credential.json file and it is not part of the parameterization template.
The question is how can we override the parameters when there is a separate credential file.
We tried to update the linked service with resourceid which contains UAMI details so that it becomes part of the parametertemplate file but it is not working.
In my ADO build pipline, I have a secure file download step. When we branch versions, we use powershell to do the heavy lifting with cloning build definitions and updating settings/info in the cloned pipeline.
One issue I've run into is that the Secure File Download step doesn't accept variables, and in the UI you can only select names of files that already exist, so we've had to manually update it after every new branch we create.
I've grabbed the definition task step in powershell (as $step) and was hoping I could set the $step.inputs.fileInputs to a variable I assign to something like cert-$newVersion, however it currently is set to a guid.
Does anyone know if it possible to get the guid of secure files in ADO via the API or have a solution?
Does anyone know if it possible to get the guid of secure files in ADO via the API or have a solution?
Yes. This API exists.
You could try to use the following Rest API:
Get https://dev.azure.com/{OrganizationName}/{ProjectName}/_apis/distributedtask/securefiles?api-version=6.1-preview.1
Result:
You could get the secure file GUID based on the file name.
I have a problem. I use Jenkins to deploy application on machines. Depending on which machine i want to deploy new version I need to use different database passwords to run db migrations. I want to store credentials in Jenkins and for each deploy job select credentials set from dropdown which will be passed (just password ) to powershell script. I have Credentials Binding Plugin and Extended Choice Parameter Plugin. I thought it might help me with my problem, but I cannot find solution for this. Do you have any ideas how to achieve this?
As I understand, you need only to do two things:
add Credential Parameter in This project is parameterized section for possibility to select credentials set from dropdown.
enable Use secret text(s) or file(s) option in Build Environment section. This will allow you to take credentials of various sorts and use them from shell build steps and the like. Each binding will define an environment variable.
If you have already created Jenkins credentials like these:
then you will be available to run your job using Build with Parameters button and select needed credentials:
So, after configuring, you don't need to dig in job configuration each time, all possible credentials will be automatically loaded, you need only select the needed one when run a job.
In TFS 2015 we used Machine groups and weren't required to enter a Admin Login and Password for the "Powershell on Target Machines" task.
We are now using tfs 2017 and do not want to define the Admin Login and Password in every build definition since we have numerous. How can you overcome this? I was thinking about using a txt file with a SecureString password and in the build definition read it in, decrypt it and assign it to the Build definition variable
Use variable groups. Create a variable group for your shared secrets, then link the variable group to any build or release definitions that need access to the secrets. Going a step further, you could store the secrets in Azure KeyVault to provide a single source of truth for secrets.
Storing secrets in source control using reversible encryption is just obfuscation, and from a security standpoint is only slightly better than storing it in plaintext.
I use Microsoft Team Foundation Server (TFS) for most of my software deployments. TFS allows me to dynamically replace text within specific configuration files during the release process to specific environments (dev, test, prod).
The text it replaces are placeholders called "tokens". For instance, during my automated deployments, TFS will allow us to replace tokens found within configuration files with pre-defined values saved in the build administration for each environment. This way, I don't store any real credentials in source control for any environment. I also don't store any script in source that would hold these sensative credentials. The credentials are dynamically inserted over top the tokens during the release, and the credentials are hosted/saved/configured inside of the release system (not in a script).
For example, I have a configuration file (web.config) that has tokens. A token looks something like this:
MySettingName=${MYSETTINGVALUE}
During the release to DEV, I want the text ${MYSETTINGVALUE} replaced with the word TEN. During the release to PROD, I want that same ${MYSETTINGVALUE} text replaced with the word ORANGE. And I want to store those two values (TEN and ORANGE) in the release administration system, and not in a script.
How do I configure Jenkins to do this same thing?
I have searched up-and-down for this specific answer. While many blogs, articles, documentation exist, none of them speak directly to this issue.
I would prefer NOT to use some additional 3rd party software to do
this.
I would prefer NOT to kick off some manual build and supply these
values each and every time.
I would also prefer NOT to use an Operating System level system
variable (aka evironment variable). In case that server dies, I
would rather not have to remember to setup those OS environment
varialbles on the next server.
Jenkins has a built-in credentials plugin for handling secrets in builds.
See this article on how to use them: https://support.cloudbees.com/hc/en-us/articles/203802500-Injecting-Secrets-into-Jenkins-Build-Jobs
Basically it stores credentials securely and injects them into your jobs as variables which can then be used like any other.