Ansible, jenkins_script module and problem with args - jenkins

I am using this module: https://docs.ansible.com/ansible/latest/collections/community/general/jenkins_script_module.html
And I want to pass arguments(dictionary: jenkins_username_password_credentials) to my Groovy script that can change credentials on Jenkins:
#! / usr / bin / env groovy
import jenkins.model.Jenkins
import com.cloudbees.plugins.credentials.domains.Domain
import org.jenkinsci.plugins.plaincredentials.impl.StringCredentialsImpl
import com.cloudbees.plugins.credentials.CredentialsScope
import hudson.util.Secret
import com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey
import com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl
instance = Jenkins.instance
domain = Domain.global ()
store = instance.getExtensionList (
"com.cloudbees.plugins.credentials.SystemCredentialsProvider") [0] .getStore ()
{% for credential in jenkins_username_password_credentials%}
usernameAndPassword = new UsernamePasswordCredentialsImpl (
CredentialsScope.GLOBAL,
"{{credential ['name']}}",
"{{credential ['description']}}",
"{{credential ['username']}}",
"{{credential ['password']}}"
)
store.addCredentials (domain, usernameAndPassword)
{% endfor%}
I do it through the given code:
- name: Upload Nexus credentials
jenkins_script:
url: "{{jenkins_url}}"
script: '{{lookup ("template", "templates / jenkins_groovy / credentials / secret_password.groovy")}}'
args:
jenkins_username_password_credentials:
- name: Nexus
description: Credentials for Nexus Server
username: Nexus
password: 25447
user: '{{jenkins_username}}'
password: '{{jenkins_password}}'
validate_certs: false
timeout: "120"
But I have an error that my variable jenkins_username_password_credentials is undefined until I add it to the defaults file in my role.
fatal: [jenkins_linux]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'jenkins_username_password_credentials' is undefined \ n \ nThe error appears to be in' / home / tbaburin / Ansible / roles / jenkins / tasks / main.yml ': line 3, column 3, but may \ nbe elsewhere in the file depending on the exact syntax problem. \ n \ nThe offending line appears to be: \ n \ n \ n- name: Upload Nexus credentials \ n ^ here \ n "}
My questions are:
Can I declare a dictionary directly in the "args" field of the jenkins_script module like in the example? https://docs.ansible.com/ansible/latest/collections/community/general/jenkins_script_module.html
If I can't, how can I change the dictionary in the "args" field so that the modified dictionary with the data I need is loaded into the script?
I just don't want to always go into the default file and change the dictionary data there to load different credentials and I want just to pass some j2 variables into args just to loop this, if I want to upload a lot of credentials.

Related

Cannot set environment variables for npm-config in PowerShell

When I am trying to set environment variables using PowerShell in Windows Terminal with the command set test1=value1, I get no errors. However, when I try to check all environment variables using the set command, I get the following prompt:
cmdlet Set-Variable at command pipeline position 1
Supply values for the following parameters:
Name[0]:
I read that when using PowerShell you set environment vars using this:
$Env:test1 = "value1";
I want to set the variables so that on my backend in custom-environment-variables.json
I can store a name by which config can extract it using config.get("test").
custom-environment-variables.json:
{
"test": "test1",
}
But every time I try this, it says Error: Configuration property "test" is not defined.
Doing the same procedure CMD (either directly or through Windows Terminal) I get no issues whatsoever. Any ideas what might be causing this?
First, the easy part:
I get no errors but when I try to check all env. variables calling "set" I get the following prompt:
That's because the set command in PowerShell behaves differently. It's an alias for the PowerShell Set-Variable cmdlet. You can see this with Get-Alias.
Also, PowerShell variables are not environment variables. As you commented, the proper way to set an environment variable in PowerShell is with:
$env:variablename = "value"
The equivalent command to set (to get a list of all environment variables and their values) in PowerShell is:
Get-ChildItem env:
# Or using the alias
dir env:
# Or using another alias
ls env:
This access the PowerShell "environment provider", which is essentially (my grossly oversimplified summary) a "virtual drive/filesystem" that PowerShell provides which contains the environment variables. You can also create variables in here.
More reading: about_Environment_Variables from the PowerShell Doc.
As for the core issue with the config module, I haven't been able to reproduce that. It works correctly for me in both PowerShell and CMD. So let me run through my results in the hopes that it will help you see what might be different. All tests were performed in Windows Terminal, although as we've determined in the comments, this is more a difference in PowerShell vs. CMD for you:
config\default.json:
{
"test": "Original Value"
}
config\custom-environment-variables.json:
{
"test": "test1"
}
CMD without test1 variable set:
Running node in CMD:
> const config = require('config')
undefined
> config.get('test')
'Original Value'
>
CMD with test1 variable set:
Exit Node, and back in CMD:
>set test1=Override
>node
In Node:
Welcome to Node.js v14.16.1.
Type ".help" for more information.
> const config = require('config')
undefined
> config.get('test')
'Override'
>
PowerShell without test1 variable set:
Welcome to Node.js v14.16.1.
Type ".help" for more information.
> const config = require('config')
undefined
> config.get('test')
'Original Value'
>
PowerShell with test1 variable set:
In PowerShell:
PS1> $env:test1="Override"
PS1> node
In Node:
Welcome to Node.js v14.16.1.
Type ".help" for more information.
> const config = require('config')
undefined
> config.get('test')
'Override'
>

Retrieving RSA key from AWS Secrets Manager in CodeBuild corrupts key "invalid format"

During a CodeBuild run I am retrieving a rsa key from SecretsManager, which is the private key to use to access private sources in BitBucket. To do this I have copied the private key into a secret, then in my buildspec file I have the following snippet:
"env": {
"secrets-manager": {
"LOCAL_RSA_VAR": "name-of-secret"
}
},
In the install portion of the buildspec:
"install": {
"commands": [
"echo $LOCAL_RSA_VAR" > ~/.ssh/id_rsa,
"chmod 600 ~/.ssh/id_rsa",
"yarn install"
]
},
HOWEVER, this always ends up with an error:
Load key "/root/.ssh/id_rsa": invalid format
git#bitbucket.org: Permission denied (publickey).
fatal: Could not read from remote repository.
To determine if the key was wrong I tried uploading the rsa_id file into S3 and then download it from there and used it that way using these commands instead:
"install": {
"commands": [
"aws s3 cp s3://the-bucket-name/id_rsa ~/.ssh/id_rsa",
"chmod 600 ~/.ssh/id_rsa",
"yarn install"
]
},
This works fine.
So I guess the question is... Has anyone tried this and had better success? Is there something that I am not doing correctly that you can think of?
I have encountered the same issue.
Copying the id_rsa generated from the the command echo $LOCAL_RSA_VAR > ~/.ssh/id_rsa in S3 I have noticed that the new lines have not been preseved.
I have resolved putting the var env between double quote "":
echo "$LOCAL_RSA_VAR" > ~/.ssh/id_rsa
I was able to get an answer by diff'ing the output of the Env Var vs the File contents from the S3 file. ('cat' will not print out the content of a secret mgr env variable) It ends up content of the env var was altered by the 'echo' command.
The solution that ended up working for me was:
printenv LOCAL_RSA_VAR > ~/.ssh/id_rsa
this command didn't alter the content of the rsa and I was able to successfully use the certificate.
As a recap this is what I was successful doing:
Generate the new key
Used command "pbcopy < id_rsa" to get local key into clipboard
Pasted that into a new secret in Secret Manager
Used the first set of code above to have the buildspec file retrieve the content into a env variable and then the 'printenv' command above, in the install command portion of the buildspec file, to save that to the default ssh location.
Hope this helps anyone that runs into the same issue.
UPDATE: I found that this works if the RSA is stored as its own secret as one big block of text. If you try and add this as part of a json object, ie:
{
"some": "thing",
"rsa_id": "<the rsa key here>"
}
this does not seem to work. I found that the content is altered with spaces in place of the newline. This is what i found when running an 'od -ax' on each and comparing them:
own secret:
R I V A T E sp K E Y - - - - - nl
json secret:
R I V A T E sp K E Y - - - - - sp
I has the same issue, fixed it my NOT Copy-Paste my private key to SecretManager, but use AWS CLI to upload my private key to SecretManager:
aws secretsmanager put-secret-value --secret-id AWS_CODECOMMIT_SSH_PRIVATE --secret-string file://myprivatekey.pem
And then CloudBuild worked fine:
version: 0.2
env:
secrets-manager:
AWS_CODECOMMIT_SSH_ID : AWS_CODECOMMIT_SSH_ID
AWS_CODECOMMIT_SSH_PRIVATE: AWS_CODECOMMIT_SSH_PRIVATE
phases:
install:
commands:
- echo "Setup CodeCommit SSH Key"
- mkdir ~/.ssh/
- echo "$AWS_CODECOMMIT_SSH_PRIVATE" > ~/.ssh/id_rsa
- echo "Host git-codecommit.*.amazonaws.com" > ~/.ssh/config
- echo " User $AWS_CODECOMMIT_SSH_ID" >> ~/.ssh/config
- echo " IdentityFile ~/.ssh/id_rsa" >> ~/.ssh/config
- echo " StrictHostKeyChecking no" >> ~/.ssh/config
- chmod 600 ~/.ssh/id_rsa
- chmod 600 ~/.ssh/config

Error in adding 4th organization in to Hyperledger Fabric 2.0

I am new to Fabric 2.0 and recently installed all samples and I was able to run test-network without an issue with 2 orgs. Then I followed the directory on addOrg3 to add 3rd organization and join the channel I created earlier.
Now the fun part came in when I wanted to add 4th organization. What I did was, I copied the addOrg3 folder and renamed almost everything in each file to represent 4th organization. I even assigned new PORT for this organization. However I am seeing the following error.
I've also added the following in Scripts/envVar.sh
export PEER0_ORG4_CA=${PWD}/organizations/peerOrganizations/org4.example.com/peers/peer0.org4.example.com/tls/ca.crt
And added the following in envVarCLI.sh
elif [ $ORG -eq 4 ]; then
CORE_PEER_LOCALMSPID="Org4MSP"
CORE_PEER_TLS_ROOTCERT_FILE=$PEER0_ORG4_CA
CORE_PEER_ADDRESS=peer0.org4.example.com:12051
CORE_PEER_MSPCONFIGPATH=/opt/gopath/src/github.com/hyperledger/fabric/peer/organizations/peerOrganizations/org4.example.com/users/Admin#.../msp
I have also added step1Org4.sh and step2Org4.sh basically following by following addOrg3's structure.
What steps do you follow to add additional organizations ? Please help.
"No such container: Org4cli"
Sorry for the formatting since I wasn't able to put in to coding style but here is the output from running the command "./addOrg4.sh up"
**Add Org4 to channel 'mychannel' with '10' seconds and CLI delay of '3' seconds and using database 'leveldb'
Desktop/blockchain/BSI/fabric-samples/test-network/addOrg4/../../bin/cryptogen
##########################################################
##### Generate certificates using cryptogen tool #########
##########################################################
##########################################################
############ Create Org4 Identities ######################
##########################################################
+ cryptogen generate --config=org4-crypto.yaml --output=../organizations
org4.example.com
+ res=0
+ set +x
Generate CCP files for Org4
Desktop/blockchain/BSI/fabric-samples/test-network/addOrg4/../../bin/configtxgen
##########################################################
####### Generating Org4 organization definition #########
##########################################################
+ configtxgen -printOrg Org4MSP
2020-05-29 13:33:04.609 EDT [common.tools.configtxgen] main -> INFO 001 Loading configuration
2020-05-29 13:33:04.617 EDT [common.tools.configtxgen.localconfig] LoadTopLevel -> INFO 002 Loaded configuration: /Desktop/blockchain/BSI/fabric-samples/test-network/addOrg4/configtx.yaml
+ res=0
+ set +x
###############################################################
####### Generate and submit config tx to add Org4 #############
###############################################################
Error: No such container: Org4cli
ERROR !!!! Unable to create config tx **
In your addOrg4.sh have condition check like this:
CONTAINER_IDS=$(docker ps -a | awk '($2 ~ /fabric-tools/) {print $1}')
if [ -z "$CONTAINER_IDS" -o "$CONTAINER_IDS" == " " ]; then
echo "Bringing up network"
Org4Up
fi
If you already run addOrg3.sh up, CONTAINER_IDS alway have value (Example: 51b4ad60d812). It is ContainerID of Org3cli. So function Org4Up will never call. Simple way is just comment code like this:
# CONTAINER_IDS=$(docker ps -a | awk '($2 ~ /fabric-tools/) {print $1}')
# if [ -z "$CONTAINER_IDS" -o "$CONTAINER_IDS" == " " ]; then
echo "Bringing up network"
Org4Up
# fi
It will bring up Org4cli you missing.
First check the container is up or not and if it is up then I think the CLI where the command is executed is not bootstrapped with the Org4 details.
I have added the 4th Organization from the three Org Hyperledger Fabric Network .Firstly, you have to create the Org4-artifacts (Crypto.yaml and Org4 docker file including the Org4Cli) and then try to follow the manual (step by step) process to add the new Organization from the official documentation.
https://hyperledger-fabric.readthedocs.io/en/release-2.0/channel_update_tutorial.html
Omit the process of editing scripts (step1 Org3.sh ...) because the workflow for adding the 4th or a new Org is slightly changed.So,you will spend a lot of time in just modifying the scripts.
I will write an article to add a new Org (4th) on medium,will paste the link here too.

How to pass a variable into the 'az pipelines variable-group variable create' command

I am trying to use the "az pipelines variable-group variable create ..." command to create a variable which references a different variable. e.g.
az pipelines variable-group variable create --project MyTestProject --
group-id 15 --name ‘ApplicationName’ --value 'TestApp-$(env)'
where the variable “env” is defined in a variable-group within the same project library.
When I run the command above it gives the error:
“Failed to load python executable” “exit /b 1”.
Despite an error being generated the variable is created; in the example above the variable 'ApplicationName' has the value 'TestApp-$(env', the trailing bracket character, ")", is missing and seems to be causing the problem.
The dollar sign, "$", and/or the opening bracket character "(" don't generate any error when used without the closing bracket ")".
I have tried escaping the closing bracket character with backslashes "\" and caret "`" characters but couldn't find any combination that would create the desired variable value, "TestApp-$(env)".
Could someone tell me how to escape the closing bracket so the variable is correctly created within the variable group.
I am running the following versions of az:
azure-cli 2.0.73
command-modules-nspkg 2.0.3
core 2.0.73
nspkg 3.0.4
telemetry 1.0.3
Extensions:
azure-devops 0.12.0
Python (Windows) 3.6.6
Many Thanks,
Gary
I have finally managed to figure out how to escape a variable whose name itself contains a different variable name. By calling the az cli command and wrapping the variable value in double quotes and a single quote, the variable is correctly created in DevOps:
pipelines variable-group variable create --project MyTestProject -- group-id 15 --name ApplicationName' --value '"TestApp-$(env)"'
DevOps-LibraryVariable-screenshot
As a further expansion on this topic I had a need to pass the value as a variable. My source was in a key value pair. In this case I used the following.
$key = $var.Key
$value = '"{0}"' -f $var.Value
az pipelines variable-group variable update --group-id $groupId --org $org --project $project `
--name $key --value $value
It depends on your OS and tools.
For example, in Windows OS, you can get environment variable with %variable_name%. So, the following would be right:
az pipelines variable-group variable create --project keyvault --group-id 1 --name "ApplicationName" --value "TestApp-%java_home%"
However, in PowerShell, you can get environment with "$env:variable_name". So, the following would be right:
az pipelines variable-group variable create --project keyvault --group-id 1 --name "ApplicationName2" --value "TestApp-$($env:java_home)"
Update:
So, in Azure Pipeline, you can use group variables as following:
pool:
name: Hosted VS2017
demands:
- msbuild
- visualstudio
- vstest
variables:
- group: vargroup
steps:
- task: AzureCLI#1
inputs:
azureSubscription: 'CSP Azure (e5b0fcfa-e859-43f3-8d84-5e5fe29f4c68)'
scriptLocation: 'inlineScript'
inlineScript: |
echo the variable var1:%var1%
I have a variable group:
And you can see that: echo the variable var1:%var1% will be echo the variable var1:value1

How to get Task ID from within ECS container?

Hello I am interested in retrieving the Task ID from within inside a running container which lives inside of a EC2 host machine.
AWS ECS documentation states there is an environment variable ECS_CONTAINER_METADATA_FILE with the location of this data but will only be set/available if ECS_ENABLE_CONTAINER_METADATA variable is set to true upon cluster/EC2 instance creation. I don't see where this can be done in the aws console.
Also, the docs state that this can be done by setting this to true inside the host machine but would require to restart the docker agent.
Is there any other way to do this without having to go inside the EC2 to set this and restart the docker agent?
This doesn't work for newer Amazon ECS container versions anymore, and in fact it's now much simpler and also enabled by default. Please refer to this docu, but here's a TL;DR:
If you're using Amazon ECS container agent version 1.39.0 and higher, you can just do this inside the docker container:
curl -s "$ECS_CONTAINER_METADATA_URI_V4/task" \
| jq -r ".TaskARN" \
| cut -d "/" -f 3
Here's a list of container agent releases, but if you're using :latest – you're definitely fine.
The technique I'd use is to set the environment variable in the container definition.
If you're managing your tasks via Cloudformation, the relevant yaml looks like so:
Taskdef:
Type: AWS::ECS::TaskDefinition
Properties:
...
ContainerDefinitions:
- Name: some-name
...
Environment:
- Name: AWS_DEFAULT_REGION
Value: !Ref AWS::Region
- Name: ECS_ENABLE_CONTAINER_METADATA
Value: 'true'
This technique helps you keep everything straightforward and reproducible.
If you need metadata programmatically and don't have access to the metadata file, you can query the agent's metadata endpoint:
curl http://localhost:51678/v1/metadata
Note that if you're getting this information as a running task, you may not be able to connect to the loopback device, but you can connect to the EC2 instance's own IP address.
We set it with the so called user data, which are executed at the start of the machine. There are multiple ways to set it, for example: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html#user-data-console
It could look like this:
#!/bin/bash
cat <<'EOF' >> /etc/ecs/ecs.config
ECS_CLUSTER=ecs-staging
ECS_ENABLE_CONTAINER_METADATA=true
EOF
Important: Adjust the ECS_CLUSTER above to match your cluster name, otherwise the instance will not connect to that cluster.
Previous answers are correct, here is another way of doing this:
From the ec2 instance where container is running, run this command
curl http://localhost:51678/v1/tasks | python -mjson.tool |less
From the AWS ECS cli Documentation
Command:
aws ecs list-tasks --cluster default
Output:
{
"taskArns": [
"arn:aws:ecs:us-east-1:<aws_account_id>:task/0cc43cdb-3bee-4407-9c26-c0e6ea5bee84",
"arn:aws:ecs:us-east-1:<aws_account_id>:task/6b809ef6-c67e-4467-921f-ee261c15a0a1"
]
}
To list the tasks on a particular container instance
This example command lists the tasks of a specified container instance, using the container instance UUID as a filter.
Command:
aws ecs list-tasks --cluster default --container-instance f6bbb147-5370-4ace-8c73-c7181ded911f
Output:
{
"taskArns": [
"arn:aws:ecs:us-east-1:<aws_account_id>:task/0cc43cdb-3bee-4407-9c26-c0e6ea5bee84"
]
}
My ECS solution as bash and Python snippets. Logging calls are able to print for debug by piping to sys.stderr while print() is used to pass the value back to a shell script
#!/bin/bash
TASK_ID=$(python3.8 get_ecs_task_id.py)
echo "TASK_ID: ${TASK_ID}"
Python script - get_ecs_task_id.py
import json
import logging
import os
import sys
import requests
# logging configuration
# file_handler = logging.FileHandler(filename='tmp.log')
# redirecting to stderr so I can pass back extracted task id in STDOUT
stdout_handler = logging.StreamHandler(stream=sys.stderr)
# handlers = [file_handler, stdout_handler]
handlers = [stdout_handler]
logging.basicConfig(
level=logging.INFO,
format="[%(asctime)s] {%(filename)s:%(lineno)d} %(levelname)s - %(message)s",
handlers=handlers,
datefmt="%Y-%m-%d %H:%M:%S",
)
logger = logging.getLogger(__name__)
def get_ecs_task_id(host):
path = "/task"
url = host + path
headers = {"Content-Type": "application/json"}
r = requests.get(url, headers=headers)
logger.debug(f"r: {r}")
d_r = json.loads(r.text)
logger.debug(d_r)
ecs_task_arn = d_r["TaskARN"]
ecs_task_id = ecs_task_arn.split("/")[2]
return ecs_task_id
def main():
logger.debug("Extracting task ID from $ECS_CONTAINER_METADATA_URI_V4")
logger.debug("Inside get_ecs_task_id.py, redirecting logs to stderr")
logger.debug("so that I can pass the task id back in STDOUT")
host = os.environ["ECS_CONTAINER_METADATA_URI_V4"]
ecs_task_id = get_ecs_task_id(host)
# This print statement passes the string back to the bash wrapper, don't remove
logger.debug(ecs_task_id)
print(ecs_task_id)
if __name__ == "__main__":
main()

Resources