I am able to create a new node via the Jenkins web GUI and then have the node running in a container connect back to the Jenkins master via the name and -secret value
ex.
docker run jenkinsci/jnlp-slave -url http://jenkins-server:port <secret> <slave name>
Is there a way to programmatically create a Jenkins node and get the secret and slave name so I don't have to do it via the GUI?
Creating an agent programmatically
You can use the create-node CLI command to create new agents with a given configuration.
For example, given this minimal JNLP agent configuration in a file config.xml:
<slave>
<remoteFS>/opt/jenkins</remoteFS>
<numExecutors>2</numExecutors>
<launcher class="hudson.slaves.JNLPLauncher" />
</slave>
you can run the create-node command via the CLI client, or the SSH interface:
cat config.xml | java -jar jenkins-cli.jar -s https://jenkins/ create-node my-agent
Viewing agent configuration
To see what the XML configuration looks like for an existing agent, you can append config.xml to an agent URL, e.g. https://jenkins/computer/some-agent-name/config.xml, or you can use the get-node CLI command.
Fetching the per-agent secret programmatically
To fetch the secret hex value without using the Jenkins web UI, you can run a script via the groovy CLI command:
echo 'println jenkins.model.Jenkins.instance.nodesObject.getNode("my-agent")?.computer?.jnlpMac' \
| java -jar ~/Downloads/jenkins-cli.jar -s https://jenkins/ groovy =
This will return the secret value directly. Note that in order to use the groovy command via the SSH interface, you need Jenkins 2.46 or newer. In earlier versions, it only works via the CLI client.
You can also create an agent using the REST API. This is especially useful when having an apache proxy in front (see issue JENKINS47279) and no direct access to the jenkins otherwise (e.g. in a corporate network) where CLI will not work.
I recommend to create an API token for this purpose. Then you can do something like this
Linux (Bash)
export JENKINS_URL=https://jenkins.intra
export JENKINS_USER=papanito
export JENKINS_API_TOKEN=xxxxxxxxxxxxxxxxxxxxxxxx
export NODE_NAME=testnode
export JSON_OBJECT="{ 'name':+'${NODE_NAME}',+'nodeDescription':+'Linux+slave',+'numExecutors':+'5',+'remoteFS':+'/home/jenkins/agent',+'labelString':+'SLAVE-DOCKER+linux',+'mode':+'EXCLUSIVE',+'':+['hudson.slaves.JNLPLauncher',+'hudson.slaves.RetentionStrategy\$Always'],+'launcher':+{'stapler-class':+'hudson.slaves.JNLPLauncher',+'\$class':+'hudson.slaves.JNLPLauncher',+'workDirSettings':+{'disabled':+true,+'workDirPath':+'',+'internalDir':+'remoting',+'failIfWorkDirIsMissing':+false},+'tunnel':+'',+'vmargs':+'-Xmx1024m'},+'retentionStrategy':+{'stapler-class':+'hudson.slaves.RetentionStrategy\$Always',+'\$class':+'hudson.slaves.RetentionStrategy\$Always'},+'nodeProperties':+{'stapler-class-bag':+'true',+'hudson-slaves-EnvironmentVariablesNodeProperty':+{'env':+[{'key':+'JAVA_HOME',+'value':+'/docker-java-home'},+{'key':+'JENKINS_HOME',+'value':+'/home/jenkins'}]},+'hudson-tools-ToolLocationNodeProperty':+{'locations':+[{'key':+'hudson.plugins.git.GitTool\$DescriptorImpl#Default',+'home':+'/usr/bin/git'},+{'key':+'hudson.model.JDK\$DescriptorImpl#JAVA-8',+'home':+'/usr/bin/java'},+{'key':+'hudson.tasks.Maven\$MavenInstallation\$DescriptorImpl#MAVEN-3.5.2',+'home':+'/usr/bin/mvn'}]}}}"
curl -L -s -o /dev/null -v -k -w "%{http_code}" -u "${JENKINS_USER}:${JENKINS_API_TOKEN}" -H "Content-Type:application/x-www-form-urlencoded" -X POST -d "json=${JSON_OBJECT}" "${JENKINS_URL}/computer/doCreateItem?name=${NODE_NAME}&type=hudson.slaves.DumbSlave"
In order to get the agent secret via REST API checkout this, which would look something like this:
curl -L -s -u ${JENKINS_USER}:${JENKINS_API_TOKEN} -X GET ${JENKINS_URL}/computer/${NODE_NAME}/slave-agent.jnlp | sed "s/.*<application-desc main-class=\"hudson.remoting.jnlp.Main\"><argument>\([a-z0-9]*\).*/\1/"
Windows (PS)
And here my solution for Windows using Powershell:
$JENKINS_URL="https://jenkins.intra"
$JENKINS_USER="papanito"
$JENKINS_API_TOKEN="xxxxxxxxxxxxxxxxxxxxxxxx"
$NODE_NAME="testnode-ps"
# https://stackoverflow.com/questions/27951561/use-invoke-webrequest-with-a-username-and-password-for-basic-authentication-on-t
$bytes = [System.Text.Encoding]::ASCII.GetBytes("${JENKINS_USER}:${JENKINS_API_TOKEN}")
$base64 = [System.Convert]::ToBase64String($bytes)
$basicAuthValue = "Basic $base64"
$headers = #{ Authorization = $basicAuthValue; }
$hash=#{
name="${NODE_NAME}";
nodeDescription="Linux slave";
numExecutors="5";
remoteFS="/home/jenkins/agent";
labelString="SLAVE-DOCKER linux";
mode="EXCLUSIVE";
""=#(
"hudson.slaves.JNLPLauncher";
'hudson.slaves.RetentionStrategy$Always'
);
launcher=#{
"stapler-class"="hudson.slaves.JNLPLauncher";
"\$class"="hudson.slaves.JNLPLauncher";
"workDirSettings"=#{
"disabled"="true";
"workDirPath"="";
"internalDir"="remoting";
"failIfWorkDirIsMissing"="false"
};
"tunnel"="";
"vmargs"="-Xmx1024m"
};
"retentionStrategy"=#{
"stapler-class"= 'hudson.slaves.RetentionStrategy$Always';
'$class'= 'hudson.slaves.RetentionStrategy$Always'
};
"nodeProperties"=#{
"stapler-class-bag"= "true";
"hudson-slaves-EnvironmentVariablesNodeProperty"=#{
"env"=#(
#{
"key"="JAVA_HOME";
"value"="/docker-java-home"
};
#{
"key"="JENKINS_HOME";
"value"="/home/jenkins"
}
)
};
"hudson-tools-ToolLocationNodeProperty"=#{
"locations"=#(
#{
"key"= 'hudson.plugins.git.GitTool$DescriptorImpl#Default';
"home"= "/usr/bin/git"
};
#{
"key"= 'hudson.model.JDK\$DescriptorImpl#JAVA-8';
"home"= "/usr/bin/java"
};
#{
"key"= 'hudson.tasks.Maven$MavenInstallation$DescriptorImpl#MAVEN-3.5.2';
"home"= "/usr/bin/mvn"
}
)
}
}
}
#https://stackoverflow.com/questions/17929494/powershell-convertto-json-with-embedded-hashtable
$JSON_OBJECT = $hash | convertto-json -Depth 5
$JSON_OBJECT
Invoke-WebRequest -Headers $headers -ContentType "application/x-www-form-urlencoded" -Method POST -Body "json=${JSON_OBJECT}" -Uri "${JENKINS_URL}/computer/doCreateItem?name=${NODE_NAME}&type=hudson.slaves.DumbSlave"
Just chiming in a bit late to the party here, but I would highly recommend looking at the Jenkins Client plugin instead. Once the plugin is installed, you need only to start the client JAR from the build node and give it the IP address of the master.
As far as the master goes, you don't need to bother configuring anything. Nodes that register with the master are available automatically to start executing jobs. This is much easier than any of the slave.jar-based approaches.
Related
I am trying to install the VSTS build agent on a Windows Server 2016 using the below Powershell script but the script fails with the below error.
$tfsUrl=”http://vwmaztfsapp:8080/tfs/Collection”
$pool = ”Cli Execution”
$username = ”dev.local\svc_tfsAcc”
$password = ”MyPassword”
$AgentName = "Aut1"
CD "$AgentFolder\vsts-agent-win7-x64-2.122.1-Aut6\"
# configure agent to run as a Windows service that logs on as a domain account
& .\config.cmd --unattended -–url "$tfsUrl" --auth "integrated" -–pool "$pool" -–agent "$AgentName" --work "$AgentFolder\$AgentName" –-runAsService -–windowsLogonAccount "$username" –-windowsLogonPassword "$password"
Error below -
PS C:\Agents\vsts-agent-win7-x64-2.122.1-Aut6> C:\Agents\vsts-agent-win7-x64-2.122.1-Aut6\InstallAgent.ps1
config.cmd : Unrecognized command-line input arguments: 'unattended'. For usage refer to: .\config.cmd --help or ./config.sh --help
At C:\Agents\vsts-agent-win7-x64-2.122.1-Aut6\InstallAgent.ps1:49 char:1
+ & .\config.cmd --unattended -–url "$tfsUrl" --auth "integrated" -–poo ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (Unrecognized co...onfig.sh --help:String) [], RemoteException
+ FullyQualifiedErrorId : NativeCommandError
Is there something that I am missing?
I need to install the agent on multiple servers and that's the reason I want to automate this process.
Update
I was using command without "" such as:
.\config.cmd --unattended --url https://dev.azure.com/patricklu2020 --auth pat --token n2upx6epgl4srovzgp6is5wytjfkspvv6uvxxxx --pool default --agent myAgent --work D:\agent_work
If you want to configure as a service, you need to configure the agent from an elevated PowerShell window, this is required.
I would suggest you directly run the command from an elevated powershell window first, and then check if you are able to configure the agent correctly.
This kind of error maybe caused by passing arguments incorrectly. Kindly verify it.
I have created a jenkins node using jenkins-cli create node command. The node gets created successfully and I could see it in the web interface.
NODE_NAME=$1
LABEL=$2
cat <<EOF | java -jar jenkins-cli.jar -s http://myjenkins/jenkins/ create-node --username userId --password testPwd $1
<?xml version='1.0' encoding='UTF-8'?>
<slave>
<name>${NODE_NAME}</name>
<description></description>
<remoteFS>/Users/jenkins1/Desktop/workspace</remoteFS>
<numExecutors>1</numExecutors>
<mode>EXCLUSIVE</mode>
<retentionStrategy class="hudson.slaves.RetentionStrategy$Always"/>
<launcher class="hudson.plugins.sshslaves.SSHLauncher" plugin="ssh-slaves#1.22">
<host>test</host>
<port>22</port>
<credentialsId>test</credentialsId>
<maxNumRetries>0</maxNumRetries>
<retryWaitTime>0</retryWaitTime>
<sshHostKeyVerificationStrategy class="hudson.plugins.sshslaves.verifiers.KnownHostsFileKeyVerificationStrategy"/>
</launcher>
<label>${LABEL}</label>
<nodeProperties>
<hudson.slaves.EnvironmentVariablesNodeProperty>
<envVars serialization="custom">
<unserializable-parents/>
<tree-map>
<default>
<comparator class="hudson.util.CaseInsensitiveComparator"/>
</default>
<int>1</int>
<string>MVN_REPOS</string>
<string>/Users/jenkins1/Desktop/workspace</string>
</tree-map>
</envVars>
</hudson.slaves.EnvironmentVariablesNodeProperty>
</nodeProperties>
</slave>
EOF
But when I try to get the node after it has been created using jenkins-cli I get no such node error. However, I am able to get the node after saving the configuration through Jenkins UI. Could you please help me to fix this error.
java -jar jenkins-cli.jar -s http://myjenkins/jenkins/ get-node 'tests' --username userId --password testPwd
ERROR: No such node 'tests'
When you create a node or job from the cli, most times you have to return to Jenkins UI and run "Manage Jenkins >> Reload Configuration From Disk", otherwise cli changes sometime remain ineffective. Have you already tried doing this?
#funkfan As I have mentioned in my post, I am able to get the node after saving the configuration through Jenkins UI. I have even tried jenkins-cli reload-configuration command before saving the configuration from the Jenkins UI without any luck.
Hello I am interested in retrieving the Task ID from within inside a running container which lives inside of a EC2 host machine.
AWS ECS documentation states there is an environment variable ECS_CONTAINER_METADATA_FILE with the location of this data but will only be set/available if ECS_ENABLE_CONTAINER_METADATA variable is set to true upon cluster/EC2 instance creation. I don't see where this can be done in the aws console.
Also, the docs state that this can be done by setting this to true inside the host machine but would require to restart the docker agent.
Is there any other way to do this without having to go inside the EC2 to set this and restart the docker agent?
This doesn't work for newer Amazon ECS container versions anymore, and in fact it's now much simpler and also enabled by default. Please refer to this docu, but here's a TL;DR:
If you're using Amazon ECS container agent version 1.39.0 and higher, you can just do this inside the docker container:
curl -s "$ECS_CONTAINER_METADATA_URI_V4/task" \
| jq -r ".TaskARN" \
| cut -d "/" -f 3
Here's a list of container agent releases, but if you're using :latest – you're definitely fine.
The technique I'd use is to set the environment variable in the container definition.
If you're managing your tasks via Cloudformation, the relevant yaml looks like so:
Taskdef:
Type: AWS::ECS::TaskDefinition
Properties:
...
ContainerDefinitions:
- Name: some-name
...
Environment:
- Name: AWS_DEFAULT_REGION
Value: !Ref AWS::Region
- Name: ECS_ENABLE_CONTAINER_METADATA
Value: 'true'
This technique helps you keep everything straightforward and reproducible.
If you need metadata programmatically and don't have access to the metadata file, you can query the agent's metadata endpoint:
curl http://localhost:51678/v1/metadata
Note that if you're getting this information as a running task, you may not be able to connect to the loopback device, but you can connect to the EC2 instance's own IP address.
We set it with the so called user data, which are executed at the start of the machine. There are multiple ways to set it, for example: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html#user-data-console
It could look like this:
#!/bin/bash
cat <<'EOF' >> /etc/ecs/ecs.config
ECS_CLUSTER=ecs-staging
ECS_ENABLE_CONTAINER_METADATA=true
EOF
Important: Adjust the ECS_CLUSTER above to match your cluster name, otherwise the instance will not connect to that cluster.
Previous answers are correct, here is another way of doing this:
From the ec2 instance where container is running, run this command
curl http://localhost:51678/v1/tasks | python -mjson.tool |less
From the AWS ECS cli Documentation
Command:
aws ecs list-tasks --cluster default
Output:
{
"taskArns": [
"arn:aws:ecs:us-east-1:<aws_account_id>:task/0cc43cdb-3bee-4407-9c26-c0e6ea5bee84",
"arn:aws:ecs:us-east-1:<aws_account_id>:task/6b809ef6-c67e-4467-921f-ee261c15a0a1"
]
}
To list the tasks on a particular container instance
This example command lists the tasks of a specified container instance, using the container instance UUID as a filter.
Command:
aws ecs list-tasks --cluster default --container-instance f6bbb147-5370-4ace-8c73-c7181ded911f
Output:
{
"taskArns": [
"arn:aws:ecs:us-east-1:<aws_account_id>:task/0cc43cdb-3bee-4407-9c26-c0e6ea5bee84"
]
}
My ECS solution as bash and Python snippets. Logging calls are able to print for debug by piping to sys.stderr while print() is used to pass the value back to a shell script
#!/bin/bash
TASK_ID=$(python3.8 get_ecs_task_id.py)
echo "TASK_ID: ${TASK_ID}"
Python script - get_ecs_task_id.py
import json
import logging
import os
import sys
import requests
# logging configuration
# file_handler = logging.FileHandler(filename='tmp.log')
# redirecting to stderr so I can pass back extracted task id in STDOUT
stdout_handler = logging.StreamHandler(stream=sys.stderr)
# handlers = [file_handler, stdout_handler]
handlers = [stdout_handler]
logging.basicConfig(
level=logging.INFO,
format="[%(asctime)s] {%(filename)s:%(lineno)d} %(levelname)s - %(message)s",
handlers=handlers,
datefmt="%Y-%m-%d %H:%M:%S",
)
logger = logging.getLogger(__name__)
def get_ecs_task_id(host):
path = "/task"
url = host + path
headers = {"Content-Type": "application/json"}
r = requests.get(url, headers=headers)
logger.debug(f"r: {r}")
d_r = json.loads(r.text)
logger.debug(d_r)
ecs_task_arn = d_r["TaskARN"]
ecs_task_id = ecs_task_arn.split("/")[2]
return ecs_task_id
def main():
logger.debug("Extracting task ID from $ECS_CONTAINER_METADATA_URI_V4")
logger.debug("Inside get_ecs_task_id.py, redirecting logs to stderr")
logger.debug("so that I can pass the task id back in STDOUT")
host = os.environ["ECS_CONTAINER_METADATA_URI_V4"]
ecs_task_id = get_ecs_task_id(host)
# This print statement passes the string back to the bash wrapper, don't remove
logger.debug(ecs_task_id)
print(ecs_task_id)
if __name__ == "__main__":
main()
Background: I am creating a script to send out a message to everyone in my Domain. I was able to complete this using Invoke-WMImethod and MSG.exe. However, my supervisor wants a more customizable message to be sent. Like changing Color, font size, font style...etc. Which i have created using PowerShell.
Script:
Invoke-WmiMethod -ComputerName $Computer -Class Win32_Process -Name Create -ArgumentList {"C:\x\x\x\Powershell.exe -File `"\\Server\Share\Folder\Script.ps1`""}
When i run this script against my Computer it works perfectly. However, when i attempt to run it on a remote computer it fails.
I don't understand why.
It's the same exact script that i used with MSG.exe, which worked, but it still doesn't work with a powershell script.
I attempted to copy the script to the remote computers 'C:\' and run it from that file path but it still didn't work.
I've verified the file path to Powershell.exe is the same as the script and that the remote workstation can access the .PS1 Script.
However, the script does run and says it is successful with a Return Value of 0. Example:
__GENUS : 2
__CLASS : __PARAMETERS
__SUPERCLASS :
__DYNASTY : __PARAMETERS
__RELPATH :
__PROPERTY_COUNT : 2
__DERIVATION : {}
__SERVER :
__NAMESPACE :
__PATH :
ProcessId : 8748
ReturnValue : 0
My Suggestion would be to check for the versions on both the machines. The CmdLet you using may not be working on the previous version.
I again have a suggestion to make, you can use the below command to get your work done, if you have powershell Version 3 or above.
$Outputreport = "Test message to send data using port 443"
[System.Net.ServicePointManager]::ServerCertificateValidationCallback = { $true }
$wc = new-object System.Net.WebClient
Invoke-RestMethod -Method Post https://hostnamedotcom/cgi-bin/dir-path/$hostname-filename -Body $Outputreport
We have an application that we would like to run a script on just like we do in the console window with access to the applications libraries and context, but we need to run it periodically like a cron job.
While the permanent answer is obviously a Quartz job, we need to the do this before we are able to patch the application.
Is there something available that gives us the same environment as the console-plugin but can be run via command-line or without a UI?
you can run a console script like the web interface does but just with a curl like this:
curl -F 'code=
class A {
def name
}
def foo = new A(name: "bar")
println foo.name
' localhost:8080/console/execute
You'll get the response as the console would print below.
With regard to #mwaisgold 's solution above, I made a couple of quick additions that helped. I added a little bit more to the script to handle authentication, plus the -F flag for curl caused an ambiguous method overloading error with the GroovyShell's evaluate method, so I addressed that by using the -d instead:
#/bin/bash
curl -i -H "Content-type: application/x-www-form-urlencoded" -c cookies.txt -X POST localhost:8080/myapp/j_spring_security_check -d "j_username=admin&j_password=admin"
curl -i -b cookies.txt -d 'code=
int iterations = 0
while (iterations < 10) {
log.error "********** Console Cron Test ${iterations++} ***********"
}
log.error "********** Console Cron Test Complete ***********"
' localhost:8080/myapp/console/execute