A process "irqbnc1" is running on my server that is hogging high CPU utilisation. I checked and found that Jenkins is starting this process. This child process starts automatically if i killed this process forcefully. Even I am not able to find the binary of this process.
systemctl status jenkins
● jenkins.service - LSB: Jenkins Automation Server
Loaded: loaded (/etc/rc.d/init.d/jenkins)
Active: active (running) since Sat 2017-06-10 16:06:49 IST; 1h 29min ago
Docs: man:systemd-sysv-generator(8)
Process: 32637 ExecStop=/etc/rc.d/init.d/jenkins stop (code=exited, status=0/SUCCESS)
Process: 32696 ExecStart=/etc/rc.d/init.d/jenkins start (code=exited, status=0/SUCCESS)
CGroup: /system.slice/jenkins.service
├─ 5791 ./irqbnc1 -c hsjwj.conf
├─13265 sh
├─24222 sh
└─32716 /etc/alternatives/java -Dcom.sun.akuma.Daemon=daemonized -Djava.awt.headless=true -DJENKINS_HOME=/var/lib/jenkins -jar /usr/lib/jenkins/jenkins.war --logfile=/var/log/jenkins/jenkins.log --webroot=/var/cache/jenkins/war --daemon --httpPort=8080 --debug=5 --handlerCountMax=100 --handlerCountMaxIdle=20
top command output-
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
5791 jenkins 20 0 370148 6976 524 S 300.0 0.0 147:59.75 ./irqbnc1 -c hsjwj.conf
Let me know the purpose of this process and how can i disable this process ?
That process is a bitcoin (cryptocoin) minning script.
You have a vulnerability in your Jenkins that allow an attacker to remote-execute code on your machine. The attacker created a script that downloads from an external source a JSON config file and a binary file. The binary file takes as an argument the config file and starts the mining process.
I would recommend to reinstall your OS + jenkins (latest versions).
Related
I'm using pulumi to manage kubernetes deployments. One of the deployments runs an image which intercepts SIGINT and SIGTERM signals to perform a graceful shutdown like so (this example is running in my IDE):
{"level":"info","msg":"trying to activate gcloud service account","time":"2021-06-17T12:19:25-05:00"}
{"level":"info","msg":"env var not found","time":"2021-06-17T12:19:25-05:00"}
{"Namespace":"default","TaskQueue":"main-task-queue","WorkerID":"37574#Paymahns-Air#","level":"error","msg":"Started Worker","time":"2021-06-17T12:19:25-05:00"}
{"Namespace":"default","Signal":"interrupt","TaskQueue":"main-task-queue","WorkerID":"37574#Paymahns-Air#","level":"error","msg":"Worker has been stopped.","time":"2021-06-17T12:19:27-05:00"}
{"Namespace":"default","TaskQueue":"main-task-queue","WorkerID":"37574#Paymahns-Air#","level":"error","msg":"Stopped Worker","time":"2021-06-17T12:19:27-05:00"}
Notice the "Signal":"interrupt" with a message of Worker has been stopped.
I find that when I alter the source code (which alters the docker image) and run pulumi up the pod doesn't gracefully terminate based on what's described in this blog post. Here's a screenshot of logs from GCP:
The highlighted log line in the image above is the first log line emitted by the app. Note that the shutdown messages aren't logged above the highlighted line which suggests to me that the pod isn't given a chance to perform a graceful shutdown.
Why might the pod not go through the graceful shutdown mechanisms that kubernetes offers? Could this be a bug with how pulumi performs updates to deployments?
EDIT: after doing more investigation I found that this problem is happening because starting a docker container with go run /path/to/main.go actually ends up created two processes like so (after execing into the container):
root#worker-ffzpxpdm-78b9797dcd-xsfwr:/gadic# ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.3 0.3 2046200 30828 ? Ssl 18:04 0:12 go run src/cmd/worker/main.go --temporal-host temporal-server.temporal.svc.cluster.local --temporal-port 7233 --grpc-port 6789 --grpc-hos
root 3782 0.0 0.5 1640772 43232 ? Sl 18:06 0:00 /tmp/go-build2661472711/b001/exe/main --temporal-host temporal-server.temporal.svc.cluster.local --temporal-port 7233 --grpc-port 6789 --
root 3808 0.1 0.0 4244 3468 pts/0 Ss 19:07 0:00 /bin/bash
root 3817 0.0 0.0 5900 2792 pts/0 R+ 19:07 0:00 ps aux
If run kill -TERM 1 then the signal isn't forwarded to the underlying binary, /tmp/go-build2661472711/b001/exe/main, which means the graceful shutdown of the application isn't executed. However, if I run kill -TERM 3782 then the graceful shutdown logic is executed.
It seems the go run spawns a subprocess and this blog post suggests the signals are only forwarded to PID 1. On top of that, it's unfortunate that go run doesn't forward signals to the subprocess it spawns.
The solution I found is to add RUN go build -o worker /path/to/main.go in my dockerfile and then to start the docker container with ./worker --arg1 --arg2 instead of go run /path/to/main.go --arg1 --arg2.
Doing it this way ensures there aren't any subprocess spawns by go and that ensures signals are handled properly within the docker container.
In my AWS Ubuntu machine, I started a Jenkins build and somehow due to this machine hang so I forcefully stop the instance and then start again after that Jenkins is not starting up. Getting below.
<pre><code>
jenkins.service - LSB: Start Jenkins at boot time
Loaded: loaded (/etc/init.d/jenkins; generated)
Active: failed (Result: exit-code) since Tue 2020-12-01 15:56:48 UTC; 4min 6s ago
Docs: man:systemd-sysv-generator(8)
Process: 27605 ExecStart=/etc/init.d/jenkins start (code=exited, status=7)
Dec 01 15:56:47 ip-172-31-7-133 systemd[1]: Starting LSB: Start Jenkins at boot time...
Dec 01 15:56:47 ip-172-31-7-133 jenkins[27605]: Correct java version found
Dec 01 15:56:47 ip-172-31-7-133 jenkins[27605]: * Starting Jenkins Automation Server jenkins
Dec 01 15:56:48 ip-172-31-7-133 jenkins[27605]: ...fail!
Dec 01 15:56:48 ip-172-31-7-133 systemd[1]: jenkins.service: Control process exited, code=exited status=7
Dec 01 15:56:48 ip-172-31-7-133 systemd[1]: jenkins.service: Failed with result 'exit-code'.
Dec 01 15:56:48 ip-172-31-7-133 systemd[1]: Failed to start LSB: Start Jenkins at boot time.
</code></pre>
I imagine it can't start because there is already a Jenkins process running which is consuming that same port. You should be able to verify that by looking at the Jenkins logfile.
You can do:
ps auxw | grep jenkins
If it returns a process, then you can kill -9 PID.
For example:
[user#server~]$ ps auxw | grep jenkins
jenkins 4656 0.3 33.2 5070780 2716228 ? Ssl Nov05 144:15 /etc/alternatives/java -Djava.awt.headless=true -DJENKINS_HOME=/var/lib/jenkins -jar /usr/lib/jenkins/jenkins.war --logfile=/var/log/jenkins/jenkins.log --webroot=/var/cache/jenkins/war --httpPort=8080 --debug=5 --handlerCountMax=100 --handlerCountMaxIdle=20
user 14665 0.0 0.0 119416 908 pts/0 S+ 23:08 0:00 grep --color=auto jenkins
[user#server~]$ kill -9 4656
Then try and start your Jenkins instance. Depending on how your Jenkins server is setup you would most likely need to do the above via sudo.
I don't know if this applies to you, but it was my problem. It is similar to Terry Sposato's answer.
We had a scenario where our Ubuntu node was running a standalone instance of Jenkins as wells an SSH started child instance managed from another node.
We saw these sort of service start errors if the SSH started child instance was running.
I resolved this by
Accessing the parent Jenkins instance and selecting disconnect for the Ubuntu node
Immediately invoking sudo /etc/init.d/jenkins start on the Ubuntu node to start it's standalone instance.
After that the jenkins parent in step 1 would eventually reconnect to the the Ubuntu node.
We have not seen this sort of behavior with our CentOS node that also has a standalone and child instance of Jenkins running. I suspect it's a defect with Ubuntu's lsb startup scripts, my debugging showed that the problem happened when /etc/init.d/jenkins sourced /lib/lsb/init-functions.
I'm looking to redirect some logs from a command run with kubectl exec to that pod's logs, so that they can be read with kubectl logs <pod-name> (or really, /var/log/containers/<pod-name>.log). I can see the logs I need as output when running the command, and they're stored inside a separate log directory inside the running container.
Redirecting the output (i.e. >> logfile.log) to the file which I thought was mirroring what is in kubectl logs <pod-name> does not update that container's logs, and neither does redirecting to stdout.
When calling kubectl logs <pod-name>, my understanding is that kubelet gets them from it's internal /var/log/containers/ directory. But what determines which logs are stored there? Is it the same process as the way logs get stored inside any other docker container?
Is there a way to examine/trace the logging process, or determine where these logs are coming from?
Logs from the STDOUT and STDERR of containers in the pod are captured and stored inside files in /var/log/containers. This is what is presented when kubectl log is run.
In order to understand why output from commands run by kubectl exec is not shown when running kubectl log, let's have a look how it all works with an example:
First launch a pod running ubuntu that are sleeping forever:
$> kubectl run test --image=ubuntu --restart=Never -- sleep infinity
Exec into it
$> kubectl exec -it test bash
Seen from inside the container it is the STDOUT and STDERR of PID 1 that are being captured. When you do a kubectl exec into the container a new process is created living alongside PID 1:
root#test:/# ps -auxf
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 7 0.0 0.0 18504 3400 pts/0 Ss 20:04 0:00 bash
root 19 0.0 0.0 34396 2908 pts/0 R+ 20:07 0:00 \_ ps -auxf
root 1 0.0 0.0 4528 836 ? Ss 20:03 0:00 sleep infinity
Redirecting to STDOUT is not working because /dev/stdout is a symlink to the process accessing it (/proc/self/fd/1 rather than /proc/1/fd/1).
root#test:/# ls -lrt /dev/stdout
lrwxrwxrwx 1 root root 15 Nov 5 20:03 /dev/stdout -> /proc/self/fd/1
In order to see the logs from commands run with kubectl exec the logs need to be redirected to the streams that are captured by the kubelet (STDOUT and STDERR of pid 1). This can be done by redirecting output to /proc/1/fd/1.
root#test:/# echo "Hello" > /proc/1/fd/1
Exiting the interactive shell and checking the logs using kubectl logs should now show the output
$> kubectl logs test
Hello
The command:
pi#raspberrypi:~ $ mosquitto
1566609792: mosquitto version 1.5.7 starting
1566609792: Using default config.
1566609792: Opening ipv4 listen socket on port 1883.
1566609792: Error: Address already in use
can be invoked to start mosquitto. Is there a better single command that can verify the broker is running. I would like to avoid using using pub and sub commands to test and use a simple query command. I would also like to avoid using the mosquitto command to determine if the installation is active / running
Assuming that you installed mosquitto using apt-get then it will have been set up as a systemd service so:
service mosquitto status
will show if it's running or not:
● mosquitto.service - LSB: mosquitto MQTT v3.1 message broker
Loaded: loaded (/etc/init.d/mosquitto; generated; vendor preset: enabled)
Active: active (running) since Mon 2019-08-12 22:39:38 BST; 1 weeks 4 days ag
Docs: man:systemd-sysv-generator(8)
Process: 32183 ExecStop=/etc/init.d/mosquitto stop (code=exited, status=0/SUCC
Process: 32220 ExecStart=/etc/init.d/mosquitto start (code=exited, status=0/SU
CPU: 8min 53.255s
CGroup: /system.slice/mosquitto.service
└─32226 /usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf
else just using ps will show if the process is running:
$ ps -efc | grep mosquitto
mosquit+ 32226 1 TS 19 Aug12 ? 00:08:53 /usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf
For some reason when using systemd unit files my docker containers start but get shut down instantly. I have tried finding logs but can not see any indication on why this is happening. Is there someone that knows how to solve this / find the logs that show what is happening?
Note: When starting them manually after boot with docker start containername then it works (also when using systemctl start nginx)
After some more digging I found this error: could not find udev device: No such device it could have something to do with this?
Unit Service file:
[Unit]
Description=nginx-container
Requires=docker.service
After=docker.service
[Service]
Restart=always
RestartSec=2
StartLimitInterval=3600
StartLimitBurst=5
TimeoutStartSec=5
ExecStartPre=-/usr/bin/docker kill nginx
ExecStartPre=-/usr/bin/docker rm nginx
ExecStart=/usr/bin/docker run -i -d -t --restart=no --name nginx -p 80:80 -v /projects/frontend/data/nginx/:/var/www -v /projects/frontend: nginx
ExecStop=/usr/bin/docker stop -t 2 nginx
[Install]
WantedBy=multi-user.target
Journalctl output:
May 28 11:18:15 frontend dockerd[462]: time="2015-05-28T11:18:15Z" level=info msg="-job start(d757f83d4a13f876140ae008da943e8c5c3a0765c1fe5bc4a4e2599b70c30626) = OK (0)"
May 28 11:18:15 frontend dockerd[462]: time="2015-05-28T11:18:15Z" level=info msg="POST /v1.18/containers/nginx/stop?t=2"
May 28 11:18:15 frontend dockerd[462]: time="2015-05-28T11:18:15Z" level=info msg="+job stop(nginx)"
Docker logs: empty (docker logs nginx)
Systemctl output: (systemctl status nginx, nginx.service)
● nginx.service - nginx-container
Loaded: loaded (/etc/systemd/system/multi-user.target.wants/nginx.service)
Active: failed (Result: start-limit) since Thu 2015-05-28 11:18:20 UTC; 12min ago
Process: 3378 ExecStop=/usr/bin/docker stop -t 2 nginx (code=exited, status=0/SUCCESS)
Process: 3281 ExecStart=/usr/bin/docker run -i -d -t --restart=no --name nginx -p 80:80 -v /projects/frontend/data/nginx/:/var/www -v /projects/frontend:/nginx (code=exited, status=0/SUCCESS)
Process: 3258 ExecStartPre=/usr/bin/docker rm nginx (code=exited, status=0/SUCCESS)
Process: 3246 ExecStartPre=/usr/bin/docker kill nginx (code=exited, status=0/SUCCESS)
Main PID: 3281 (code=exited, status=0/SUCCESS)
May 28 11:18:20,frontend systemd[1]: nginx.service holdoff time over, scheduling restart.
May 28 11:18:20 frontend systemd[1]: start request repeated too quickly for nginx.service
May 28 11:18:20 frontend systemd[1]: Failed to start nginx-container.
May 28 11:18:20 frontend systemd[1]: Unit nginx.service entered failed state.
May 28 11:18:20 frontend systemd[1]: nginx.service failed.
Because you have not specified a Type in your systemd unit file, systemd is using the default, simple. From systemd.service:
If set to simple (the default if neither Type= nor BusName=, but
ExecStart= are specified), it is expected that the process
configured with ExecStart= is the main process of the service.
This means that if the process started by ExecStart exits, systemd
will assume your service has exited and will clean everything up.
Because you are running the docker client with -d, it exits
immediately...thus, systemd cleans up the service.
Typically, when starting containers with systemd, you would not use
the -d flag. This means that the client will continue running, and
will allow systemd to collect any output produced by your application.
That said, there are fundamental problems in starting Docker containers with systemd. Because of the way Docker operates, there really is no way for systemd to monitor the status of your container. All it can really do is track the status of the docker client, which is not the same thing (the client can exit/crash/etc without impacting your container). This isn't just relevant to systemd; any sort of process supervisor (upstart, runit, supervisor, etc) will have the same problem.