I'm using Grails 2.0.4 and elasticsearch 0.19.8, with a homegrown build of the elasticsearch-grails plugin to use 0.19.8, running on a micro Ubuntu instance of EC2.
I'm building my grails project locally as a war. When I drop this was in a jetty container locally (on my laptop), jetty starts and I'm able to navigate to the root context (as expected) and get my site.
However, when I take the same war (confirmed via md5sum) in the same version of jetty (confirmed via md5sum) on a micro EC2 instance, I get strange results.
Occasionally, jetty starts and my application is available...all is well. But then other times, jetty won't start...and the logs are complaining about elasticsearch.
Now, I'm the first to admit that running both elasticsearch and jetty (with a grails war) on the same node is, well, not the greatest idea...but, this is a test environment, and I'm okay with swapping to disk (not a perf testing lab).
But, what's driving me batty is the inconsistency in running my application. Currently, I cannot get it to start (which, I suppose, counters my "inconsistency" argument). But even taking a "starting" war, with MINIMAL changes in the war (no domain object changes, just changes in controller and/or view), my app won't start - complaining about elasticsearch.
Here's the gist
In the past, I've solved this by stopping elasticsearch, and blowing away /var/lib/elasticsearch/elasticsearch (which is basically, as I understand it, a crude way of blowing away the elasticsearch index).
But now, this doesn't appear to be working. Neither does uninstall/reinstall elasticsearch.
Can anybody offer any advice on what to look for and/or where to look?
Ok, this is driving my crazy, but I found an answer.
/data contains an elasticsearch directory which, apparently, had bad data...I deleted this directory and restarted jetty and...
Viola!
I have NO idea how it got into this state, nor the actual ramifications of deleting the directory (I think elasticsearch data is persisted in the elasticsearch node...)
I suspect an incomplete shutdown of my app...
Thanks for everybody's input.
Related
Ok, I'm deploying my Rails App using Capistrano. I'm also using Puma. I've followed this tutorial to get it to work, although I'm using Debian rather then Ubuntu.
Everything works fine and I can deploy my app without issues. However if my server crashes or the server restarts, the App doesn't restart itself and the only way I got it to restart was deploying it again with the following command cap production deploy from within my App in my local machine, which we all can agree that's not ideal.
There's loads of information on the web on how to deploy a Rails App with Passenger, which I'd rather avoid to use due to lack of resources on the server part. I've also found this tutorial which seems to be a bit outdated.
Can someone please point me to an updated tutorial or give some directions on how I could get my App to start/restart who the server?
Many thanks
EDIT
As per #mudasobwa's comments, I'm detailing the steps I've taken after reading this page:
I have copied the contents of https://github.com/puma/puma/blob/master/tools/jungle/init.d/puma into /etc/init.d/puma made it executable. I've also copied the contents of https://github.com/puma/puma/blob/master/tools/jungle/init.d/run-puma into /usr/local/bin/run-puma also made it executable.
Lastly I've created a puma.conf file in /etc.
After that I've created the following directory: /path/to/app/tmp/puma and added these two files: pid and state. Note that I've also added the aforementioned folders into Capistrano's shared links structure.
After the above I've restarted my server and the App did not start as I expected.
What am I missing here?
I have a Jenkins set-up where there is a master running on OS X with a Windows slave running on the same box as VM.
On many occasions when the VM is restarted the Jenkins service appears to either not start or possible encounters an error.
The set-up of the service looks correct and the VM is configured to automatically login as the Jenkins user, when its manually started everything seems to work fine so I can only assume the problem is on start-up of the box.
I have two questions:
Are there any well known gotcha's that can cause this?
Does anyone have some good strategies for debugging this? I'm assuming the answer will be somewhere in the Windows Logs but finding it is proving difficult (since the box and the user both contain the word Jenkins a simple find isn't helpful).
I just got a chance to test-drive a VPS for a week, and decided to try Grails on it. The problem is, that it shuts itself down.
Details:
VPS - 512MB Ram, Ubuntu 12.10 x64 (no particular reason for x64)
Oracle Java 7u17
Latest GVM 0.9.5
Grails 2.2.1
What I did was follow along this tutorial http://grails.org/Quick+Start, which is very basic. Everything went smooth until I did grails run-app.
After doing the initialization, it showed running for like 5s, and I could even start loading the page, but it suddenly showed Killed in the terminal. This is what the terminal showed:
root#jp:/var/grails/my-project# grails run-app
| Running Grails application
Killed
There was no input during that time whatsoever. Any ideas on the cause of this problem?
You should only ever run Grails with the run-app command when developing locally. The reason behind this is because run-app starts your Grails app with a lot of dynamic behavior that is great for rapid development, but horrible performance-wise for running on an actual server.
Refer to Grails' User Guide on how best to deploy your application:
http://grails.org/doc/latest/guide/gettingStarted.html#deployingAnApplication
As the docs above state, the correct way to run your Grails app is to embed it in a servlet container. Tomcat is a good place to start since Grails uses that by default when running locally. You may also need to play around with the VM flags of your servlet container, depending on your environment (again, the docs give a few suggestions here).
You can redirect output of your command if it get's killed immediately on your terminal.
grails run-app > output.txt
Then open output.txt and from there you can dissect the problem.
For my case, i got an incorrect JAVA_HOME directory.
Hope it helps.
Recently I configured my instance into a micro environment in EC2 with glassfish and mysql in windows..
I deployed my war and i was able to access my site through http.
I changed my application and redeployed the war and it also worked.
When I was about to redeploy the war for 4th or 5th time, the application got deployed, I saw the message in the log file. But I was unable to access the site through http.
Then I tried the command "asadmin list-applications" and I got the following message.
Error occurred during initialization of VM
Could not reserve enough space for object heap
After that I was not able to connect to my instance through RDP and I had to reboot, I was able to access it again after that. I started the servers again (glassfish mysql), but no luck.
I noticed that the memory usage is around 90% or more. CPU isage is low.
now I can not access my site through http. what shall i do ?
Thanks in advance !
Honestly, there are a couple issues working against you here:
1) Windows requires FAR more RAM than Ubuntu to run at a minimum decent level.
2) GlassFish has a much larger footprint than Tomcat or Jetty.
Is there any particular reason you need Windows? Like is there a specific need that your server run some executables for file processing or something like that outside the JVM? Most would agree that Linux (Ubuntu or other) will give you much better results in performance and stability to run an App Server like GlassFish in any environment.
It is the first time I have ever launched live a website (with Grails web framework under Amazon EC2 platform and Cloud Foundry) and I realized quickly that I am not ready for monitoring and maintening correctly my application in production mode (fortunately the website is accessible to a very limited number of users) .
The issues I have faced so far are:
Cannot change my views. I need to redeploy my application
I have no monitoring. I don't know who is connected, when do they sign in / sign out...
Redploying my application (upload WAR + deploy) takes at least 30 minutes.
I don't know how to restart my Tomcat server without a redeploy through Cloud Foundry !
...
So, my question is very simple:
What tools (including grails plugins) and methods can you recommend me for taking me out from my current blindness?
I am not sure how much this will help, however I use the JavaMelody Grails Plugin(http://www.grails.org/plugin/grails-melody) I use it to see if the site is being used before I pull down the service.
Hope that helps.
I tried Cloud Foundry in it's early days and found it a little rough. Sounds like that's still the case with 30+ minute deploys and the inability to restart your tomcat server. Half of your problems could be solved if you just created your own EC2 instance, installed tomcat and managed your own deployments. That'll let you bounce tomcat through shell access:
sudo /etc/init.d/tomcat6 restart
and redeploy your app
sudo /etc/init.d/tomcat6/stop
cp my.war /to/tomcat/dir
sudo /etc/init.d/tomcat6 start
or else you could do it through the tomcat admin console, but I find it to be flaky.
Regarding monitoring, there are a couple of ways you can do that. The easiest is to add CloudWatch monitoring to your system. That'll give you more insight into the performance of the application.
For more detailed monitoring regarding who's connecting to your app, I'd suggest looking at Google Analytics or Mint. If you need to get beyond that (with per user monitoring), you'll likely have to roll your own logging/tracking for what meets your needs. There are also other paid packages out there along the lines of Google Analytics and Mint that you can integrate with, but what fits your needs best, I can't say.
For actual monitoring of deployed system you can also use Hiperic HQ. It's a monitoring solution from Spring Source, who also are owners of Grails Framework.
It can manage, at your case:
tomcat server
database
linux
network
etc
btw redeploying app with changes is ok. it's a very bad practice to modify running app, on the production server.
I don't know if JMX and JConsole can help, but that might be a good way to see what the status of JMX-enabled POJOs is. Spring makes this easy to do.
Yes, you need to re-deploy your app when you change stuff in it, there's no way around that.
Deploy/re-deploy time has been cut significantly recently if using the grails plugin (btw, what version of the plugin do you use?) In some cases the upload time is as short as 15 seconds. Add 2-3 minutes for Amazon to spin up the instances and the deploy time is still pretty manageable. For re-deploy the instances don't have to be started, so it's even less than that.
To restart Tomcat login into your CloudFoundry account, click on the Deployment details, click on the instance that's running your Tomcat, and there will be a button "Restart service" that will do just that - restart Tomcat service.
You don't have to start your own EC2 instances in order to get a shell access. Copy the public DNS name of the instance from DeploymentDetails, and SSH into it using the private key you entered when registering for Cloudfoundry. Example:
ssh -i /path/to/gsg-keypair.pem root#your_instance_DNS_name