I understand how to setup a machine to run Resque with the environment, etc. What I cannot find, is now to point a Web (etc) server to (use) a queue on a separate machine.
For example, I am using Resque to handle mail (with the resque_mailer gem). Resque::Mailer is included in my ApplicationMailer setup, and everything works fine all running on one machine. But, I do not wish to use the resources of the Web machine for the background-processes.
How do I tell the ApplicationMailer, or configure Resque::Mailer, so that the request is routed to another machine on the internal network?
The answer was - you don't have to do anything other than ensure the system that is generating the work (the 1st machine) can reach the Redis instance, so it can update the queue there.
The 2nd machine, which carries-out the jobs, also only needs to be able to find the Redis server, which it will check to find new jobs to do.
The Redis server can be anywhere reachable by the two machines - on the 1st, 2nd, or a 3rd machine - in the context of this example.
Related
I'm sure this is possible somehow, but never really had a need to do so previously. I have a bunch of Docker containers that run on an UBUNTU host. One of the containers is an NGINX server that serves as a webserver and reverse proxy. What I would like to do is setup some sort of 'switch' or mechanism on the host or preferably another device that does something like described below. The server does have production and development versions of the web applications, so I really just want to set a header if it is running but in maintenance.
Set up something on the host other device that indicates or detects the state of the NGINX server.
a. Running normally.
b. Running but undergoing maintenance (would have to be manually set into that mode)
c. Not running.
Depending upon the state in 1 it would do the following:
a. Just pass the request through to the server.
b. Pass through the request, but possibly set a header or something to indicate it is in maintenance mode.
c. Redirect the request to an external URL, basically the public facing page for the business.
Not really sure how to approach that since it seems that I would need an HTTP listener on the host, or possibly on a router or firewall or other device (we have a Fortigate and a Watchguard). That would check the HTTP request, and then take the appropriate action based upon what "mode" we are in. The UBUNTU host is pretty much bare bones, i.e. without Apache or another web server because everything is pretty much in the Docker package.
If I were to set that up on the Debian host, seems like we could just have an environment variable that defaults to PROD and then just set it to DEV when we are working on the code, and then set it back to PROD when we are done, and then the process would.
ping the DOCKER NGINX instance to see if it is running (i.e. status code).
if it is running and PROD, just forward on.
if it is running and DEV, set a header to indicate so and forward on.
if it is not running, redirect externally.
If the server is completely down though, that would fail.
Any ideas as to how to actually do that ? Ideally the 'processor' would not reside on the UBUNTU server at all and would always be running.
I have two questions about rails server:
Do I have to start the server from within the application folder?
Is the server I started only for that application?
If they are true, this does not quite make sense to me, since why do I need to start multiple servers?
Or is there some kind of master configuration, so that one server can route to different applications? Is Capistrano for this purpose?
I'm going to assume you're talking about the rails server command, for running a local rails server to test your application, and that you're not talking about setting up a rails application on a remote server. Please mention if that is not the case.
Yes, you must execute rails server from within the root folder of your rails application.
Yes, the server you started is only for that application. It's a self-contained thing.
You should not need to start multiple servers. Even if you have multiple applications, you probably don't need to have more than one running at a time. So, you can shut down the rails server in one application (Ctrl-C) and then cd to your new application, and start a new rails server there with rails server.
If you do need to run two local rails applications at once, you can do so by running them on different ports. So, the first one, you can just execute rails server and it will make your site available at localhost:3000 (because port 3000 is the default port). The next one, you can specify a port other than 3000 - eg. rails server -p 3001 to get a rails app at localhost:3001.
Capistrano is for deploying your applications to a remote server, not for running them locally on your own computer. So, it is not relevant here. What you may be interested in is http://pow.cx/
Again, I've assumed you're talking about running your rails app locally on your own computer. If you're referring to deploying it to the internet on a server, then you can ignore this answer.
Amazon Web Service has now a worker tiers in their Elastic Beanstalk. But, it nevertheless confuse us who come from the days of Worker dyno.
As a comparison, in Heroku, one can configure two dynos (something like processor?) each for web and worker. The web will work for any request, and will timeout normally at 15 secs. Thus, if you have a request that last more than that, your request will simply timed-out although not terminated per se. In that case, you should use worker and your web dyno should visit the endpoint several times per minutes (maybe) to check if there is any result to be brought back to the user. To make either worker or web dyno, what you need is just slide the slider and you are good to go. Sometimes, you may need a Procfile. But there is nothing fancy, or something really difficult, or confusing about it.
In AWS EBS (Elastic Beanstalk), since day 1 you hit eb init, you will be asked whether it is a Standard or Worker. When you hit Standard, it seems there is no way to make it as worker as well.
In our situation, the worker and standard web is located under one application. So, how could we use an EBS instance both for worker and standard. Our worker is using sidekiq, and redis. Please, point to any guidance or help us in this matter.
AWS Elastic Beanstalk has two types of Environments - Web tier and Worker tier.
Web tier environments are meant for web applications - http/https request processing. You get one or more EC2 instances behind a load balancer. You can get other resources like database per your requirement. You can choose the platform you wish e.g. Ruby, Python, Java, Node.js, PHP, Docker.
Worker environments are meant for asynchronous message processing. When you create a worker environment you do not have a load balancer. All your EC2 instances are in an autoscaling group. All these instances are running a daemon which is polling a single SQS queue for messages. When a message is pulled by the daemon from the SQS queue, the daemon sends a HTTP Post request on localhost:80. You can configure the port but the important thing is that the daemon posts the message as an HTTP request on localhost. Your worker application is actually a web application that receives the post request and processes the message. After the message is successfully processed the worker daemon expects that your web application running on localhost returns a HTTP 200 OK response. The daemon then deletes the message from SQS queue. You can write your worker application for any platform just like standard web server applications - Ruby, Python, Java, Node.js, PHP, Docker.
Based on my understanding of your usecase I would recommend creating two Elastic Beanstalk environments - one Standard and one Worker environment. The Standard web server receives HTTP requests and processes them synchronously. This environment puts the relevant data in an SQS queue. The second environment is a worker and the daemon running on this environment polls this SQS queue for messages. Your second environment is a web application that is NOT open to the internet. The worker daemon posts the messages as HTTP requests to your worker environment. Thus you can process long running workloads asynchronously using this second worker environment.
With worker environments you can use your own queues or Elastic Beanstalk can generate a queue for you. You can configure parameters like message visibility timeout, http connections based on your requirements or you can use the defaults.
Below are some links that may be useful for you:
http://aws.amazon.com/blogs/aws/background-task-handling-for-aws-elastic-beanstalk/
http://blogs.aws.amazon.com/application-management/post/Tx1Y8QSQRL1KQZC/Elastic-Beanstalk-Video-Tutorial-Worker-Tier
https://stackoverflow.com/a/23942498/161628
Does this meet your requirements? Please let me know if you have further questions.
Update
You need to upload your source code at two places - once for the worker environment and once for the web server environment. If someone was starting from scratch then they might have two separate code bases. But I think in your case I think it should be perfectly fine to have a single code base shared between the two environments. Suppose your web request arrives at '/register', then the register() method in your application can post messages to an SQS queue and be done with the HTTP request. Now your worker environment will poll the SQS queue and post messages over HTTP on localhost to the URL '/async_register' which will invoke a method async_register() in your application and do the asynchronous processing. These two methods can live in the same source code bundle which can be shared by both the worker and web server environment. The code path taken by worker and web server will be different so that web server environments will invoke register() and worker environments will invoke the async_register() method.
Another caveat is that HTTP requests sent by the worker daemon on localhost will contain an HTTP header - "User-Agent": "aws-sqsd/1.1". Read more here. So in your web application you can have a single listener to post requests on "/register" and depending on whether this header is present or not you invoke register() or async_register() methods internally.
Also I think if you want to share the code base between the two environments you can upload the code base at only one place. Your environments are logically grouped into applications. So you can have a single application. You upload your source code to this application using the "CreateApplicationVersion" API call. Suppose you upload an application version with label 'v1'. You can now create a worker environment and a web server environment under the same application. When you create an environment you need to provide a version to deploy to your enviroment. In this case you can deploy v1 to both environments. So you will be sharing the same source code for both environments. When you have a new version "v2". You upload this version and then perform an update on both environments changing their version to "v2".
The same version of the source code can be deployed to both environments. They will be running on different EC2 instances because one environment is dedicated for responding to web requests and one environment is dedicated for responding to asynchronous web requests (worker).
I have a Rails (2.3.8) application that will need to start and maintain a SSH tunnel whenever the application is started using 'script/server' or when started using Passenger. When script/server is ^C'd or the Passenger instance is shut down the SSH tunnel should be destroyed.
I do not want the tunnel to be started when I run 'script/console' - so using config/environment.rb doesn't look like a good option.
As such, I don't want the tunnel to be backgrounded - I want it to be attached and owned by the ruby process, and I only want one tunnel per server.
The tunnel itself will most likely be started by running SSH directly, but if there's a simple way to do it using the SSH libraries I'll use that instead.
Is there a way to do this in Rails? I can think of a way to do it using config/environment.rb using a series of lock files and other bits of messing around, but I was hoping for some sort of :on_server_start and :on_server_exit hook.
--
For the curious, I need to do this as the Rails application is running in location A and will receive updates from a series of services in location B. Location B, however, does not have a direct route to location A. I will be starting a SSH tunnel from the application to a machine in location B and the services in location B will send updates to that machine instead.
Managed to solve this one using config/environments/*.rb and a quick fork hack. See gist: http://gist.github.com/466267
So, it'll be run if you use RAILS_ENV=production but not for test/development.
I've got everything working with ferret and acts_as_ferret for development (or localhost DRb), but I can't get my multiple host deployment working. All of the remote systems get ECONNREFUSED when accessing the port. On the ferret server, the daemon is listening on localhost only despite the configuration listing the FQDN as the host.
I also tried switching to a UNIX socket to share data between the ferret DRb daemon and the app code but it too gets ECONNREFUSED. (The socket is available to all of the machines via an NFS mount).
Is there a better way to do this or should I be looking for another search indexer? Thanks.
I did figure out that if the address is changed to druby://0.0.0.0:port that it would listen on all ips on the DRb server; however, it doesn't provide any protection against bad code injection into the DRb process.
Basically don't use ferret. I'm on to Xapian with acts_as_xapian for RoR. It supports multiple processes reading but only one writing, so it's an offline index. However, I will be able to make use of sharing the same index between multiple servers via the shared file system (NFS).
Check out Pitfalls of acts_as_ferret, with DrbServer to the rescue
http://www.subelsky.com/2007/03/pitfalls-of-actsasferret-with-drbserver.html
worked pretty well for me. The only thing I'd add is be sure to set the host value to where you're ferret is running.