Our rails app acts as a go-between for the UI and a java-powered API at the back. Basically, rails takes requests from the browser, tells the API what it needs, does some normalization and formatting, then hands it back to the browser. Our app is ALSO an installed app, often without internet access. Oh, and it all gets compiled into a JRuby war.
Currently, to get statuses of various things over time, we're running some javascript intervals every X seconds. We've started discussing a better solution to this, but given the requirements above, AND required support for IE9, I'm not sure what the best course of action is, or how to go about implementing it. Basically, I'm looking for some solid examples of either long polling in rails 3, or some kind of sockets implementation that will work for IE9 + our wonky installation/compiling requirement.
I'd appreciate any thoughts or feedback.
The IE 9 requirement rules out a WebSockets implementation.
Basically, I'm looking for some solid examples of either long polling in rails 3
An example for a long polling/polling solution is the MessageBus the Discourse forum software uses. It is a Rails app but they're not on Rails 3 any longer.
https://meta.discourse.org/t/why-does-discourse-not-use-web-sockets/18302
The "message bus" is component that allows us to easily publish information to our clients and between the rails processes in the farm.
https://meta.discourse.org/t/how-discourse-stays-online-message-bus-faye-long-polling/3238/7
Message Bus is opinionated, it only supports the protocol it needs to drive Discourse. It only supports redis for storage. Message Bus does not support web sockets. It only supports polling and long polling.
Related
I created an application that uses Watir to automate logging in and perform a couple of functions within a site.
Right now it's written 100% purely in ruby classes that I was just executing in irb, but I want to put it into a Rails application and put it online. I haven't been able to find much information about using something like Capybara or Watir for anything other than testing. Is this because of how slow they are or is it a capabilities issue?
Would I be able to run a background process that opens a browser with Watir and performs a few functions for each user in production?
Another question I have is how to keep the session over a longer period of time. There are two sites that require 2FA that my app logs into. If I wanted to log in and perform a function once an hour with a Watir browser, I could create it as a background process (if that works). But when the process is done the browser closes and when the background process runs again in an hour it requires 2FA again.
My other worry is speed. If I have 50 users that all need to run a Watir browser at the same time I imagine that will be slow. I am not worried as much about speed as long as they run and collect the data and perform the few actions we need, but how it will effect the applications integrity.
WATIR is specifically designed as a testing tool.. WATIR stands for Web Application Testing In Ruby. It's design is centered around interacting with a browser the way a user would, effectively simulating the same actions a user would take when using a site. That it would be sub-optimal for other tasks is to be expected. Due to scraping and testing having very similar activities, there a a number of people that use watir for that task, but it is not designed for that purpose, and it is unlikely that the WATIR developers would ever add features specific to data scraping verses testing.
With the things you are contemplating, you should ask yourself if you are doing the equivalent of using a socket wrench as a hammer, and if there might be a better tool you could use.
If the sites you are interacting with support an API, then that would be the preferred way to interact with them, to get information from the site. If that is not supported, you may want to consider looking at other gems that would let you request the site HTML or parse the HTML directly (e.g. Nokogiri)
You should also inspect the terms of service for the sites you are interacting with (if you don't own them) to ensure that there are not prohibitions against using 'robots' or other automated means to access the site. If so, then using Watir in the way you propose may end up getting you banned from access to the site if the pattern of your access is obviously the result of an automated process.
This is actually done more often than people think. Maybe not specifically with Watir, but running a browser automation task in a job. The jobs should be queued and run asynchronously, preferably in a different process than your main web app.
I wrote about this strategy here: https://blogstephenarifin.wordpress.com/2018/08/23/integrating-a-third-party-without-an-api-using-rails-5-and-heroku/
If you find yourself having to use Watir then the best way is to use it to render the page (for example in headless mode for javascript), save it and then use Nokogiri to process it. Apis are suggested a lot by people who don't and can't find a use for scraping but at times is necessary and perfectly legit (you may even be scraping your own data). Apis are not a universal option.
Second you should probably regulate its use to a background job. If you have end users (and you really shouldn't have many simultaneous users) many services inform customers data will be available in a few hours to a few days
we are currently developing a chat (like facebook, with stored messages).
at the moment, theres a minimum of 500 online users (its a dating website) and at the peak there is a max of 3000 users simultanously online.
switching to websockets is "the thing" for us, but while using the gem "websocket-rails" we fear a little the performance.reading articles like https://www.igvita.com/2008/11/13/concurrency-is-a-myth-in-ruby/ is causing some doubts.
so our question is:
does websocket-rails is killing our application or not? the other choice would be running a jsnode server and switch to faye which shouldnt be a problem in our scalabitliy. does somebody is having any expereince with the scalability of websocket-rails?
The GIL is still here, but should not be the main issue. The main problem is that the Rails approach does not fit well in a massive chat approach.
My suggestion is to switch to event machine for this specific part, and still use websocket (or others push mechanism, like pusher), and use this kind of WebSocket EventMachine Client.
You will be then event driven, with a single ruby thread and you still can use all the others rails existing libraries (that's the node.js model)
I'm writing a simple chat room application in Rails 3.1 - for learning purposes.
For starters I have all the needed models (messages, users, rooms, etc.) and things work great.
The clients poll the server every minute (for example) and get new messages if they have any.
I would like to change the simple polling to long polling and can't figure out if this can be done in the same app or do I have to create some other Push server for the long polling.
I read a lot about EventMachine and changed my rails app to user it as I wanted to use EventMachine for the event driven mechanics. I thought that the EventMachine channel would come in handy for this.
A client would connect and wait for a message in the chat room and it will receive a message only when one was sent to the room.
What I can't figure out is how can I share the EventMachine::Channel instance between all my client connections.
Is this approach even possible or am I going at it the wrong way?
If possible I would like a solution that can run as a single rails application hosted on Heroku.
Yeah sure. I just wrote a demo using event machine. My case is that player walking around a map, and other players should be able to see it.
The demo looks like that:
A client establishes a connection, reporting its own coordinate(generated randomly)
There is an array preserving all the coordinates for each client
When a client moves, it sends its new coordinate to the server. Then the server finds out people near him(from the array), and push the new coordinate to those clients.
I tested it with nearly 5000 clients, and each second 20-30 players moves its position. And the server process only takes less that 100M memory & 50%-60% cpu usage(on a single core).
In your case, I think you should probably try faye too. It's based on event machine, and an appropriate solution to things like chat room.
Expanding what I've mentioned on the comment, check this blog post that explains how to create a text based chat app using EM, and uses AMQP to broadcast the messages to the other users.
I think you can probably do the same or use some in memory queues to share messages, and this definitely should work on heroku, as you don't have a dependency to an external service such as RabbitMQ.
Here's a good discussion about different queue frameworks: ActiveMQ or RabbitMQ or ZeroMQ or
Rails will have streaming added in version 4.
For now, you can streaming (long polling) like in this example with Sinatra and Redis's Pub/Sub feature as a backend. You will have to add another action to handle user sent messages, adding them to Redis's with PUBLISH command. You should use an evented server like Thin or Puma.
I'm writing a Rails web service that interacts with various pieces of hardware scattered throughout the country.
When a call is made to the web service, the Rails app then attempts to contact the appropriate piece of hardware, get the needed information, and reply to the web client. The time between the client's call and the reply may be up to 10 seconds, depending upon lots of factors.
I do not want to split the web service call in two (ask for information, answer immediately with a pending reply, then force another api call to get the actual results).
I basically see two options. Either run JRuby and use multithreading or else run several regular Ruby instances and hope that not many people try to use the service at a time. JRuby seems like the much better solution, but it still doesn't seem to be mainstream and have out of the box support at Heroku and EngineYard. The multiple instance solution seems like a total kludge.
1) Am I right about my two options? Is there a better one I'm missing?
2) Is there an easy deployment option for JRuby?
I do not want to split the web service call in two (ask for information, answer immediately with a pending reply, then force another api call to get the actual results).
From an engineering perspective, this seems like it would be the best alternative.
Why don't you want to do it?
There's a third option: If you host your Rails app with Passenger and enable global queueing, you can do this transparently. I have some actions that take several minutes, with no issues (caveat: some browsers may time out, but that may not be a concern for you).
If you're worried about browser timeout, or you cannot control the deployment environment, you may want to process it in the background:
User requests data
You enter request into a queue
Your web service returns a "ticket" identifier to check the progress
A background process processes the jobs in the queue
The user polls back, referencing the "ticket" id
As far as hosting in JRuby, I've deployed a couple of small internal applications using the glassfish gem, but I'm not sure how much I would trust it for customer-facing apps. Just make sure you run config.threadsafe! in production.rb. I've heard good things about Trinidad, too.
You can also run the web service call in a delayed background job so that it's not hogging up a web-server and can even be run on a separate physical box. This is also a much more scaleable approach. If you make the web call using AJAX then you can ping the server every second or two to see if your results are ready, that way your client is not held in limbo while the results are being calculated and the request does not time out.
This is my very first question in stack overflow and I am quite excited. Liked a lot the interface of this site and the buoyant community!
I am building a rails app that receives text as an email and creates a post.
I ask or your expert opinion on which is the best option to receive the mail message and process it?
To send the mail to the application:
A1: X accesses a POP/IMAP account periodically (30s cronjob?) and processes the message
A2: the message is piped from the mailserver to the application X
To process the application
B1: rails with MMS2R
B2: PHP that processes the message and sends a POST in rails
B3: PERL that processes the message and sends a POST to rails
Which combination A+B would you recommend for a big volume of mails?
Any other A or B option?
Thank you very much and good luck with all your scripting!
Any of them that doesn't then perform an HTTP POST to the web app, it's an unnecessary middle man and will only slow you down under volume. Having said that, the upside is one location for any business logic if necessary. So, weigh those two points before making a decision.
Now, for this personally, I'd probably go Perl over the others, though PHP would be a closer option. That, of course, is because I don't know Ruby. For you, though, I'd probably suggest Ruby so that your language is consistent across the application. That makes maintenance a lot easier in the long run and it probably allows for sharing of business logic with the web app without needing to specific use the web app.
if you are concerned about volume, I would just use perl + db (bypass rail app).