Firstly let me mention that I'm new to web-frameworks.
I have to write my first web-app for a Uni project. I spent two weeks learning Grails and Django. Started working with Rails yesterday and loved it. So I've decided to go with it and discard my work in the other frameworks.
About the app
It's supposed to be a Twitter app that utilizes Twitter's Streaming API to record tweets which match a set of specified filters. (I'm going to use the Tweetstream gem which takes care of connecting to Twitter and capturing matching tweets).
The app's web interface should have the following functionality -
Creating new requests The user inputs a set of filter parameters (keywords to track) & URL/username/password of an existing PostgreSQL or MySQL database.
When a request is created, the web-app spawns a background ruby process. This process connects to Twitter via the Tweetstream gem. It also connects to the database specified by the user to stores received tweets.
View/terminate of existing requests
The user should be able to see a list of requests that are running as background processes by visiting a URL such as /listRequests.
See further details about a process/terminate the process
The user should be able to go to URL such as /requests/1/detail to view some details (e.g how long request has been running, number of tweets captured, etc). The user should also be able to terminate the process.
My inexperience is showing as I'm unable to comprehend -
what my models should be (maybe Request should be a model. Tweet doesn't need to be a model as it's not being stored locally)
how I'm going to connect to remote databases.
how I can create background processes (backgroundrb??) and associate them with request objects so that I can terminate then when the user asks.
At the end of the day, I've got to build this myself, so I'm not asking for you to design this for me. But some pointers in the right direction would be extremely helpful and appreciated!
Thanks!
Hmm.
Since the web app is just a thin wrapper around the heavy-lifting processes, it might be more appropriate to just use something like Sinatra here. Rails is a big framework that pulls in lots of stuff that you won't need for this project, even though it will work.
Does the "background process" requirement here strictly mean a separate process, or does it just mean concurrency? TweetStream uses the EventMachine gem to handle updates as they come, which uses a separate thread for each connection. It would be quite possible to spawn the TweetStream clients from a simple Sinatra web app, keep them in a big array, have them all run concurrently with no trouble, and simply run stop on a given client when you want it to stop. No need for a database or anything.
I'm not sure exactly what your prof is looking for you to do here, but MVC doesn't really fit. It's better to work with the requirements than to mush it into a design pattern that doesn't match it :/
Even so, I <3 Rails! Definitely get on that when you're working primarily with objects being represented in a database :)
Quite a project. Most of what will be challenging is not related to rails itself, but rather the integration with background processes. backgroundrb is a bit out of fashion. The last commit on the main github project is over a year ago, so it's likely not up to snuff for Rails 3. Search around and evaluate your options. Resque is popular, but I'm not sure if your real-time needs match with its queue-based structure.
As for your app, I see only a single model, but don't call it request. That's a reserved name in rails. Perhaps a Search model, or something along that line.
Connecting to different databases is straight forward but will require direct configuration of your ActiveRecord class during operation rather than using database.yml.
Related
Background
I have a fairly typical Rails application, which uses Devise for authentication management. While building this app, I realized that realtime chat would be a great feature to have. Ideally, of course, this would make use of Websockets, in order to reduce the polling load on the server (as well as making it marginally easier to implement, as you don't have to manage polling).
I realized quickly that Ruby isn't really a great fit for having a large number of concurrent connections open at one time. Phoenix, however, is written in Elixir, so I can make use of the Erlang VM, which is quite good at long connections. It also seems like it could be greatly beneficial if all the chat data was stored separate from the main application database, which should also reduce load in the future.
The Problem
I want to be able to make this separation completely invisible to the user. They visit www.example.com/chat, and it loads all the relevant data in from chat.example.com and starts up the websockets, without requiring them to login to a separate service. I think using an <iframe> is probably the way to go about doing this.
My problem is sharing authentication and data between the two applications. The Rails app needs to be able to create conversations on the Phoenix app in response to certain events. The Phoenix app needs to know what user is currently authenticated into Rails, as well as general data about the user.
An OAuth flow with the Rails app as the ID provider seemed like a good fit at first, but I can't figure out a way for the Phoenix app to automatically be granted access. I also have some concerns about user records existing inside the Phoenix app—it should be aware of all users on the main application, so you can start a chat with a user even if they haven't ever opened chat.
What would be the best way to go about doing this? My intuition says that this is going to involve window.postMessage and some kind of token system, but I wanted to ask what the generally accepted way of doing this was before I accidentally created an insecure mess.
Sharing the session isn't too hard, assuming you are running at least Rails 4.1 and using JSON serialization (default for apps created with >=4.1). A quick google search finds PlugRailsCookieSessionStore, which accomplishes this.
For more information on what it takes to share a session between Rails and another language, Matt Aimonetti has an excellent blog post with detailed information.
Lastly, if you would prefer to stay entirely in Ruby, it's definitely doable. Ryan Stout discusses scalability around persistent connections in the FAQ for Volt, which uses a persistent connection for every user. The article he links is also a great read. Just mentioning it to help you weigh the trade off of building a separate app in another language.
My friend has setting up a database for a Ragnarok Online server, and he wants me to code the relative website, which is going to use some of that data (and obviously, i'll have to add tables for the news system, website accounts, etc). Since i'm learning RoR i was going to do it that way.
I have a few "best practice" questions related to this :
Should I create a different database for the website, since it's going to have its particular data alongside the game data ? (i already have a few clues to link multiple databases with Rails, but that seems too much of a hassle for what it is).
If not, do i have to create Model/Controller for each of the tables composing the database, despite the fact that i'm not going to use 90% of it ? Or just the ones that i need ?
An example of this problem : the game database has its own "user" table, but i have to have another "user" table for the website, and do some Joins between those two. So, what's the best practice here ?
Uhm, best practice is not making your own user table. This will cause you much pain. Best practice? Use an API. Expose the game's database in some way to your website, and fetch that info with external requests in your web application.
The reason why making a second user table is a hassle:
1) You'll constantly have to update it, pulling data from the original
to keep it up-to-date.
And I mean furthermore, you're gonna have to create a CRON job or something pulling data from that original table to keep it up to date. Yuck. Also what if that CRON job makes a mistake? (It will)
2) It's almost inevitable that there will be inconsistencies if two
separate tables are maintained. Are you sure your web application is
really fail-proof?
Update:
What you're gonna need is essentially a second Rails application that acts as a REST API for that database. For a good idea of what REST is, I'd read through this to get you started: http://tomayko.com/writings/rest-to-my-wife
Once you have a good understanding of that, start making your app, and test if it's working by using tools like cURL to send requests to your API.
Once you have that done, I'd take a look into the Ruby rest-client gem like Nobita mentioned. This is what you're going to use from your web application to request information from your API application.
Just let me note, I think this would be a terrible first Rails project, unless you're already really well versed in other web development tools, preferably MVC frameworks.
We are planning to make a "large" website for I'd say 5000 up to many more users. We think of putting in lots of real time functionality, where data changes instantly propagate to all connected clients. New frameworks like Meteor and DerbyJS look really promising for this kind of stuff.
Now, I wonder if it is possible to do typical backend stuff like sending (bulk) emails, cleaning up the database, generating pdfs, etc. with those new frameworks. And in a way that is productive and doesn't suck. I also wonder how difficult it is to create complex forms with them. I got used to the convenient Rails view helpers and Ruby gems to handle those kind of things.
Meteor and DerbyJS are both quite new, so I do expect lots of functionality will be added in the near future. However, I also wonder if it might be a good idea to combine those frameworks with a "traditional" Rails app, that serves up certain complex pages which do not need realtime updates. And/or with a Rails or Sinatra app that provides an API to do the heavy backend processing. Those Rails apps could then access the same databases then the Meteor/DerbyJS app. Anyone thinks this is a good idea? Or rather not? Why?
It would be nice if anyone with sufficient experience with those new "single page app realtime" frameworks could comment on this. Where are they heading towards? Will they be able to handle "complete" web apps with authentication and backend processing? Will it be as productive/convenient to program with them as with Rails? Well, I guess no one can know that for sure yet ;-) Well, any thoughts, guesses and ideas are welcome!
For things like sending bulk emails and generating PDFs, Derby let's you simply use normal Node.js modules. npm now has over 10,000 packages, so there are packages for most things you might want to do on the server. Derby doesn't control your server, and it works on top of any normal Express server. You should probably stick with Node.js code as much as possible and not use Rails along with Derby. That is not to say that you can't send messages to a separate Rails app, but since you already have to have a Node.js app running to host Derby, you might as well use it for stuff like this.
To communicate with such server-side code, you can use Derby's model events. We are still exploring how this kind of code works and we don't have a lot of examples, but it is something that we will have a clear story around. We are building an app ourselves that communicates with an email server, so we should have some real experience with this pretty soon.
You can also just use a normal AJAX request or send a message over Socket.IO manually if you don't want to use the Derby model to do this kind of communication. You are free to make your own server-side only routes with Express along with your Derby app routes. We think it is nice to have this kind of flexibility in case there are any use cases that we didn't properly anticipate with the framework.
As far as creating forms goes, Derby has a very powerful templating system, and I am working on making it a lot better still. We are working on a new UI components feature that will make it possible to build libraries of self-contained UI widgets that can simply be dropped into a Derby app while still playing nicely with automatic view-model bindings and data syncing. Once this feature is completed, I think form component libraries will be written rather quickly.
We do expect to include all of the features needed for a normal app, much like Rails does. It won't look like Rails or work like Rails, but it will be similarly feature complete eventually.
For backend tasks (such as sending emails, cleaning up the database, generating pdfs) it's better to use resque or sidekiq
Now, I wonder if it is possible to do typical backend stuff like
sending (bulk) emails, cleaning up the database, generating pdfs, etc.
with those new frameworks. And in a way that is productive and doesn't
suck. I also wonder how difficult it is to create complex forms with
them. I got used to the convenient Rails view helpers and Ruby gems to
handle those kind of things.
Also, my question is not only about background jobs, but also about stuff one can might do during a request, like generating a pdf, or simply rendering complex views with rails helpers or code from gems. –
You're mixing metaphors here - a single page app is just a site where the content is loaded without doing a full page reload, be that a front end in pure js or you could use normal html and pjax.
The kind of things you are describing would be done in a background task regardless of the fornt-end framework you used. But +1 for sidekiq if you're using ruby.
As for notifying all the other users of things that have changed, you can look into using http://pusher.com or http://pubnub.com if you don't want to maintain a websocket server.
So, this may be a kind of dumb question, but I checked the Google and got no hits. We want to host multiple Rails apps in a way that makes them look homogeneous. We want all the apps to have the same look and feel, and all the apps to use the same sign-on database.
Theming I think we could accomplish by just putting the site theme into a gem, and requiring that gem from our github repository in each app. However, auth is trickier.
I know that I can achieve this "for free" by just not making the different portions of the site (store, chat forums, etc.) different apps. If they're all, say, Rails Engines, we can basically drop them into the same application with their own namespaced routes, and have a single plugin that does auth.
However, for various reasons we'd like to keep these separate apps, if that's technically possible. The number one reason is scalability; since this will be a hosted site, we want the flexibility to spin up more instances of, say, the store (perhaps to handle a holiday sale rush), without needing to spin up the chat forums. Also, we want to be able to completely isolate the portions of the code that AREN'T intertwined.
Ideally, the databases would be separate too (keeping us from falling back into the rut of "put everything including the kitchen sink in the db"), but I do know that one "cheap" way to do cross-app auth is just to use the same plugin (say, Devise), and just point to the same DB.
So, I'm thinking that maybe the way to do this is to auth via a web service call. Is this prior art -- does anyone have a gem for this that "just works" so that authentication can be shared across all apps? Or am I just entering into a world of pain by trying to build things this way?
Thanks in advance!
You could do a single sign on approach described at:
http://blog.joshsoftware.com/2010/12/16/multiple-applications-with-devise-omniauth-and-single-sign-on/
The single sign on approach with oauth and devise has some drawbacks. The main problem I had was I was unable to extend the timeout time across multiple apps.
I'm going to be collaborating with a Python developer on a web
application. I'm going to be building a part of it in Ruby and he is
going to build another part of it using Django. I don't know much about
Django.
My plan for integrating the two parts is to simply map a certain URL
path prefix (say, any request that begins with /services) to the Python
code, while leaving Rails to process other requests.
The Python and Ruby parts of the app will share and make updates to the
same MySQL datastore.
My questions:
What do people think generally of this sort of integration strategy?
Is there a better alternative (short of writing it all in one language)?
What's the best way to share sensitive session data (i.e. a logged in
user's id) across the two parts of the app?
As I see it you can't use Django's auth, you can't use Django's ORM, you can't use Django's admin, you can't use Django's sessions - all you are left with is URL mapping to views and the template system. I'd not use Django, but a simpler Python framework. Time your Python programmer expanded his world...
One possible way that should be pretty clean is to decide which one of the apps is the "main" one and have the other one communicate with it over a well-defined API, rather than directly interacting with the underlying database.
If you're doing it right, you're already building your Rails application with a RESTful API. The Django app could act as a REST client to it.
I'm sure it could work the other way around too (with the rest-client gem, for instance).
That way, things like validations and other core business logic are enforced in one place, rather than two.
A project, product, whatever you call it, needs a leader.
This is the first proof that you don't have one. Someone should decide either you're doing ruby or python. I prefer ruby myself, but I understand those who prefer python.
I think starting a product asking yourself those kind of questions is a BAD start.
If your colleague only knows prototype, and you only know JQuery, are you going to mix the technologies too? Same for DB? And for testing frameworks?
This is a never ending arguing subject. One should decide, IMHO, if you want so;ething good to happen. I work with a lot of teams, as a consultant, Agile teams, very mature teams for some of them, and that's the kind of stuff they avoid at all cost.
Except if one of you is going to work on some specific part of the project, which REALLY needs one or other of the technologies, but still think the other one is best for the rest of the application.
I think, for example, at a batch computing. You have ALL your web app in ror or django, and you have a script, called by CRON or whatever, computing huge amounts of data outside the web app, filling a DB or whatever.
My2Cts.