I am currently constructing a RESTful web service using node.js for one of my current iPhone applications. At the moment, the system works as follows:
client makes requests to node.js server, server does appropriate computations and MySQL lookups, and returns data
client's reactor handles the response and updates the UI
One thing that I've been thinking about is the differences (in terms of performance and best practice) of making multiple API calls to my server vs one call which executes multiple join statements in the MySQL database and then returns a constructed object.
For example:
Lets say I am loading a user profile to display in the UI. A user has a profile picture, basic info, and news feed items. Using option one I would do as follows:
Make a getUser request to the server, which would do a query in the DB like this:
Select * from user join user_info on user.user_id=user_info.user_id left join user_profile_picture on user_profile_picture.user_id=user.user_id.
The server would then return a constructed user object containing the info from each table
Client waits for a response for the server and updates everything at once
Option 2 would be:
Make 3 asynchronous requests to the server:
getUser
getUserInfo
getUserProfile
Whenever any of the requests are received, the UI is updated
So given these 2 options, I am wondering which would offer better scalability.
At the moment, I am thinking of going with option 2 for these reasons:
Each of the async requests will be faster than the query in option a, therefore displaying something to the user faster
I am also integrating Memecache and I feel that the 3 separate calls will be easier for caching specific results (e.g not caching a user profile, but caching user, user_info and user_profile_picture).
Any thoughts or experiences?
I think the key question here is whether or not these API calls will always be made together. If they are, it makes more sense to setup a of a single endpoint and perform a join. However, if that is not the case then you should keep the separate.
Now, what you can do is of course use a query syntax that let's you specify whether or not a particular endpoint should give you more data and combine it with a join. This does require more input sanitation, but it might be worth it, since you could then minimize requests and still get an adaptable system.
On the server side, it's unlikely that either of your two approaches should be noticably slower than the other unless you're dealing with thousands of rows at a time
Related
I have an app that allows users to sort and filter through 30,000 items of data. Right now I make fetch requests from Redux actions to my rails API, with the queries being handled by scope methods on my rails end. My instructor is recommending that I move all my querying to my front-end for efficiency, but I'm wondering if it really will be more performant to manage a Redux state object with 30,000 objects in it, each with 50 of their own attributes.
(A couple extra notes: Right now I've only run the app locally and I'm doing the pagination server-side so it runs lightning fast, but I'm a bit nervous about when I launch it somewhere like Heroku. Also, I know that if I move my querying to the front-end I'll have more options to save the query state in the URL with react-router, but I've already sort of hacked a way around that with my existing set-up.)
Let's have a look at the pros and cons of each approach:
Querying on Front End
👍 Querying does not need another network request
👎 Network requests are slower because there is more data to send
👎 App must store much more data in memory
👎 Querying is not necessarily more efficient because the client has to do the filtering and it usually does not have the mechanisms to do so effectively (caching and indexing).
Querying on Back End
👍 Less data to send to client
👍 Querying can be quite fast if database indexes are set up properly
👍 App is more lightweight, it only holds the data it needs to display
👎 Each query will require a network request
The pros of querying on Back End heavily outweighs that on Front End. I would have to disagree with your instructor's opinion. Imagine you want to search for something on Google and Google sends all relevant results you want to your browser and does the pagination and sorting within your browser, your browser would feel extremely sluggish. With proper caching and adding database indexes to your data, network requests will not be a huge disadvantage.
Currently we are using Breeze.js and Angular to develop our applications. Due to some persistent legacy issues, we have two databases ('Kenya' and 'Rwanda') that cannot be merged at this time, but have the same schema and metadata. Most of the time, the client knows which database to hit and passes the request through the .withParameters() function or the .saveOptions() function. Sometimes we want to request the same query from both databases (for example, if we are requesting a list of all available countries), and we use a EntityManager wrapper on the client to manage this and request the same query from each database. This is implemented through a custom EFContextProvider which uses the data returned to determine the appropriate database and creates the appropriate context in CreateContext().
To further complicate things, in some instances one or the other database won't exist (these are local deployments created through filtered replication), but the client won't know this. Therefore, when querying for a list of all countries, it issues two requests and one will cause failures because the context cannot be instantiated properly.
This is easy enough to detect on the Server. What I would like to do is to detect whether the requested context is available and, if not, return a 200 response and an empty set.
I can detect this in the Breeze DBContextProvider CreateContext() method, but cannot figure out how to cause the request to fallback gracefully to a empty-set response.
Thanks
Not exactly what I was looking for, but it probably makes more sense since most of the work is being done on the client-side:
Instead of trying to change the controller, I added a getAvailableDatabases to the C# controller actions and use that to determine which of the databases I will query from the client.
I have a rails app that calls a third party API for weather.
The problem is that the API call is generally very slow and sometimes fails.
Showing the weather is not a necessity but it adds a nice bit of extra and pertinent information.
Right now I call the Wunderground API using Barometer gem in the controller which means the pages takes forever to load if the API is slow or fails.
I was hoping to move to this call to an AJAX call from the page once the page is loaded. I don't mind if the information shows but a bit delayed because as mentioned it is not hugely important.
I was just curious the best practices for making such a call? What is the Rails way?
The recommended way is to call to the API in the background (using a scheduler) and save the result in the database. Then in the controller you can get the data from the database and there won't be any delay.
I would say that you are quite correct in moving to an AJAX call from the browser- that way your page load is unaffected and it can take as long as it likes without your server having to wait on it. This is a classic case for loading the data asynchronously ( through callbacks and/or jQuery's deferredapproach ) so that everything else is available while the data loads and your users aren't waiting on some information that they might not be very interested in to start with.
In terms of keeping it Rails, your main consideration is whether you can and/or want to make the call directly from the browser to the service, or whether you want to proxy it through your application to some degree, which would save on potential cross-domain request problems. Again this is very much your decision and will depend on whether you have any API keys you need to transmit with requests and so on, but if the request can run directly from the user to the weather API then that would allow you to cut out the intermediate step on your part.
Hi i am a student doing my academic project.I need some guidance in completing my project.
My project is based on grails framework which searches for books from 3 different bookstores and gives d price from all the 3 stores.I need help in searching part.
how to direct the search for those bookstores once user types for required book.
thanks in advance
You need to give more details. By searching bookstores, do you mean searching in a database or are these like Amazon etc?
I would find out if these online bookstores have APIs, or if you have a choice, select the online bookstores that do have APIs that you can use to do your searching. For example, Amazon has a "Product Advertising API" that can be used to perform searching of its catalogue (see http://docs.amazonwebservices.com/AWSECommerceService/latest/DG). You usually have to register as an affiliate to get access these sort of things.
Once you have several online bookstores that are accessible via APIs, it is relatively easy to write some grails code to call them, and coordinate the results. APIs usually take the form of Web requests, either REST or SOAP (e.g. see Amazon - AnatomyOfaRESTRequest). Groovy's HTTPBuilder can be used to call and consume the bookstores' API web services if you can use simple REST, or I believe there are a couple of Grails plugins (e.g. REST Client builder). For SOAP, consider the Grails CXF Client Grails plugin.
You could do the searches on the APIs one by one, or if you want to get more advanced, you could try calling all 3 APIs at the same time asynchronously using the new servlet 3.0 async feature (see how to use from Grails 2.0.x: Grails Web Features - scroll to "Servlet 3.0 Async Features"). You would probably need to coordinate this via the DB, and perhaps poll through AJAX on your result page to check when results come in.
So the sequence would be as follows:
User submits search request from a form on a page to the server
Server creates and saves a DB object to track requests, kicks off API calls asynchronously (i.e. so the request is not blocked), then returns a page back to the user.
The "pending results" page is shown to user and a periodic AJAX update is used to check the progress of results.
Meanwhile your API calls are executing. When they return, hopefully with results, they update the DB object (or better, a related object) to store the results and status of the call.
Eventually all your results will be in the DB, and your periodic AJAX check to the server which is querying the results will be able to return them to the page. It could wait for all of the calls to the 3 bookstores to finish or it could update the page as and when it gets results back.
Your AJAX call updates the page to show the results to the user.
Note if your bookstore doesn't have an API, you might have to consider "web scraping" the results straight from bookstore's website. This is a bit harder and can be quite brittle since web pages obviously change frequently. I have used Geb (http://www.gebish.org/) to automate the browsing along with some simple string matching to pick out things I needed. Also remember to check terms & conditions of the website involved since sometimes scraping is specifically not allowed.
Also note that the above is a server oriented method of accomplishing this kind of thing. You could do it purely on the client (browser), calling out to the webservices using AJAX and processing via JavaScript. But I'm a server man :)
I am building a Ruby on Rails application where I need to be able to consume a REST API to fetch some data in (Atom) feed format. The REST API has a limit to number of calls made per second as well as per day. And considering the amount of traffic my application may have, I would easily be exceeding the limit.
The solution to that would be to cache the REST API response feed locally and expose a local service (Sinatra) that provides the cached feed as it is received from the REST API. And of course a sweeper would periodically refresh the cached feed.
There 2 problems here.
1) One of the REST APIs is a search API where search results are returned as an ATOM feed. The API takes in several parameters including the search query. What should be my caching strategy so that cached feed can be uniquely identified against the parameters? That is, for example, if I search for say
/search?q=Obama&page=3&per_page=25&api_version=4
and I get a feed response for these parameters. How do I cache the feed so that for the exact same parameters passed in a call some time later, the cached feed is returned and if the parameters change, a new call should be made to the REST API?
2) The other problem is regarding the sweeper. I don't want to sweep a cached feed which is rarely used. That is, search query Best burgers in Somalia would obviously be very less wanted than say Barak Obama. I do have the data of how many consumers have subscribed to the feed. The strategy here should be that given the number of subscribers to this search query, sweep the cached feeds based on how large this number is. Since the caching needs to happen in the Sinatra application, how would one go about implementing this kind of sweeping strategy? Some code will help.
I am open to any ideas here. I want these mechanisms to be very good on performance. Ideally I would want to do this without database and by pure page caching. However, I am open to possibility of trying other things.
Why would you want to replicate the REST service as a Sinatra app? You could easily just make a model inside your existing Rails app to cache the Atom feeds (storing the whole feed as a string inside for example).
a CachedFeed Model which is updated when its "updated_at" is far enough away to be renewed.
You could even use static caching for your cachedFeed Controller to reduce the strain on your system.
Having the cache inside your Rails app would greatly reduce complexity in terms of when to renew your cache or even count the requests performed against the rest api you query.
You could have model logic to distribute the calls you have to the most popular feeds. Tthe search parameter could just an attribute of your model so you can easily find and distinguish them