I am developing a Rails web shop application and I have the following system set up:
2 separate web services (very simple Rails apps with the same code but different databases)
Main Rails application which stores information from both web services.
The main application gets some information from both web services (in JSON format) and has to choose items (based on price). For testing purposes I currently take all items from both and add them to the main application's database. However, when items are being stored in the main database (with a simple .create and a hash with all parameters it seems as if it's adding on item multiple times and thus it takes a very long time.
First, what is generally a good strategy for doing this type of thing - getting data from the web services and storing it? Also, at what point do I want to ask for an update of the main database? It seems too much if it is every time a user connects.
I assume there is a key value for id in the data... if not you should define one. Most likely an auto incrementing integer ID since this is tagged as rails. Although you'll probably want a UUID (perhaps SecureRandom.uuid) since the two data sources are independent of each other, which adds significant complexity in a rails app
In that case you could use #model = Model.find_or_create_by(key_value: value) to avoid duplicates being created, and #model.update_attributes (essentially use an update action) to only modify what has changed.
Related
I have already read Rails - How do I temporarily store a rails model instance? and similar questions but I cannot find a successful answer.
Imagine I have the model Customer, which may contain a huge amount of information attached (simple attributes, data in other tables through has_many relation, etc...). I want the application's user to access all data in a single page with a single Save button on it. As the user makes changes in the data (i.e. he changes simple attributes, adds or deletes has_many items,...) I want the application to update the model, but without committing changes to the database. Only when the user clicks on Save, the model must be committed.
For achieving this I need the model to be kept by Rails between HTTP requests. Furthermore, two different users may be changing the model's data at the same time, so these temporary instances should be bound to the Rails session.
Is there any way to achieve this? Is it actually a good idea? And, if not, how can one design a web application in which changes in a model cannot be retained in the browser but in the server until the user wants to commit them?
EDIT
Based on user smallbutton.com's proposal, I wonder if serializing the model instance to a temporary file (whose path would be stored in the session hash), and then reloading it each time a new request arrives, would do the trick. Would it work in all cases? Is there any piece of information that would be lost during serialization/deserialization?
As HTTP requests are stateless you need some kind of storeage between requests. The session is the easiest way to store data between requests. As for you the session will not be enough because you need it to be accessed by multiple users.
I see two ways to achive your goal:
1) Get some fast external data storage like a key-value server (redis, or anything you prefer http://nosql-database.org/) where you put your objects via serializing/deserializing (eg. JSON).
This may be fast depending on your design choices and data model but this is the harder approach.
2) Just store your Objects in the DB as you would regularly do and get them versioned: (https://github.com/airblade/paper_trail). Then you can just store a timestamp when people hit the save-button and you can always go back to this state. This would be the easier approach i guess but may be a bit slower depending on the size of your data model changes ( but I think it'll do )
EDIT: If you need real-time collaboration between users you should probably have a look at something like Firebase
EDIT2: Anwer to your second question, whether you can put the data into a file:
Sure you can do that. But you would need some kind of locking to prevent data loss if more than one person is editing. You will need that aswell if you go for 1) but tools like redis already include locks to achive your goal (eg. redis-semaphore). Depending on your data you may need to build some logic for merging different changes of different users.
3) Another aproach that came to my mind would be doing all editing with Javascript and save it in one db-transaction. This would go well with synchronization tools like firebase (or your own synchronization via Rails streaming API)
I am looking for solution of logging data changes for public API.
There is a need to tell client app which tables form db has changed and need to be synchronised since the app synchronised last time and also need to be for specific brand and country.
Current Solution:
Version table with class_names of models which is touched from every model on create, delete, touch and save action.
When we are touching version for specific model we also look at the reflected associations and touch them too.
Version model is scoped to brand and country
REST API is responding to a request that includes last_sync_at:timestamp, brand and country
Rails look at Version with given attributes and return class_names of models which were changed since lans_sync_at timestamp.
This solution works but the problem is performance and is also hard to maintenance.
UPDATE 1:
Maybe the simple question is.
What is the best practice how to find out and tell frontend apps when and what needs to be synchronized. In terms of whole concept.
Conditions:
Front end apps needs to download only their own content changes not whole dataset.
Does not invoked synchronization when application from different country or brand needs to be synchronized.
Thank you.
I think that the best solution would be to use redis (or some other key-value store) and save your information there. Writing to redis is much faster than any sql db. You can write some service class that would save the data like:
RegisterTableUpdate.set(table_name, country_id, brand_id, timestamp)
Such call would save given timestamp under key that could look like i.e. table-update-1-1-users, where first number is country id, second number is brand id, followed by table name (or you could use country and brand names if needed). If you would like to find out which tables have changed you would just need to find redis keys with query "table-update-1-1-*", iterate through them and check which are newer than timestamp sent through api.
It is worth to rmember that redis is not as reliable as sql databases. Its reliability depends on configuration so you might want to read redis guidelines and decide if you would like to go for it.
You can take advantage of the fact that ActiveModel automatically logs every time it updates a table row (the 'Updated at' column)
When checking what needs to be updated, select the objects you are interested in and compare their 'Updated at' with the timestamp from the client app
The advantage of this approach is that you don't need to keep an additional table that lists all the updates on models, which should speed things up for the API users and be easier to maintain.
The disadvantage is that you cannot see the changes in data over time, you only know that a change occurred and you can access the latest version. If you need to track changes in data over time efficiently, than I'm afraid you'll have to rework things from the top.
(read last part - this is what you are interested in)
I would recommend that you use the decorator design pattern for changing the client queries. So the client sends a query of what he wants and the server decides what to give him based on the client's last update.
so:
the client sends a query that includes the time it last synched
the server sees the query and takes into account the client's nature (device-country)
the server decorates (changes accordingly) the query to request from the DB only the relevant data, and if that is not possible:
after the data are returned from the database manager they are trimmed to be relevant to where they are going
returns to the client all the new stuff that the client cares about.
I assume that you have a time entered field on your DB entries.
In that case the "decoration" of the query (abstractly) would be just to add something like a "WHERE" clause in your query and state you want data entered after the last update.
Finally, if you want that to be done for many devices/locales/whatever implement a decorator for the query and the result of the query and serve them to your clients as they should be served. (Keep in mind that in contrast with a subclassing approach you will only have to implement one decorator for each device/locale/whatever - not for all combinations!
Hope this helped!
I am trying to convert my asp.net mvc4 app, which had fairly heavy use of SessionState, into a stateless app. I understand that I can store this information in the DB, and intend to do so.
My question, though, is about my particular architecture. My app has a main 'page' consisting of a number of partial view panels, which each have actions in them that can affect the other panels. What i've been doing up to now is storing the entire state of the viewModel (lots of inter-related EF list collections and 'record' objects) in the session, and its been working great. Except when the session just randomly dies.
So, I need to get this data out of the session, and into the DB where I can rebuild the thing at need. My concern is that, if I store the info in the database, every single action done on screen might affect 3-5 different panels, each with their own State updates, thats a minimum of 10 round trips to the DB for every interaction!
What are some strategies I can use to make this idea more scalable?
EXTRA INFO
The view in question here is a sort of POS shopping cart system. There are panels for selecting events, selecting/adding items to the cart, editing cart items, selecting contacts, editing contacts, displaying the cart items, displaying the cart 'subtotals', and finally, a panel with a [checkout] button.
Selecting a new event will change the list of available items. Selecting an item to add to the cart will change the cart item list, subtotals, as well as the checkout panel. Same for editing a cart item.
The main concern is how to recover from a lost session, as I've found the built-in asp.net session code too unreliable. My testers have encountered issues with sessions timing out, and then my app not having any kind of recovery process. When its installed on 1500 sites, each with an average of 10 users, its going to be a plague of lost session issues, and I need to combat that before it becomes a real problem.
I agree that I'm not going stateless...wrong choice of words used in a rush. I'm just trying to move that state into a form that I can rely on past the session failure. My main idea presently is to continue using the session as the local cache for the viewModel data, but to have a fallback operation that can rebuild the viewModel from DB if the session one is lost somehow.
You shouldn't necessarily be using a database to store (what sounds like) data that only needs to be persisted in the short term.
If these changes to the other partials are only relevant in the context of the current "master view," then I would suggest using jQuery AJAX to send off the requests, parse the response JSON and update the other views. Tutorials on jQuery AJAX and ASP.NET MVC are easy to find, if you don't already have the knowledge:
http://www.codeproject.com/Articles/41828/JQuery-AJAX-with-ASP-NET-MVC
This way, you don't need to make a bunch of round trips. If the changes need to be persisted beyond the context of the current view, make ONE round trip to the database to perform the update and then simply update all of the other partials from the in-memory response from the AJAX call.
You don't need to read from secondary storage multiple times when you already have all of the information you need in-memory. Just do the reading and writing once.
I decided to go with a hybrid approach. I'm still using session, but I'm building out a DB 'recovery' option, so that if the session portion is lost, the DB will be able to provide the values needed to rebuild the session seamlessly.
Seems to be working well, so far.
I have a system I've built in MVC 3 that currently provides a yearly submission cycle where the system proceeds through a serious of seven steps tied to dates stored in the web.config as AppSettings. However, each year, I always have to roll the system back and forth between previous steps in order to accommodate the end users. I would like to give the administrator the ability to control the system status without having to contact a developer. What is the best way to do this?
I plan to build a page with proper validation that lets the administrator set the dates. I've considered a couple options of how I should store those date, but none of them seem correct. Our entire permission system uses these dates, and various bits of text on the pages turns on and off based on what period we're currently in.
So far I've come up with two options:
Option 1: Create a database table – This was my first thought. I’ve set up properties on the MvcApplication class in the global.asax and pulled them from the database. Using a lazy loader, I can set the properties the first time they're needed. However, when they change in the database, I don't have a way to force the system to “reset” and read the date changes. If I do this action on Begin_Request(), I'm constantly opening the connection and resetting the properties for each file that the web browser opens on the server, regardless if it's static content or not.
I could directly fetch the dates from the database every time I need one of the dates, but then I'm having to redo a lot of functionality to reduce repeated database calls. I'd like to cache the dates for each request, and only pull them when I need them,
Option 2: Allow editing a config file through the application – I've looked up how to split the web.config file so I can have a separate file that just contains the appSettings. Then I could just update the new config file from a controller action. I think this would work nicely, and not require me to rewrite any of the existing functionality, but it feels like I would be introducing a bad design pattern into the code.
I'd vote for the database. For the sake of performance you can cache those parameter values in a static class inside your app and provide a method to reread them from DB in the same class. So:
When a user makes request, check if those properties are already cached. If they are - use cached values, if no - read them from DB
When administrator makes changes to those parameters - store them to database and enforce your static caching class to reread them from DB.
I would suggest an approach that doesn't care whether the settings are stored in database or key/value pairs in config file.
Since you want the settings to be accessed globally by all users you can cache the settings and the cache implementation should be generic and distributed. There are plenty of online resources available how to create such an interface.
Since you want the cache to be sync with the underlying data you have to set cache dependencies (AppFabric won't supports sql cache dependency see this thread, while NCache supports both sql and file).
I would store the values in a database and use a distributed cache to persist the data across the web farm. MS AppFabric Caching has worked well for me. You will need to implement a standard caching pattern (check the cache, if null load from db and insert into cache).I would probably just create a static Load() method that abstracts this logic away. When the admins update the db you could update the cache or just delete the cachekey.
Therr are other considerations to be added to performance. Namely if you modify the config file thr application pool is re iniyializrd, while the database solution doesnt cause application reinitialization
...so do you need to re initialize the app after the changes or not?...If there i no way to avoid the inizialization whitout drastic changmes to the application ptobably the config filr solution is better
I have an application that has different data sets depending on which company the user has currently selected (dropdown box on sidebar currently used to set a session variable).
My client has expressed a desire to have the ability to work on multiple different data sets from a single browser simultaneously. Hence, sessions no longer cut it.
Googling seems to imply get or post data along with every request is the way, which was my first guess. Is there a better/easier/rails way to achieve this?
You have a few options here, but as you point out, the session system won't work for you since it is global across all instances of the same browser.
The standard approach is to add something to the URL that identifies the context in which to execute. This could be as simple as a prefix like /companyx/users instead of /users where you're fetching the company slug and using that as a scope. Generally you do this by having a controller base class that does this work for you, then inherit from that for all other controllers that will be affected the same way.
Another approach is to move the company identifying component from the URL to the host name. This is common amongst software-as-a-service providers because it makes sharding your application much easier. Instead of myapp.com/companyx/users you'd have companyx.myapp.com/users. This has the advantage of preserving the existing URL structure, and when you have large amounts of data, you can partition your app by customer into different databases without a lot of headache.
The answer you found with tagging all the URLs using a GET token or a POST field is not going to work very well. For one, it's messy, and secondly, a site with every link being a POST is very annoying to work with as it makes navigating with the back-button or forcing a reload troublesome. The reason it has seen use is because out of the box PHP and ASP do not have support routes, so people have had to make do.
You can create a temporary database table, or use a key-value database and store all data you need in it. The uniq key can be used as a window id. Furthermore, you have to add this window id to each link. So you can receive the corresponding data for each browser tab out of the database and store it in the session, object,...
If you have an object, lets say #data, you can store it in the database using Marshal.dump and get it back with Marshal.load.