For about a year now, I've been running a social iOS app on a parse server with thousands of users. All my data are stored in a MongoDB. The thing is, the structure of my database has to be improved. For example, the current "Following" system has to be changed in a way that it is gonna be completely different.
My question is, what is the best practise of making that change, considering that not all of my users are going to do the update? How can I be sure that there will be no confusion with the data between the old and the new users?
Edit:
Current Following system:
Here, every time I follow someone, I find the "user" and I append my userId, to his "followers" array...
|objectId <String>| -- |createdAt <Date>| -- |user <Pointer _User>| -- |followers <Array>|
New Following system:
Here, every time I follow someone, I create a new object, where "following" is the current user and "follower", is the user I just followed...
|objectId <String>| -- |createdAt <Date>| -- |following <Pointer _User>| -- |follower <Pointer _User>|
Related
I have websocket price data streaming in to my rails api app which I want to keep updated so any api requests get an updated response. It would be too expensive to save each update to the database. How can I do this? In Ember I can modify the model and it persists. It doesn't seem to happen in rails.
Channel controller:
def receive(message)
#ActionCable.server.broadcast('channel', message)
platform = Platform.find(params[:id]);
market = platform.markets.find_by market_name: message["market_name"]
market.attributes = {
market.price = message.values["price"],
etc......
}
#market.save [this is too expensive every time]
end
Am I going about this in the right way? It also seems inefficient to use find every time I want to update which could be multiple times per second. In Ember I created a record Id lookup array so I could quickly match the market_name, I don't see how to do this in rails.
Persistence to some store is the only way you can have other threads respond with latest value.
Instead of 3 queries( 2 selects and 1 update) you can do it with just 1 update
Market.where(platform_id: params[:id], market_name: message["market_name"]).
update_all(price: message.values["price"])
With proper index, you might have a sub-ms performance for each update.
Depending on your business need:
If you are getting tons of updates for a market every second(making all prior stale and useless), you can choose to ignore few and not fire update at all.
I am attempting to get all the orders from a magento instance. Once a day we grab all the orders.. (sometimes a few thousand)
Extra stuff that's more why I ask:
I'm using ruby-on-rails to grab the orders. This involves sending the soap call to the magento instance. It's easy as.
Once I have the response, I convert it into a Hash (a tree) and then pick out the increment id's of the orders and proceed to call getOrder with the increment id.
I have two problems with what's going on now, one operational, and one religious.
Grabbing the XML response to the list request takes really really long and when you tack on the work involved in converting the XML to a hash, I'm seeing a really slow processes.
The religious bit is that I just want the increment_ids so why do I have to pay for the processing/bandwidth to support a hugely bloated response.
Ok so the question...
Is there a way to set the response returned from Magento, to include only specific fields? Only the updated_at and the increment_id for instance.
If not, is there another call I'm not aware of, that can get just the increment_ids and date?
Edit
Below is an example of what I'm looking for from magento but it's for ebay. I send this xml up to ebay, and get back a really really specific bit of info about the product. It works for orders and such too. I can say "only this" and get just that. I want the same from Magento
<GetItemRequest xmlns="urn:ebay:apis:eBLBaseComponents">
<SKU>b123-332</SKU><OutputSelector>ItemId</OutputSelector>
</GetItemRequest>
I've created a rubygem that gives you your salesOrderList response in the form of a hash, and you can do what you want with the orders after you've received them back (i.e. select the fields you want including increment_id). Just run
gem install magento_api_wrapper
To do what you want to do, you would do something like this:
api = MagentoApiWrapper::Sales.new(magento_url: "yourmagentostore.com/index.php", magento_username: "soap_api_username", magento_api_key: "userkey123")
orders = api.order_list(simple_filters: [{key: "status" value: "complete"}])
orders.map {|o| [o.increment_id, o.items.first.sku] }
Rough guess, but you get the idea. You would get the array of hashes back and you can do what you want with them after that. Good luck!
I am writing an app which will sit between a vendors proprietary inventory management system and their Shopify shop. The app will periodically update Shopify from new data generated by the inventory management system. It will also provide end-points for Shopify webhooks.
I am currently doing something like this (pseudo-ruby with much stuff omitted):
def update_product_with_proxy(product_proxy)
product_proxy.variant_proxies.dirty.each do |variant_proxy|
update_variant_with_proxy(variant_proxy)
end
if product_proxy.dirty_proxy
shopify_product = ShopifyAPI::Product.find(product_proxy.shopify_id)
shopify_product.update_attributes({some attributes here})
end
end
Elsewhere:
def update_variant_with_proxy(variant_proxy)
shopify_variant = ShopifyAPI::Variant.find(variant_proxy.shopify_id)
shopify_variant.update_attributes({some attributes here})
end
This seems terribly inefficient as I have to fetch each updated ShopifyAPI::Product and ShopifyAPI::Variant before I can update them (I have their id's cached locally). It takes about 25 minutes for an update cycle updating 24 products each with 16 variants. Rails spends less than 2 seconds updating my product/variant proxies. The other 99% of the time is spent talking to Shopify. I must be doing something wrong.
Given that I know the id of the remote object is there a way to updated it directly without having to fetch it first?
cheers,
-tomek
First things first: You can update variants through their parent product. Once you've grabbed the product it'll have the variant info with it so you can edit them, save, and the changes will be persisted in a single API call. That'll save you some time.
Second: You can create an object locally using the gem, give it an id, and then call save to initiate the PUT request without first fetching the object from Shopify. Something like this should do the trick:
product = ShopifyAPI::Product.new(:id => 1, :title => "My new title")
product.save
Putting those two things together should give you what you want: The ability to update a product's variants in a single API call.
Note: For future reference, the shopify_api gem is built on Active Resource, so anything you can do with that library you can do with the gem.
I'm a new to ejabberd but the first thing I noticed is the completely absence of documentation and code comments.
I have many doubts, but the main are:
inside the record jid what is the difference between user and luser, server and lserver, ... and ...?
-record(jid, {user, server, resource,
luser, lserver, lresource}).
what is useful for the record iq?
-record(iq, {id = "",
type,
xmlns = "",
lang = "",
sub_el}).
what is a subscription inside ejabber? a relation between two users?
what is the jid inside the roster?
I know that these questions can be also quite stupid, but I don't really know how to understand without asking, thanks
what is the difference between user and luser?
luser,lserver and lresource are the corresponding parts of the jid after being processed with the appropiate stringprep profile. See https://www.rfc-editor.org/rfc/rfc3920#section-3 . In short, inside ejabberd you will most likely always use the processed versions, and the raw ones only when serializing the JID back to the wire.
what is useful for the record iq?
it make it easier to match on the IQ namespace, id or type (get|set|error) than to retrieve that info from the underling xml each time.
what is a subscription inside ejabber? a relation between two users?
basically, yes. A subscription from user A to user B means A is interested in B presence. But the subscription can be in different states (as the other user has to accept it, etc.). See http://xmpp.org/rfcs/rfc3921.html#sub .
what is the jid inside the roster?
sorry, didn't understand you on that, what do you want to know?
We recently lost a database and I want to recover the data from de Production.log.
Every request is logged like this:
Processing ChamadosController#create (for XXX.XXX.XXX.40 at 2008-07-30 11:07:30) [POST]
Session ID: 74c865cefa0fdd96b4e4422497b828f9
Parameters: {"commit"=>"Gravar", "action"=>"create", "funcionario"=>"6" ... (all other parameters go here).
But some stuff to post on de database were in the session. In the request I have the Session ID, and I also have all the session files from the server.
Is there anyway I can, from this Session ID, open de session file and get it's contents?
It's probably best to load the session file into a hash -- using the session-id as the key -- and then go through all the log files in chronological order, and parse out the relevant info for each session, and modify your database with it.
I guess you're starting out with an old database backup? Make sure to do this in a separate Rails environment -- e.g. don't do this in production; create and use a separate "recovery" environment / DB.
think about some sanity checks you can run on the database afterwards, to make sure that the state of the records makes sense
Going forward:
make sure that you do regular backups going forward (e.g. with mysqldump if you use MySQL).
make sure to set up your database for master/slave replication
hope this helps -- good luck!
Have you tried using Marshal#load? I'm not sure how you're generating those session files, but it's quite possible Rails just uses Marshal.
A client exactly had the same problem a few weeks ago. I came up with the following solution:
play back the latest backup you have (in our case it was one year
old)
write a small parser that moves all the requests from production in a temporary database (i chose mongodb for that): i used a rake task and "eval" to create the hash.
play back the data in the following order
play in the first create of an object, if it does not already exist.
find the last update (by date) and play it back.
here is the regex for scanning the production.log:
file = File.open("location_of_your_production.log", "rb")
contents = file.read
contents.scan(/(Started POST \"(.*?)\" for (.*?) at (.*?)\n.*?Parameters: \{(.*?)\}\n.*?Completed (.*?) in (.*?)ms)/m).each do |x|
# now you can collect all the important data.
# do the same for GET requests as well, if you need it.
end
In my case, the temporary database speeded up the process of the logfile parsing, so the above noted steps could be taken. Of course, everything that was not sent over production.log will be lost. Also, updates of the objects would send the whole information, it might be different in your case. I could also recreate the image uploads, since the images were sent base64 encoded in the production.log.
good luck!