Right now, my program has a tweeting system. I want to be able to save the current tweet whenever there's an update, and save the last tweeted thing. Then, every 10 minutes, a rake script runs and the tweet gets sent out. This is to prevent multiple tweets. The problem is that I have no way of saving the last tweeted phrase. I've tried using something like:
MY_VARS = {}
in application.rb, and then using MY_VARS[:old] = MY_VARS[:current], but that keeps being overridden. I also tried using the caching in rails, specifically:
Rails.cache.write("current", myTweet)
But that gets overridden to nil everytime the script runs. Does anyone know how I should be doing this?
A good place is to store it at some file, since it's a worker script it's not that important to be fast, in your case
Related
I'm working on a rails based web backend, and I've ran into a bit of an issue. I'm building a crypto trading based application, which relies on knowing the exact current price of many cryptos/stocks. To do this I seem to need a websocket to update certain data, however I can't figure out how to store this data. I need to be able to write to it on every websocket update, as well as read from it when sending out data to the front end. Both of these actions seem too fast to rely on my database, so I'm wondering if there is a better option. My idea was to use a class with a class method that is set on server startup. Then read from/write to that method when needed. The class looks something like this
class CryptoSocket
def self.start
##cryptos = {
BTC: 0,
ETH: 0,
DOGE: 0
}
end
def self.value(symbol)
##cryptos[symbol]
end
end
Inside the start method is a websocket which gets opened, and on message writes to ##cryptos with the updated value of the coin. I call CryptoSocket.start when the server boots up
To get the value for a symbol I can just call CryptoSocket.value(symbol) anywhere in my app. This seemed like it was working, however I've noticed sometimes it fails telling me NameError: uninitialized class variable ##cryptos in CryptoSocket
It seems like the issue is running reload! in the rails console, or entering a binding.pry then exiting. My guess is some garbage collection is happening, but overall it's something I'd like to avoid.
Does anyone have a suggestion for a better way to go about setting this class up? Does rails have a better way to persist an object in memory? It's fine to lose it when the server shuts down, but I would like to keep access to it when the server stays up
I have a class method (placed in /app/lib/) which performs some heavy calculations and sub-http requests until a result is received.
The result isn't too dynamic, and requested by multiple users accessing a specific view in the app.
So, I want to schedule a periodic run of the method (using cron and Whenever gem), store the results somewhere in the server using JSON format and, by demand, read the results alone to the view.
How can this be achieved? what would be the correct way of doing that?
What I currently have:
def heavyMethod
response = {}
# some calculations, eventually building the response
File.open(File.expand_path('../../../tmp/cache/tests_queue.json', __FILE__), "w") do |f|
f.write(response.to_json)
end
end
and also a corresponding method to read this file.
I searched but couldn't find an example of achieving this using Rails cache convention (and not some private code that I wrote), on data which isn't related with ActiveRecord.
Thanks!
Your solution should work fine, but using Rails.cache should be cleaner and a bit faster. Rails guides provides enough information about Rails.cache and how to get it to work with memcached, let me summarize how I would use it in your case
Heavy method
def heavyMethod
response = {}
# some calculations, eventually building the response
Rails.cache.write("heavy_method_response", response)
end
Request
response = Rails.cache.fetch("heavy_method_response")
The only problem here is that when ur server starts for the first time, the cache will be empty. Also if/when memcache restarts.
One advantage is that somewhere on the flow, the data u pass in is marshalled into storage, and then unmartialled on the way out. Meaning u can pass in complex datastructures, and dont need to serialize to json manually.
Edit: memcached will clear your item if it runs out of memory. Will be very rare since its using a LRU (i think) algoritm to expire things, and I presume you will use this often.
To prevent this,
set expires_in larger than your cron period,
change your fetch code to call the heavy_method if ur fetch fails (like Rails.cache.fetch("heavy_method_response") {heavy_method}, and change heavy_method to just return the object.
Use something like redis which will not delete items.
I'm working on a project where i want to do a mysql query from time to time. The query is too long, and actually it's done when the user does a request.
I'm afraid if many users does the request, the application will be too slow to respond. So, I want to do the query and load it with the query response from time to time, and then, on a request, the action from the controller will use this variable, instead of doing the query again and again.
How can I do that using Whenever?
on the schedule.rb
every 5.minutes do
runner "variable = Model.method"
end
and on the controller
def some_action
"the variable should be loaded here"
end
I agree with Damien Roche, you need to cache the results of the query. But, I don't think the example he gives is the best answer for you because you don't want for a user to wait for the query when it isn't cached, at the times when the cache is expired, even if this is a rare occurrence.
So you need to combine the periodic query with whenever, like you suggested, with a caching mechanism to store your query result, and retrieve it from the cache in your controller. since the runner is a different process, you will have to use a cache that is available to both the runner and your app. I recommend you look into Redis. it should be very simple to get it to work so that the runner runs the query and when it finishes writes a result set to the Redis cahce. The controller will then read the result set from the cache.
What I'm doing
I'm using the twitter gem (a Ruby wrapper for the Twitter API) in my app, which is run on Heroku. I use Heroku's Scheduler to periodically run caching tasks that use the twitter gem to, for example, update the list of retweets for a particular user. I'm also using delayed_job so scheduler calls a rake task, which calls a method that is 'delayed' (see scheduler.rake below). The method loops through "authentications" (for users who have authenticated twitter through my app) to update each authorized user's retweet cache in the app.
My question
What am I doing wrong? For example, since I'm using Heroku's Scheduler, is delayed_job redundant? Also, you can see I'm not catching (rescuing) any errors. So, if Twitter is unreachable, or if a user's auth token has expired, everything chokes. This is obviously dumb and terrible because if there's an error, the entire thing chokes and ends up creating a failed delayed_job, which causes ripple effects for my app. I can see this is bad, but I'm not sure what the best solution is. How/where should I be catching errors?
I'll put all my code (from the scheduler down to the method being called) for one of my cache methods. I'm really just hoping for a bulleted list (and maybe some code or pseudo-code) berating me for poor coding practice and telling me where I can improve things.
I have seen this SO question, which helps me a little with the begin/rescue block, but I could use more guidance on catching errors, and one the higher-level "is this a good way to do this?" plane.
Code
Heroku Scheduler job:
rake update_retweet_cache
scheduler.rake (in my app)
task :update_retweet_cache => :environment do
Tweet.delay.cache_retweets_for_all_auths
end
Tweet.rb, update_retweet_cache method:
def self.cache_retweets_for_all_auths
#authentications = Authentication.find_all_by_provider("twitter")
#authentications.each do |authentication|
authentication.user.twitter.retweeted_to_me(include_entities: true, count: 200).each do |tweet|
# Actually build the cache - this is good - removing to keep this short
end
end
end
User.rb, twitter method:
def twitter
authentication = Authentication.find_by_user_id_and_provider(self.id, "twitter")
if authentication
#twitter ||= Twitter::Client.new(:oauth_token => authentication.oauth_token, :oauth_token_secret => authentication.oauth_secret)
end
end
Note: As I was posting this, I noticed that I'm finding all "twitter" authentications in the "cache_retweets_for_all_auths" method, then calling the "User.twitter" method, which specifically limits to "twitter" authentications. This is obviously redundant, and I'll fix it.
First what is the exact error you are getting, and what do you want to happen when there is an error?
Edit:
If you just want to catch the errors and log them then the following should work.
def self.cache_retweets_for_all_auths
#authentications = Authentication.find_all_by_provider("twitter")
#authentications.each do |authentication|
being
authentication.user.twitter.retweeted_to_me(include_entities: true, count: 200).each do |tweet|
# Actually build the cache - this is good - removing to keep this short
end
rescue => e
#Either create an object where the error is log, or output it to what ever log you wish.
end
end
end
This way when it fails it will keep moving on to the next user but will still making a note of the error. Most of the time with twitter its just better to do something like this then try to do with each error on its own. I have seen so many weird things out of the twitter API, and random errors, that trying to track down every error almost always turns into a wild goose chase, though it is still good to keep track just in case.
Next for when you should use what.
You should use a scheduler when you need something to happen based on time only, delayed jobs for when its based on an user action, but the 'action' you are going to delay would take to long for a normal response. Sometimes you can just put the thing plainly in the controller also.
So in other words
The scheduler will be fine as long as the time between updates X is less then the time it will take for the update to happen, time Y.
If X < Y then you might want to look at calling the logic from the controller when each indvidual entry is accessed, isntead of trying to do them all at once. The idea being you would only update it after a certain time as passed so. You could store the last time update either on the model itself in a field like twitter_udpate_time or in a redis or memecache instance at a unquie key for the user/auth.
But if the individual update itself is still too long, then thats when you should do the above, but instead of doing the actually update, call a delayed job.
You could even set it up that it only updates or calls the delayed job after a certain number of views, to further limit stuff.
Possible Fancy Pants
Or if you want to get really fancy you could still do it as a cron job, but have a point system based on views that weights which entries should be updated. The idea being certain actions would add points to certain users, and if their points are over a certain amount you update them, and then remove their points. That way you could target the ones you think are the most important, or have the most traffic or show up in the most search results etc etc.
Next off a nick picky thing.
http://api.rubyonrails.org/classes/ActiveRecord/Batches.html
You should be using
#authentications.find_each do |authentication|
instead of
#authentications.each do |authentication|
find_each pulls in only 1000 entries at a time so if you end up with a lof of Authentications you don't end up pulling a crazy amount of entries into memory.
We recently lost a database and I want to recover the data from de Production.log.
Every request is logged like this:
Processing ChamadosController#create (for XXX.XXX.XXX.40 at 2008-07-30 11:07:30) [POST]
Session ID: 74c865cefa0fdd96b4e4422497b828f9
Parameters: {"commit"=>"Gravar", "action"=>"create", "funcionario"=>"6" ... (all other parameters go here).
But some stuff to post on de database were in the session. In the request I have the Session ID, and I also have all the session files from the server.
Is there anyway I can, from this Session ID, open de session file and get it's contents?
It's probably best to load the session file into a hash -- using the session-id as the key -- and then go through all the log files in chronological order, and parse out the relevant info for each session, and modify your database with it.
I guess you're starting out with an old database backup? Make sure to do this in a separate Rails environment -- e.g. don't do this in production; create and use a separate "recovery" environment / DB.
think about some sanity checks you can run on the database afterwards, to make sure that the state of the records makes sense
Going forward:
make sure that you do regular backups going forward (e.g. with mysqldump if you use MySQL).
make sure to set up your database for master/slave replication
hope this helps -- good luck!
Have you tried using Marshal#load? I'm not sure how you're generating those session files, but it's quite possible Rails just uses Marshal.
A client exactly had the same problem a few weeks ago. I came up with the following solution:
play back the latest backup you have (in our case it was one year
old)
write a small parser that moves all the requests from production in a temporary database (i chose mongodb for that): i used a rake task and "eval" to create the hash.
play back the data in the following order
play in the first create of an object, if it does not already exist.
find the last update (by date) and play it back.
here is the regex for scanning the production.log:
file = File.open("location_of_your_production.log", "rb")
contents = file.read
contents.scan(/(Started POST \"(.*?)\" for (.*?) at (.*?)\n.*?Parameters: \{(.*?)\}\n.*?Completed (.*?) in (.*?)ms)/m).each do |x|
# now you can collect all the important data.
# do the same for GET requests as well, if you need it.
end
In my case, the temporary database speeded up the process of the logfile parsing, so the above noted steps could be taken. Of course, everything that was not sent over production.log will be lost. Also, updates of the objects would send the whole information, it might be different in your case. I could also recreate the image uploads, since the images were sent base64 encoded in the production.log.
good luck!