So I have this currency .xml file:
http://www.ecb.int/stats/eurofxref/eurofxref-daily.xml
Now, I am wondering, how can I make my rails application read it? Where do I even have to put it and how do I include it?
I am basically making a currency exchange rate calculator.
And I am going to make the dropdown menu have the currency names from the .xml table appear in it and be usable.
First of all you're going to have to be able to read the file--I assume you want the very latest from that site, so you'll be making an HTTP request (otherwise, just store the file anywhere in your app and read it with File.read with a relative path). Here I use Net::HTTP, but you could use HTTParty or whatever you prefer.
It looks like it changes on a daily basis, so maybe you'll only want to make one HTTP request every day and cache the file somewhere along with a timestamp.
Let's say you have a directory in your application called rates where we store the cached xml files, the heart of the functionality could look like this (kind of clunky but I want the behaviour to be obvious):
def get_rates
today_path = Rails.root.join 'rates', "#{Date.today.to_s}.xml"
xml_content = if File.exists? today_path
# Read it from local storage
File.read today_path
else
# Go get it and store it!
xml = Net::HTTP.get URI 'http://www.ecb.europa.eu/stats/eurofxref/eurofxref-daily.xml'
File.write today_path, xml
xml
end
# Now convert that XML to a hash. Lots of ways to do this, but this is very simple xml.
currency_list = Hash.from_xml(xml_content)["Envelope"]["Cube"]["Cube"]["Cube"]
# Now currency_list is an Array of hashes e.g. [{"currency"=>"USD", "rate"=>"1.3784"}, ...]
# Let's say you want a single hash like "USD" => "1.3784", you could do a conversion like this
Hash[currency_list.map &:values]
end
The important part there is Hash.from_xml. Where you have XML that is essentially key/value pairs, this is your friend. For anything more complicated you will want to look for an XML library like Nokogiri. The ["Envelope"]["Cube"]["Cube"]["Cube"] is digging through the hash to get to the important part.
Now, you can see how sensitive this will be to any changes in the XML structure, and you should make the endpoint configurable, and that hash is probably small enough to cache up in memory, but this is the basic idea.
To get your list of currencies out of the hash just say get_rates.keys.
As long as you understand what's going on, you can make that smaller:
def get_rates
today_path = Rails.root.join 'rates', "#{Date.today.to_s}.xml"
Hash[Hash.from_xml(if File.exists? today_path
File.read today_path
else
xml = Net::HTTP.get URI 'http://www.ecb.europa.eu/stats/eurofxref/eurofxref-daily.xml'
File.write today_path, xml
xml
end)["Envelope"]["Cube"]["Cube"]["Cube"].map &:values]
end
If you do choose to cache the xml you will probably want to automatically clear out old versions of the cached XML file, too. If you want to cache other conversion lists consider a naming scheme derived automatically from the URI, e.g. eurofxref-daily-2013-10-28.xml.
Edit: let's say you want to cache the converted xml in memory--why not!
module CurrencyRetrieval
def get_rates
if defined?(##rates_retrieved) && (##rates_retrieved == Date.today)
##rates
else
##rates_retrieved = Date.today
##rates = Hash[Hash.from_xml(Net::HTTP.get URI 'http://www.ecb.europa.eu/stats/eurofxref/eurofxref-daily.xml')["Envelope"]["Cube"]["Cube"]["Cube"].map &:values]
end
end
end
Now just include CurrencyRetrieval wherever you need it and you're golden. ##rates and ##rates_retrieved will be stored as class variables in whatever class you include this module within. You must test that this persists between calls in your production setup (otherwise fall back to the file-based approach or store those values elsewhere).
Note, if the XML structure changes, or the XML is unavailable today, you'll want to invalidate ##rates and handle exceptions in some nice way...better safe than sorry.
Related
What would be the best and more efficient way in Rails if I want to use a hash of about 300-500 integers (but it will never be modified) and use it in more than one view in the application?
Should I save the data in the database?, create the hash in each action that is used? (this is what I do now, but the code looks ugly and inefficient), or is there another option?
Why don't you put it in a constant? You said it will never change, so it fits either configuration or constant.
Using the cache has the downside that it can be dropped out of cache, triggering a reload, which seems quite useless in this case.
The overhead of having it always in memory is none, 500 integers are 4KB or something like that at most, you are safe.
You can write the hash manually or load a YAML file (or whatever) if you prefer, your choice.
My suggestion is create a file app/models/whatever.rb and:
module Whatever
MY_HASH = {
1 => 241
}.freeze
end
This will be preloaded by rails on startup (in production) and kept in memory all the time.
You can access those valus in view with Whatever::MY_HASH[1], or you can write a wrapper method like
module Whatever
MY_HASH = {
1 => 241
}.freeze
def self.get(id)
MY_HASH.fetch(id)
end
end
And use that Whatever.get(1)
If the data will never be changed, why not just calculate the values before hand and write them directly into the view?
Another option would be to put the values into a singleton and cache them there.
require 'singleton'
class MyHashValues
include Singleton
def initialize
#
#results = calculation
end
def result_key_1
#results[:result_key_1]
end
def calculation
Hash.new
end
end
MyHashValues.instance.result_key_1
Cache it, it'll do exactly what you want and it's a standard Rails component. If you're not caching yet, check out the Rails docs on caching. If you use the memory store, your data will essentially be in RAM.
You will then be able to do this sort of thing
# The block contains the value to cache, if there's a miss
# Setting the value is done initially and after the cache
# expires or is cleared.
# put this in application controller and make it a helper method
def integer_hash
cache.fetch('integer_hash') { ... }
end
helper_method :integer_hash
I want to parse the JSONP data and save that data into my data base .
I have my jsonp url lets say this >>http://a0.awsstatic.com/pricing/1/ec2/pricing-data-transfer-with-regions.min.js?callback=callback&_=1409722308446
Its not normal json object so how can i parse this json in ruby/ruby on rails and save the data into my database. I want to save the data in table having filed lets say region , name ,type, price .
What are the ways to do the same.
JSONP is a work around for the same origin policy in client side JavaScript. It isn't required for processing data in Ruby. Firstly I would try to find the data available in plain JSON, without the callback.
If, however, that URL is absolutely the only one you can call for this data, then I would strip away the callback using a regular expression and then parse the plain JSON data.
If you have already loaded the contents of that file into a variable called jsonp, then something like this might work:
require 'net/http'
require 'json'
uri = URI.parse('http://a0.awsstatic.com/pricing/1/ec2/rhel-od.min.js?callback=callback&_=1409731896563')
jsonp = Net::HTTP.get(uri)
jsonp.gsub!(/^.*callback\(/, '') # removes the comment and callback function from the start of the string
jsonp.gsub!(/\);$/, '') # removes the end of the callback function
jsonp.gsub!(/(\w+):/, '"\1":')
hash = JSON.parse(jsonp)
then hash will have the parsed JSON from the response.
Please note, this code has no error handling and should be treated as a starting point for your final solution.
[edit] Added the third gsub to change the JavaScript style keys to JSON style keys. This works in this case because the keys all appear to be simple enough to fit that regex.
[edit2] Added way to load the JSONP with Net::HTTP
If what you're really trying to do is parse the Amazon price list JS file into Ruby, there is a better (read: safer--no evals) way to do it:
require 'net/http'
require 'json'
JSON.parse(
Net::HTTP.get(
URI.parse('http://a0.awsstatic.com/pricing/1/ec2/rhel-od.min.js')
).split('callback(')[1].sub(');', '').gsub(/(\w+):/, '"\1":')
)
As the data is as a raw JS object form, it's actually valid Ruby:
require 'json'
data = '{key: "value", key2: 33}'
obj = eval data
obj[:key]
#=> "value"
obj.to_json
# => "{\"key\":\"value\",\"key2\":33}"
You must trust your source completely though - this would allow the running of abstract Ruby code if the data were tampered with - which could in turn run abstract command-line terminal code, using the back-tick operators. This could delete your hard drive for example.
I'm building a bulk-file-uploader. Multiple files are uploaded in individual requests, and my UI provides progress and success/fail. Then, once all files are complete, a final request processes/finalizes them. For this to work, I need to create many temporary files that live longer than a single request. Of course I also need to guarantee filenames are unique across app instances.
Normally I would use Tempfile for easy unique filenames, but in this case it won't work because the files need to stick around until another request comes in to further process them. Tempfile auto-unlinks files when they're closed and garbage collected.
An earlier question here suggests using Dir::Tmpname.make_tmpname but this seems to be undocumented and I don't see how it is thread/multiprocess safe. Is it guaranteed to be so?
In c I would open the file O_EXCL which will fail if the file exists. I could then keep trying until I successfully get a handle on a file with a truly unique name. But ruby's File.open doesn't seem to have an "exclusive" option of any kind. If the file I'm opening already exists, I have to either append to it, open for writing at the end, or empty it.
Is there a "right" way to do this in ruby?
I have worked out a method that I think is safe, but is seems overly complex:
# make a unique filename
time = Time.now
filename = "#{time.to_i}-#{sprintf('%06d', time.usec)}"
# make tempfiles (this is gauranteed to find a unique creatable name)
data_file = Tempfile.new(["upload", ".data"], UPLOAD_BASE)
# but the file will be deleted automatically, which we don't want, so now link it in a stable location
count = 1
loop do
begin
# File.link will raise an exception if the destination path exists
File.link(data_file.path, File.join(UPLOAD_BASE, "#{filename}-#{count}.data"))
# so here we know we created a file successfully and nobody else will take it
break
rescue Errno::EEXIST
count += 1
end
end
# now unlink the original tempfiles (they're still writeable until they're closed)
data_file.unlink
# ... write to data_file and close it ...
NOTE: This won't work on Windows. Not a problem for me, but reader beware.
In my testing this works reliably. But again, is there a more straightforward way?
I would use SecureRandom.
Maybe something like:
p SecureRandom.uuid #=> "2d931510-d99f-494a-8c67-87feb05e1594"
or
p SecureRandom.hex #=> "eb693ec8252cd630102fd0d0fb7c3485"
You can specify the length, and count on an almost impossibly small chance of collision.
I actually found the answer after some digging. Of course the obvious approach is to see what Tempfile itself does. I just assumed it was native code, but it is not. The source for 1.8.7 can be found here for instance.
As you can see, Tempfile uses an apparently undocumented file mode of File::EXCL. So my code can be simplified substantially:
# make a unique filename
time = Time.now
filename = "#{time.to_i}-#{sprintf('%06d', time.usec)}"
data_file = nil
count = 1
loop do
begin
data_file = File.open(File.join(UPLOAD_BASE, "#{filename}-#{count}.data"), File::RDWR|File::CREAT|File::EXCL)
break
rescue Errno::EEXIST
count += 1
end
end
# ... write to data_file and close it ...
UPDATE And now I see that this is covered in a prior thread:
How do open a file for writing only if it doesn't already exist in ruby
So maybe this whole question should be marked a duplicate.
I'm using carrierwave in a Rails 4 project with the file storage for development and testing and the fog storage (for storing on Amazon S3) for production.
I would like to save my files with paths like this:
/model_class_name/part_of_hash/another_part_of_hash/hash-model_id.file_extension
(example: /images/12/34/1234567-89.png where 1234567 is the SHA1 hash of the file content and 89 is the id of the associated image model in the database).
What I tried so far is this:
class MyUploader < CarrierWave::Uploader::Base
def store_dir
"#{model.class.name.underscore}/#{sha1_for(file)[0..1]}/#{sha1_for(file)[2..3]}"
end
def filename
"#{sha1_for(file)}-#{model.id}.#{file.extension}" if original_file
end
private
def sha1_for file
Digest::SHA1.hexdigest file.read
end
end
This does not work because:
model.id is not available when filename is called
file is not always available when store_dir is called
So, coming to my questions:
is it possible to use model ids/attributes within filename? This link says it should not be done; is there a way to work around it?
is it possible to use file content/attributes within store_dir? I found no documentation on this but my experiences so far say "no" (see above).
how would you implement file/directory naming to get something as close as possible to what I outlined in the beginning?
Including the id in the filename on create may not be possible, since the filename is stored in the database but the id isn't available yet. An (admittedly rather extreme) workaround would be to use a temporary value on create, and then after_commit on: :create, move the file and change the name in the database. It may be possible to optimize this with an after_create, but I'll leave that up to you. (This is where carrierwave actually uploads the file.)
Including file attributes directly within the store_dir isn't possible, since store_dir is used to calculate the url—url would require knowing the sha1, which requires having access to the file, which requires knowing the url, etc. The workaround is pretty obvious: cache the attributes in which you're interested (in this case the sha1) in the model's database record, and use that in the store_dir.
The simpler variant on the id-in-filename approach is to use some other value, such as a uuid, and store that value in the database. There are some notes on that here.
Taavo's answer strictly answers my questions. But I want to quickly detail the final solution I implemented since it may helps someone else, too...
I gave up the idea to use the model id in the filename and replaced it with a random string instead (the whole idea of the model id in the filename was to just ensure that 2 identical files associated with different models end up with different file names; and some random characters ensure that as well).
So I ended up with filenames like filehash-randomstring.extension.
Since carrierwave saves the filename in the model, I realized that I already have the file hash available in the model (in the form of the first part of the filename). So I just used this within store_dir to generate a path in the form model_class_name/file_hash_part/another_file_hash_part.
My final implementation looks like this:
class MyUploader < Carrierwave::Uploader::Base
def store_dir
# file name saved on the model. It is in the form:
# filehash-randomstring.extension, see below...
filename = model.send(:"#{mounted_as}_identifier")
"#{model.class.name.underscore}/#{filename[0..1]}/#{filename[3..4]}"
end
def filename
if original_filename
existing = model.send(:"#{mounted_as}_identifier")
# reuse the existing file name from the model if present.
# otherwise, generate a new one (and cache it in an instance variable)
#generated_filename ||= if existing.present?
existing
else
"#{sha1_for file}-#{SecureRandom.hex(4)}.#{file.extension}"
end
end
end
private
def sha1_for file
Digest::SHA1.hexdigest file.read
end
end
I came across the same problem recently, where the model.id was not available yet when storing the filename in the DB, upon creation of the uploader record. I found this workaround. I am not sure if it is respecting RESTful principles, I am open to suggestions.
I modified the controller, so that right after the creation of the image, an update_attributes is executed, so that the filename including the now existing model.id value is saved in the DB.
def create
#uploader = Uploader.new(uploader_params)
if #uploader.save
if #uploader.update_attributes(uploader_params)
render json: #uploader, status: :created
end
else
render json: #uploader.errors, status: :unprocessable_entity
end
end
I'm trying to do a simple benchmark to know how many octets are read from the cache in each page of my Rails site. I'm talking about the data retrieve from Rails.cache.
I would like to display something like 123Ko/145Ko at the bottom of my pages.
Does a gem exist to perform this task or perhaps is there something included in the ruby standard library?
One option is to subclass the store you're using and extend the protected read_entry method declared in ActiveSupport::Cache::Store, which FileStore and the other caches themselves subclass.
FileStoreWithReadTracking < ActiveSupport::Cache::FileStore
def start_page
#octets_read = 0
end
def octets_read
#octets_read
end
protected
def read_entry(key, options)
entry = super(key, options)
#octets_read += entry.size if entry
entry
end
end
When starting a page, you can call start_page to zero out the octet count. Since read_entry is a low-level method used every time the cache tries to perform a read, you can intercept any data read and get its size before returning it. You might have to convert size to octets.
To set this as your custom cache store, add config.cache_store = FileStoreWithReadTracking.new('/path/to/file/store') to your environment. I think you can subclass all the stores this way.