I have an rails app where one of the attributes on an object is a data set which consists of an array of x,y coordinates. I am currently storring this in the sql database using the rails serialize helper :
serialize :data, Array
This converts the array to yaml and then stores it in a string field in the sql database. The problem is that our database is getting really big doing this and we need to keep it smaller. Is it possible to serialize to raw binary instead of a string and store in a blob?, this would dramatically reduce the size and help our problem.
I have had a search for a gem to do this, or even a ruby method that will turn an array in to binary data without much help. Any suggestions would be appreciated.
You may be interested in Array.pack and String.unpack methods. See ruby documentation for it: type ri Array.pack
You may want to use a 'packed_data' attribute in your database, then add accessors to pack/unpack it:
def data
packed_data.unpack('....')
end
def data=(v)
self.packed_data = v.pack('....')
end
To make it more useful, you may store the unpacked form in a variable, but you have to remember to clear it when the packed_data attribute changes, like when you call .reload
before_validation :pack_data
UNPACK_FORMAT = '.....' # See ri Array.pack
def data
#data ||= packed_data.unpack(UNPACK_FORMAT)
end
def data=(v)
#data = v
end
def reload(options=nil)
#data = nil
super
end
def pack_data
self.packed_data = self.data.pack(UNPACK_FORMAT)
true # Because we are in a before_.. callback
end
The format of the magic string used to pack/unpack the data depends on the data you have in your array. The documentation will help you to choose the right one.
I believe the format for pack and unpack will be the same, but don't trust me too much. ;)
Related
I am using Ruby on Rails 4.1, Ransack and attr_encrypted. I have sensitive data being stored in my database and I want to protect it using the gem attr_encrypted.
As I expected, I got zero results when searching encrypted test data with Ransack.
I tried the following solution to but it didn't seem to work for me. I was under the impression that the load function was used to return the decrypted value.
ReportsController
def index
#report_list = Report.all.load
#q = #report_list.search(params[:q])
#reports = #q.result(distinct: true).order('created_at DESC')
end
Has anyone had any experience searching across encrypted data and could help me generate a working solution?
load will cause the Active Record Collection to execute a query and retrieve the results matching your query (in addition to running after_create call backs which I believe is where the decrypt you were expecting is happening).
def index
#returns all records in DB
#report_list = Report.all.load
#I'm surprised these aren't throwing undefined method search of Array (or something similar)
#q = #report_list.search(params[:q])
#reports = #q.result(distinct: true).order('created_at DESC')
end
I would like to precede this with, I normally do this thing manually and am not familiar with attr_encrypted or Ransack, but I believe these concepts are general enough they could be applied to any setup. So, as to your question, 2 possibilities.
If your ok searching for exact values:
Model.where(encrypted_field: encrypt(params[:value])).first
where encrypt is a method that encrypts and returns the passed string.
Secondly (and painfully)
Model.all.delete_if{|m| !m.encrypted_field.include?(params[:value]) }
This will literally pull, decrypt, and scan every entry in your database.
I would highly recommend not doing this, but you need to do what you need to do.
If you absolutely need to have the information encrypted but still need to be able to do searches like this. I would highly recommend adding tags of some sort to your model. This would allow you to remove sensitive information but still search by some attributes.
I have a model and in that model I'm generating a more complex field than I've done before. I've serialized hashes and arrays, but this field is the result of Gibberish::RSA.generate_keypair ( https://github.com/mdp/gibberish ). Which is more or less a private/public key pair in a ruby wrapper, to my understanding.
Working from the command line, I can do an update_attributes and the result of the generation gets stored in the text field. When doing rake db:seed or creating an instance, this doesn't work, I get a yaml string that indicates several types of Gibberish objects.
How do I do more complex activerecord serialization beyond hashes and arrays? Or how do I approach a greater understanding of what I'm trying to do?
Code:
def generate_keypair
self.update_attributes(:rsakey => Gibberish::RSA.generate_keypair(1024) )
end
which I call on the associated model creation, basic call the Gibberish wrapper
Then the output I get for the field myresource.rsakey
"--- !ruby/object:Gibberish::RSA::KeyPair\nkey:
!ruby/object:OpenSSL::PKey::RSA {}\ncipher:
!ruby/object:OpenSSL::Cipher::Cipher {}\n"
Updating the attributes works from the rails command line, but not while seeding or creating. Other ways I attempted to add serialize so far have completely ruined the process or the created instances.
EDIT: solved bluntly by just calling 'to_s' on the result of the keypair generation method, which just saves it as a text field that 'works for now' until it needs to be more elegant.
The underlying issue seems to be that the openssl library doesn't implement YAML dumping:
YAML.dump(OpenSSL::PKey::RSA.generate(1024))
#=> "--- !ruby/object:OpenSSL::PKey::RSA {}"
If you are using rails 3.1 you can define custom serializers, like so
class KeySerializer
def dump(key)
key.to_pem
end
def load(data)
data && OpenSSL::Pkey::RSA.new(data)
end
end
Then in your class you can do
class Foo < ActiveRecord::Base
serialize :key, KeySerializer.new
end
I am consuming JSON data from a third party API, doing a little bit of processing on that data and then sending the models to the client as JSON. The keys for the incoming data are not named very well. Some of them are acronyms, some just seem to be random characters. For example:
{
aikd: "some value"
lrdf: 1 // I guess this is the ID
}
I am creating a rails ActiveResource model to wrap this resource, but would not like to access these properties through model.lrdf as its not obvious what lrdf really is! Instead, I would like some way to alias these properties to another property that is named better. Something so that I can say model.id = 1 and have that automatically set lrdf to 1 or puts model.id and have that automatically return 1. Also, when I call model.to_json to send the model to the client, I dont want my javascript to have to understand these odd naming conventions.
I tried
alias id lrdf
but that gave me an error saying method lrdf did not exist.
The other option is to just wrap the properties:
def id
lrdf
end
This works, but when I call model.to_json, I see lrdf as the keys again.
Has anyone done anything like this before? What do you recommend?
Have you tried with some before_save magic? Maybe you could define attr_accessible :ldrf, and then, in your before_save filter, assign ldrf to your id field. Haven't tried it, but I think it should works.
attr_accessible :ldrf
before_save :map_attributes
protected
def map_attributes
{:ldrf=>:id}.each do |key, value|
self.send("#{value}=", self.send(key))
end
end
Let me know!
You could try creating a formatter module based on ActiveResource::Formats::JsonFormat and override decode(). If you had to update the data, you'd have to override encode() also. Look at your local gems/activeresource-N.N.N/lib/active_resource/formats/json_format.rb to see what the original json formatter does.
If your model's name is Model and your formatter is CleanupFormatter, just do Model.format = CleanupFormatter.
module CleanupFormatter
include ::ActiveResource::Formats::JsonFormat
extend self
# Set a constant for the mapping.
# I'm pretty sure these should be strings. If not, try symbols.
MAP = [['lrdf', 'id']]
def decode(json)
orig_hash = super
new_hash = {}
MAP.each {|old_name, new_name| new_hash[new_name] = orig_hash.delete(old_name) }
# Comment the next line if you don't want to carry over fields missing from MAP
new_hash.merge!(orig_hash)
new_hash
end
end
This doesn't involve aliasing as you asked, but I think it helps to isolate the gibberish names from your model, which would never have to know those original names existed. And "to_json" will display the readable names.
I am using redis as my web cache, and I want to store those activerecord objects to redis directly, but using redis-rb I get an error.
It seems that I can't serialize it or some what. Is there a lib to do this for me?
Am I have to serialize it to json format?
Which serialization format would be the most efficient?
Redis stores strings (and a few other data structures of strings); so you can serialize into Redis values however you like so long as you end up with a string.
JSON is probably the best place to start as it's lean, not overly brittle, works well with live upgrade patterns, and is readable in situ. Later you can add more complexity to meet your goals as needed, e.g., compression. #to_json and #from_json are already on ActiveRecord if you want to use JSON (with YAJL or its ilk that shouldn't be excessively slow, relatively speaking.) #to_xml is also there, if you're into S&M.
Raw marshaling can also work, but occasionally goes horrifically wrong (I've had marshaled objects exceed 2MB after LZO compression that were only a few K in JSON.)
If it's really a bottleneck for you, you'll want to run your own efficiency tests for your goal(s), e.g., write speed, read speed, or storage size, with your own objects and data patterns.
You can convert your model to a hash using attributes method and then save it with mapped_hmset
def redis_set()
redis.mapped_hmset("namespace:modelName:#{self.id}", self.attributes)
end
def redis_get(id)
redis.hgetall("namespace:modelName:#{id}")
end
def self.set(friend_list, player_id)
redis.set("friend_list_#{player_id}", Marshal.dump(friend_list)) == 'OK' ? friend_list : nil
end
def self.get(player_id)
friend_list = redis.get("friend_list_#{player_id}")
Marshal.load(friend_list) if friend_list
end
There are a lot of ways to store site preferences in database. But what if I need to manage datatypes. So some preferences will be boolean, others strings, others integers.
How can I organize such store?
I wrote a gem that does exactly this, and recently updated it for Rails 3:
For Rails 3:
http://github.com/paulca/configurable_engine
For Rails 2.3.x
http://github.com/paulca/behavior
Enjoy!
I am quite lazy with preferences and store the data as serialized JSON or YAML Hashes. Works really well, and generally preserves the data types as well.
I used a single table with a single row, and each column representing one preference. This makes it possible to have different datatypes.
To be able to retrieve a preference, I overrode method_missing to be able to retrieve the preference value directly from the class name without requiring an instance, something like this:
class Setting < ActiveRecord::Base
##instance = self.first
def self.instance
##instance
end
def self.method_missing(method, *args)
option = method.to_s
if option.include? '='
var_name = option.gsub('=', '')
value = args.first
##instance[var_name] = value
else
##instance[option]
end
end
end
Thus, to retrive a setting, you would use:
a_setting = Setting.column_name
Rails Migrations are used to create and update the database.
http://guides.rubyonrails.org/migrations.html