I have a standard rails application, that uses a mysql database through Active Record, with data loaded through a separate parsing process from a rather large XML file.
This was all well and good, but now I need to load data from an Oracle database, rather than the XML file.
I have no control how the database looks, and only really need a fraction of the data it contains (maybe one or two columns out of a few tables). As such, what I really want to do is make a call to the database, get data back, and put the data in the appropriate locations in my existing, Rails friendly mysql database.
How would I go about doing this? I've heard* you can (on a model by model basis) specifiy different databases for Rails Models to use, but that sounds like they use them in their entirety, (that is, the database is Rails friendly). Can I make direct Oracle calls? Is there a process that makes this easier? Can Active Record itself handle this?
A toy example:
If I need to know color, price, and location for an Object, then normally I would parse a huge XML file to get this information. Now, with oracle, color, price, and location are all in different tables, indexed by some ID (there isn't actually an "Object" table). I want to pull all this information together into my Rails model.
Edit: Sounds like what I'd heard about was ActiveRecord's "establish_connection" method...and it does indeed seem to assume one model is mapped to one table in the target database, which isn't true in my case.
Edit Edit: Ah, looks like I might be wrong there. "establish_connection" might handle my situation just fine (just gotta get ORACLE working in the first place, and I'll know for sure... If anyone can help, the question is here)
You can create a connection to Oracle directly and then have ActiveRecord execute a raw SQL statement to query your tables (plural). Off the top of my head, something like this:
class OracleModel < ActiveRecord::Base
establish_connection(:oracle_development)
def self.get_objects
self.find_by_sql("SELECT...")
end
end
With this model you can do OracleModel.get_objects which will return a set of records whereby the columns specified in the SELECT SQL statement are attributes of each OracleModel. Obviously you can probably come up with a more meaningful model name than I have!
Create an entry named :oracle_development in your config/database.yml file with your Oracle database connection details.
This may not be exactly what you are looking for, but it seems to cover you situation pretty well: http://pullmonkey.com/2008/4/21/ruby-on-rails-multiple-database-connections/
It looks like you can make an arbitrarily-named database configuration in the the database.yml file, and then have certain models connect to it like so:
class SomeModel < ActiveRecord::Base
establish_connection :arbitrary_database
#other stuff for your model
end
So, the solution would be to make ActiveRecord models for just the tables you want data out of from this other database. Then, if you really want to get into some sql, use ActiveRecord::Base.connection.execute(sql). If you need it as a the actual active_record object, do SomeModel.find_by_sql(sql).
Hope this helps!
I don't have points enough to edit your question, but it sounds like what you really need is to have another "connection pool" available to the second DB -- I don't think Oracle itself will be a problem.
Then, you need to use these alternate connections to "simply" execute a custom query within the appropriate controller method.
If you only need to pull data from your Oracle database, and if you have any ability to add objects to a schema that can see the data you require . . . .
I would simplify things by creating a view on the Oracle table that projects the data you require in a nice friendly shape for ActiveRecord.
This would mean maintaining code to two layers of the application, but I think the gain in clarity on the client-side would outweigh the cost.
You could also use the CREATE OR REPLACE VIEW Object AS SELECT tab1., tab2. FROM tab1,tab2 syntax so the view returned every column in each table.
If you need to Insert or Update changes to your Rails model, then you need to read up on the restrictions for doing Updates through a view.
(Also, you may need to search on getting Oracle to work with Rails as you will potentially need to install the Oracle client software and additional Ruby modules).
Are you talking about an one-time data conversion or some permanent data exchange between your application and the Oracle database? I think you shouldn't involve Rails in. You could just make a SQL query to the Oracle database, extract the data, and then just insert it into the MySQL database.
Related
I have a Rails application and a JSON file which contains the data, instead of creating table for a model i want to point my model to read that jSON file, and i should be table to treat that file like a table, please suggest.
If I understand you correctly, you want a model that uses a JSON file as it's "backend DB" instead of a normal DB?
In order to get a Rails model to point to a JSON file, you would need to use a JSON DB adapter, and I'm not sure if there is one for Rails.
Rails uses what are called "adapters", where the same code:
Model.find(1)
Will work on any DB (PostgreSQL, MySQL, SQLite3, etc...) because the Model.find() method has been implemented by each adapter.
This allows the interface for the developer to remain the same as long as the adapter implements it.
It avoids the problem where every DB creator implements a different interface, and now everyone has to learn those particular methods (convention for the win!).
All that said, I can't find a JSON DB adapter, so if you want that functionality you'll have to read a JSON file and search against it.
However, if you're talking about using client-side storage with a JSON file, this isn't possible because client-side only understand JavaScript and a Ruby model (class) is on the backend server. They don't directly talk to each other. In that case, you'll have to implement a JavaScript model that maps to the JSON data.
MySQL as in-memory-database
Rails with in memory database has a way to use MySQL in-memory; then you load your data from the JSON file at the start and dump it out at the end (or after commits).
In-memory DB adapter
https://github.com/maccman/supermodel exists but looks dead (4 years old). Maybe you find others?
Rolling it yourself with Nulldb
Use https://github.com/nulldb/nulldb to throw away all SQL statements and register some hooks (after_save etc.) to store them in some hash. You then load that has into memory at the start and dump it out to JSON later.
Separating concerns
My favourite approach, maybe too late if you have lots of working code already:
Separate your active-record code from your actual domain model. That means, if you today have a model class Stuff < ActiveRecord::Base, then separate that into class StuffAR < ActiveRecord::Base and class Stuff.
Stuff then contains an instance of StuffAR.
Using the proper ruby mechanisms (i.e., BasicObject::method_missing), you can delegate all calls from Stuff to StuffAR by default.
You now have complete control over your data and can do whatever with it.
(This is only one way to do it, depending on how dynamic/flexible you want to be, and if you need a real DB part-time, you can do it different; i.e. class Stuff < StuffAR at the one extreme, or not using a generic method_missing but explicitly coded methods which call StuffAR etc. - Stuff is PORO (plain old ruby objects) now and you use StuffAR just for the DB contact)
In this approach, be careful not to use Stuff like an AR object. I.e., do not use Stuff.where(name: 'xyz') from outside, but create domain methods for that (e.g., in this example, Stuff.find_by_name(...).
Yes, this is coding overhead, but it does wonders to improve your code when your models become big and unwieldy after a time.
Don't need AR at all?
If you do only want to use JSON ever, and never use a real DB, then do the same as before, just leave StuffAR out. It's just PORO then.
I think you have to import your JSON file to the database (e.g. sqlite3) to handle it as a table.
The other workaround would be:
Create a JSON importer for your model which fills the Users from the JSON into an Array of users.
If you do that, you'll have to write the whole searching/ordering by yourself in plain ruby.
I don't know what your current circumstances are, but if you would like to change some data or add data, I suggest using a simple and lightweight database like sqlite3
I have a previously separately managed sql file containing rather simple but large database. Would there be a way to import this sql file and generate ruby code as models using this data as a starting point for my future development?
Thank you for your help!
Yes!
It will take some work!
And you'll need to post a WHOLE HELL OF A LOT more detail to get more than that. ;-)
Taking a stab:
Rails can use legacy databases with a lot of effort manually specifying foreign key columns, table names, etc. It can be done. My suggestion, though, would be to convert the data in-place in whatever database you have by using a lot of ALTER TABLE RENAME... work and same for columns to make the old DB conform to Rails' convetions (primary key == 'id', table name is plural underscore'd version of model name, all that) before doing the import, and then you can just use plain vanilla ActiveRecord and all will be easy.
I am new to ruby and rails and I am having difficulty conceptualizing the MVC techniques in conjunction with database views. I am dealing with a legacy database that has several viiews that are used to generate reports.
Where I get lost is how do I actually use a database view. Should it be put in a model? If so what exactly would that look like?
As an example the legacy db has a view called qryTranscriptByGroup. It is used in the legacy application in an SQL statement such as "SELECT * FROM qryTranscriptByGroup WHERE group='test_group'". This returns a small number of records usually less than 100.
If i create a model, Transcript, how would I define a method like Transcript.find_by_group(group)? As well, it would seem that I might need to prevent any other "find" methods as they would be invalid in this context.
There is also the the fact that the view is read-only and I would need to prevent any attempts to create, update or destroy it.
Perhaps I am going about this entirely the wrong way. The bottom line is that I need to get information from several tables (models?) that represent the information about a user (a transcript). Actually one or more users (transcripts plural).
-Thanks!
You can use a database view like a normal model.
In your case:
class Transcript < ActiveRecord::Base
set_table_name "qryTranscriptByGroup"
set_primary_key "if_not_id"
end
The query will be then:
Trascript.find_by_group('test_group')
without you need to declare anything.
Rails uses the method_missing method to magically generate find_by_column_name methods.
For the create/update/delete action you can simply delete them or not create them in the controller.
I want to structure my database in such a way that certain tables (that don't have any relationships with other tables, so no joins required) have to be put on seperate mysql databases.
I know each model has a establish_connection property.
What I want to do:
I will fetch 10 rows from a particular model, based on a clientID.
The clientID will determine which database this model will be fetched from.
I want to have this database routing logic baked into the model logic somehow.
Is this possible?
You can point individual models to different databases using establish_connection in your models. See here for examples.
If you want a single model to access multiple databases based on an attribute, you probably need to use database sharding, for example, with DataFabric or ShardTheLove.
Don't know if this can help you :
https://github.com/brianmario/mysql2
This allows you to do connections and requests like this :
client = Mysql2::Client.new(:host => "localhost", :username => "root")
results = client.query("SELECT * FROM users WHERE group='githubbers'")
Looks like what you need is 'Sharding' support
Unfortunately, ActiveRecord doesn't have this built in. There are a bunch of gems that hack in Sharding support
The gem 'octopus' seems promising. See https://github.com/tchandy/octopus
Note: I haven't used octopus myself
If you are willing to go one step further, I recommend Sequel ( http://sequel.rubyforge.org/ ) which is a stable, feature-rich ORM that I have used before.
The documentation is excellent. What you need is http://sequel.rubyforge.org/rdoc/files/doc/sharding_rdoc.html
It might take sometime to get used to Sequel if you are accustomed to AR 2.x. There is a nice migration guide. See http://sequel.rubyforge.org/rdoc/files/doc/active_record_rdoc.html
Hoping this helps
thanks for your time first...after all the searching on google, github and here, and got more confused about the big words(partition/shard/fedorate)，I figure that I have to describe the specific problem I met and ask around.
My company's databases deals with massive users and orders, so we split databases and tables in various ways, some are described below:
way database and table name shard by (maybe it's should be called partitioned by?)
YZ.X db_YZ.tb_X order serial number last three digits
YYYYMMDD. db_YYYYMMDD.tb date
YYYYMM.DD db_YYYYMM.tb_ DD date too
The basic concept is that databases and tables are seperated acording to a field(not nessissarily the primary key), and there are too many databases and too many tables, so that writing or magically generate one database.yml config for each database and one model for each table isn't possible or at least not the best solution.
I looked into drnic's magic solutions, and datafabric, and even the source code of active record, maybe I could use ERB to generate database.yml and do database connection in around filter, and maybe I could use named_scope to dynamically decide the table name for find, but update/create opertions are bounded to "self.class.quoted_table_name" so that I couldn't easily get my problem solved. And even I could generate one model for each table, because its amount is up to 30 most.
But this is just not DRY!
What I need is a clean solution like the following DSL:
class Order < ActiveRecord::Base
shard_by :order_serialno do |key|
[get_db_config_by(key), #because some or all of the databaes might share the same machine in a regular way or can be configed by a hash of regex, and it can also be a const
get_db_name_by(key),
get_tb_name_by(key),
]
end
end
Can anybody enlight me? Any help would be greatly appreciated~~~~
Case two (where only db name changes) is pretty easy to implement with DbCharmer. You need to create your own sharding method in DbCharmer, that would return a connection parameters hash based on the key.
Other two cases are not supported right away, but could be easily added to your system:
You implement sharding method that knows how to deal with database names in your sharded dabatase, this would give you an ability to do shard_for(key) calls to your model to switch db connection.
You add a method like this:
class MyModel < ActiveRecord::Base
db_magic :sharded => { :sharded_connection => :my_sharding_method }
def switch_shard(key)
set_table_name(table_for_key(key)) # switch table
shard_for(key) # switch connection
end
end
Now you could use your model like this:
MyModel.switch_shard(key).first
MyModel.switch_shard(key).count
and, considering you have shard_for(key) call results returned from the switch_shard method, you could use it like this:
m = MyModel.switch_shard(key) # Switch connection and get a connection proxy
m.first # Call any AR methods on the proxy
m.count
If you want that particular DSL, or something that matches the logic behind the legacy sharding you are going to need to dig into ActiveRecord and write a gem to give you that kind of capability. All the existing solutions that you mention were not necessarily written with your situation in mind. You may be able to bend any number of solutions to your will, but in the end you're gonna have to probably write custom code to get what you are looking for.
Sounds like, in this case, you should consider not use SQL.
If the data sets are that big and can be expressed as key/value pairs (with a little de-normalization), you should look into couchDB or other noSQL solutions.
These solutions are fast, fully scalable, and is REST based, so it is easy to grow and backup and replicate.
We all have gotten into solving all our problems with the same tool (Believe me, I try to too).
It would be much easier to switch to a noSQL solution then to rewrite activeRecord.