How to retrieve data from a DSN using Rails - ruby-on-rails

I have a DSN (data source name) of the following format:
<driver>://<username>:<password>#<host>:<port>/<database>
and I am asked to retrieve rows from the corresponding database, which has a single table in this specific example and it is on AWS. I would like to do it using an endpoint in a Rails app.
I did some research online to look for an example about DSN, but couldn't find any help.
I am looking for some high level explanation of how to work with DSN, and ideally how to use Rails to communicate with the database

I am not sure if it is going to be useful for anybody, but this is as much info I could gather.
The DSN format is something like:
<driver>://<username>:<password>#<host>:<port>/<database>
Within rails it should be used like, assuming you want an array of users:
require 'pg'
conn = PG.connect('<driver>://<username>:<password>#<host>:<port>/<database>')
puts conn.exec("SELECT count(*) FROM users").to_a

Related

Using raw Postgresql queries in Ruby on Rails

First of all, I'm not a native English speaker, so excuse me if I make any mistakes related to my writing.
I'm starting as a dev and I was told that I should use a Postgresql query to accomplish a specific task in Ruby on Rails. This is the query:
ORDER BY array_position(ARRAY[1, 2, 3], spree_products.id)
The goal is to show a view with different Spree products (probably the "ARRAY[1,2,3, etc]" part) of the database ordered in the same way they are placed in an input of an admin form.
Anyway, my question is more simple than that.
I understand this query is raw Postgresql, using an array_position function
https://www.postgresql.org/docs/12/functions-array.html
I just want to know how can I implement this kind of raw queries in Rails. It probably has to be in an .rb file, but I don't really understand how, since query methods used in Rails (.order, for example) are much more common to see being used (that's what I commonly use), but apparently, here I should use raw Postgresql mixed with Rails.
Can you give me some examples and/or a basic documentation, please? I know the basics of both languages, but I've never mixed them in this way, so that's what is probably confusing me the most.
Thank you all.
you could create a scope and use arel wrap the array function array_position
class Product < ApplicationRecord
scope :order_ids, ->(arr_ids) {
order(Arel.sql("array_position(ARRAY#{arr_ids}, id::integer)"))
}
end
now you could query as below
Product.order_ids([3,2,1])
Product.where(active: true).order_ids([3,2,1]).limit(10)
# ...

Observe the MongoDB and perform actions accordingly if the tables are updated

So, I'm developing an application in Rails with mongodb as the database. So, this mongodb serves as a common database for 3 other websites.
Now, my problem is I also have another MySQL database(I know, my data is spread everywhere!!) and whenever this mongodb database is updated, I also want to update the MySQL database. Basically, I want to observe for any changes in the mongodb database and reflect the same changes to the MySQL.
Show me a way to accomplish this. Thanks a ton. :)
I found a precise solution for my need:
Using this gem called "MongoRiver": https://github.com/stripe/mongoriver
mongo = Mongo::MongoClient.from_uri(mongo_uri)
tailer = Mongoriver::Tailer.new([mongo], :existing)
outlet = YourOutlet.new(your_params)
stream = Mongoriver::Stream.new(tailer, outlet)
stream.run_forever(starting_timestamp)
In YourOutlet.rb, you define a class extending the Outlet class in the gem and define events like after_insert, after_update, before_insert, before_update etc. and they will be executed accordingly.
So, how this works is simple: As someone mentioned in the comments, you have to enable the Oplog in your MongoDB. And set a oplog size that's suitable to your need, usually of 1 GB.
Read about it here: http://docs.mongodb.org/manual/tutorial/deploy-replica-set/

Bulk upsert with Ruby on Rails

I have a Rails 3 application where I need to ingest an XML file provided by an external system into a Postgres database. I would like to use something like ActiveRecord-Import but this does not appear to handle upsert capabilities for Postgres, and some of the records I will be ingesting will already exist, but will need to be updated.
Most of what I'm reading recommends writing SQL on the fly, but this seems like a problem that may have been solved already. I just can't find it.
Thanks.
You can do upserting on MySQL and PostgreSQL with upsert.
If you're looking for raw speed, you could use nokogiri and upsert.
It might be easier to import the data using data_miner, which uses nokogiri and upsert internally.
If you are on PostgreSQL 9.1 you should use writeable common table expressions. Something like:
WITH updates (id) AS (
UPDATE mytable SET .....
WHERE ....
RETURNING id
)
INSERT INTO mytable (....)
SELECT ...
FROM mytemptable
WHERE id NOT IN (select id from updates);
In this case you bulk process thins in a temp table first, then it will try to update the records from the temptable according to your logic, and insert the rest.
Its a two step thing. First you need to fetch the XML File. If its provided by a user via a form that luck for you otherwise you need to fetch it using the standard HTTP lib of ruby or otherwise some gem like mechanize (which is actually really great)
The second thing is really easy. You read all the XML into a string and then you can convert it into a hash with this pice of code:
Hash.from_xml(xml_string)
Then you can parse and work with the data...

rails and multi dbs, using establish_connection on the fly to route to the correct database?

I want to structure my database in such a way that certain tables (that don't have any relationships with other tables, so no joins required) have to be put on seperate mysql databases.
I know each model has a establish_connection property.
What I want to do:
I will fetch 10 rows from a particular model, based on a clientID.
The clientID will determine which database this model will be fetched from.
I want to have this database routing logic baked into the model logic somehow.
Is this possible?
You can point individual models to different databases using establish_connection in your models. See here for examples.
If you want a single model to access multiple databases based on an attribute, you probably need to use database sharding, for example, with DataFabric or ShardTheLove.
Don't know if this can help you :
https://github.com/brianmario/mysql2
This allows you to do connections and requests like this :
client = Mysql2::Client.new(:host => "localhost", :username => "root")
results = client.query("SELECT * FROM users WHERE group='githubbers'")
Looks like what you need is 'Sharding' support
Unfortunately, ActiveRecord doesn't have this built in. There are a bunch of gems that hack in Sharding support
The gem 'octopus' seems promising. See https://github.com/tchandy/octopus
Note: I haven't used octopus myself
If you are willing to go one step further, I recommend Sequel ( http://sequel.rubyforge.org/ ) which is a stable, feature-rich ORM that I have used before.
The documentation is excellent. What you need is http://sequel.rubyforge.org/rdoc/files/doc/sharding_rdoc.html
It might take sometime to get used to Sequel if you are accustomed to AR 2.x. There is a nice migration guide. See http://sequel.rubyforge.org/rdoc/files/doc/active_record_rdoc.html
Hoping this helps

Rails: Accessing a database not meant for Rails?

I have a standard rails application, that uses a mysql database through Active Record, with data loaded through a separate parsing process from a rather large XML file.
This was all well and good, but now I need to load data from an Oracle database, rather than the XML file.
I have no control how the database looks, and only really need a fraction of the data it contains (maybe one or two columns out of a few tables). As such, what I really want to do is make a call to the database, get data back, and put the data in the appropriate locations in my existing, Rails friendly mysql database.
How would I go about doing this? I've heard* you can (on a model by model basis) specifiy different databases for Rails Models to use, but that sounds like they use them in their entirety, (that is, the database is Rails friendly). Can I make direct Oracle calls? Is there a process that makes this easier? Can Active Record itself handle this?
A toy example:
If I need to know color, price, and location for an Object, then normally I would parse a huge XML file to get this information. Now, with oracle, color, price, and location are all in different tables, indexed by some ID (there isn't actually an "Object" table). I want to pull all this information together into my Rails model.
Edit: Sounds like what I'd heard about was ActiveRecord's "establish_connection" method...and it does indeed seem to assume one model is mapped to one table in the target database, which isn't true in my case.
Edit Edit: Ah, looks like I might be wrong there. "establish_connection" might handle my situation just fine (just gotta get ORACLE working in the first place, and I'll know for sure... If anyone can help, the question is here)
You can create a connection to Oracle directly and then have ActiveRecord execute a raw SQL statement to query your tables (plural). Off the top of my head, something like this:
class OracleModel < ActiveRecord::Base
establish_connection(:oracle_development)
def self.get_objects
self.find_by_sql("SELECT...")
end
end
With this model you can do OracleModel.get_objects which will return a set of records whereby the columns specified in the SELECT SQL statement are attributes of each OracleModel. Obviously you can probably come up with a more meaningful model name than I have!
Create an entry named :oracle_development in your config/database.yml file with your Oracle database connection details.
This may not be exactly what you are looking for, but it seems to cover you situation pretty well: http://pullmonkey.com/2008/4/21/ruby-on-rails-multiple-database-connections/
It looks like you can make an arbitrarily-named database configuration in the the database.yml file, and then have certain models connect to it like so:
class SomeModel < ActiveRecord::Base
establish_connection :arbitrary_database
#other stuff for your model
end
So, the solution would be to make ActiveRecord models for just the tables you want data out of from this other database. Then, if you really want to get into some sql, use ActiveRecord::Base.connection.execute(sql). If you need it as a the actual active_record object, do SomeModel.find_by_sql(sql).
Hope this helps!
I don't have points enough to edit your question, but it sounds like what you really need is to have another "connection pool" available to the second DB -- I don't think Oracle itself will be a problem.
Then, you need to use these alternate connections to "simply" execute a custom query within the appropriate controller method.
If you only need to pull data from your Oracle database, and if you have any ability to add objects to a schema that can see the data you require . . . .
I would simplify things by creating a view on the Oracle table that projects the data you require in a nice friendly shape for ActiveRecord.
This would mean maintaining code to two layers of the application, but I think the gain in clarity on the client-side would outweigh the cost.
You could also use the CREATE OR REPLACE VIEW Object AS SELECT tab1., tab2. FROM tab1,tab2 syntax so the view returned every column in each table.
If you need to Insert or Update changes to your Rails model, then you need to read up on the restrictions for doing Updates through a view.
(Also, you may need to search on getting Oracle to work with Rails as you will potentially need to install the Oracle client software and additional Ruby modules).
Are you talking about an one-time data conversion or some permanent data exchange between your application and the Oracle database? I think you shouldn't involve Rails in. You could just make a SQL query to the Oracle database, extract the data, and then just insert it into the MySQL database.

Resources