Rails with PostGIS - ruby-on-rails

First of all, I'm using Rails3 with Ruby 1.9.2.
I have a problem using PostgreSQL with PostGIS. I've tried two gems:
https://github.com/nofxx/georuby
https://github.com/rgeo/activerecord-postgis-adapter
The problem is, that the first one from nofxx works well, but it doesn't provide rake tasks, so that the postgis.sql and spatial_ref_sys.sql get inserted into the database. I need this, because it's comfortable and essential using tests (create automatically the database, insert the postgis sql and perform the migrations when running "rake test"). The strange thing is, that in the README on github the author writes something about PostGIS helpers and rake tasks. But they are not implemented (or I'm blind).
The second gem listed above provides this functionality for rake perfectly! But I ran into an issue that I can't solve. I followed the instructions in the README, but everytime when I try to set a position on a spatial column, i get the following output:
ruby-1.9.2-p136 :010 > v = Venue.new :location => "POINT(102.0, 47.2)"
=> #<Venue id: nil, last_updated: nil, created_at: nil, updated_at: nil, location: nil>
ruby-1.9.2-p136 :011 > v.location
=> nil
As you can see, I set a location using WKT (well known text), but it's always nil. I also tried to save it to the database, but it's also nil.
I figured out, that when I try to use the Objects provided by RGeo
RGeo::Geos.factory.point(-122, 47)
it says, that the factory is nil. So, there must be a problem by creating the factory provided by RGeo. As mentioned in the README of the activerecord-postgis-adapter I used the following line in my Venue model to provide this support:
self.rgeo_factory_generator = RGeo::Geos.factory_generator
but it doesn't work.
So to find a solution, there are two ways: Finding a way to get the rake tasks into the georuby gem of nofxx, or solve the problem with the factory of the activerecord-postgis-adapter.
Hope someone can help me,
thx!

I'v spent about 4 hours on solving the same problem with "activerecord-postgis-adapter". The answer is simple : "NEVER USER COMAS IN WKT". In your situation it must be v = Venue.new :location => "POINT(102.0 47.2)" Then everything works just fine!

Finally I solved it on my own. I was playing around with rake tasks and I ran into some issues (like, you can't provide a password from script with psql...) that anoyed me.
But i found out, that you can provide a template option to the database.yml file. So, I just wrote:
development:
adapter: postgresql
pool: 5
encoding: unicode
database: myDatabase
username: root
password: *****
su_username: root
su_password: *****
port: 5432
template: template_postgis
test:
adapter: postgresql
pool: 5
encoding: unicode
database: myDatabase_test
username: root
password: *****
su_username: root
su_password: *****
port: 5432
template: template_postgis
production:
adapter: postgresql
pool: 5
encoding: unicode
database: myDatabase_production
username: root
password: *****
su_username: root
su_password: *****
port: 5432
template: template_postgis
Now every time I create my database (during tests and so on) all the PostGIS stuff gets added! Yeah!

Related

ActiveRecord::ConnectionNotEstablished (No connection pool with 'primary' found.)

The postgresql table looks like:
id | name | size | md5sum
----+-----------------------------------------+------+----------------------------------
1 | MyPreferredGenomeName | 1000 | df768fde8e10e15511c41895363bcce6
I am trying to retrieve the data:
myData = myClass.find_by name: "MyPreferredGenomeName" # also tried with other columns
# or
myData = myClass.where name: "MyPreferredGenomeName"
and I get the error stated in the title from activerecord-5.2.4.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:1032:in retrieve_connection'`.
I don't have the most basic understanding of what's going on... Help.
Also, what does 'primary' stand for?
Thank you!
I suspect your config/database.yml is not configured correctly.
In that file you should have something similar to:
...
default: &default
adapter: postgresql
encoding: unicode
pool: <%= ENV.fetch("RAILS_MAX_THREADS") { 5 } %>
development:
<<: *default
database: database_name_development
...
You want to make sure that database_name_development is the name of the database in your local PostgreSQL instance. Is your database actually called primary as that is what the error seems to be saying.
You show the psql output with the table named MyPreferredGenomeName so whatever database that is in is needed in database.yml.
You can test what database Rails is looking for with:
rails console
ActiveRecord::Base.connection.current_database
=> "database_name_development"
(Do not post any secrets in any replies e.g. db passwords or full connection strings.)

How can I build a front end for querying a Redshift database (hopefully with Rails)

So I have a Redshift database with enough tables that it feels worth my time to build a front end to make querying it a little bit easier than just typing in SQL commands.
Ideally, I'd be able to do this by connecting the database to a Rails app (because I have a bit of experience with Rails). I'm not sure how I'd connect a remote Redshift database to a local Rails application though, or how to make activerecord work with redshift.
Does anyone have any suggestions/resources to help me get started? I'm open to other options to connect the Redshift database to a front end if there are pre-made options easier than Rails.
#app/models/data_warehouse.rb
class DataWarehouse < ActiveRecord::Base
establish_connection "redshift_staging"
#or, if you want to have a db per environment
#establish_connection "redshift_#{Rails.env}"
end
Note that we are connecting on 5439, not the default 5432 so I specify the port
Also, I specify a schema, beta, which is what we use for our unstable aggregates, you could either have a different db per environment as mentioned above, or use various schemas and include them in the search path for ActiveRecord
#config/database.yml
redshift_staging:
adapter: postgresql
encoding: utf8
database: db03
port: 5439
pool: 5
schema_search_path: 'beta'
username: admin
password: supersecretpassword
host: db03.myremotehost.us #your remote host here, might be an aws url from Redshift admin console
###OPTION 2, a direct PG Connection
class DataWarehouse < ActiveRecord::Base
attr_accessor :conn
def initialize
#conn = PG.connect(
database: 'db03',
port: 5439,
pool: 5,
schema_search_path: 'beta',
username: 'admin',
password: 'supersecretpassword',
host: 'db03.myremotehost.us'
)
end
end
[DEV] main:0> redshift = DataWarehouse
E, [2014-07-17T11:09:17.758957 #44535] ERROR -- : PG::InsufficientPrivilege: ERROR: permission denied to set parameter "client_min_messages" to "notice" : SET client_min_messages TO 'notice'
(pry) output error: #<ActiveRecord::StatementInvalid: PG::InsufficientPrivilege: ERROR: permission denied to set parameter "client_min_messages" to "notice" : SET client_min_messages TO 'notice'>
UPDATE:
I ended up going with option 1, but using this adapter for now for multiple reasons:
https://github.com/fiksu/activerecord-redshift-adapter
Reason 1: ActiveRecord postgresql adapter sets client_min_messages
Reason 2: adapter also attempts to set Time Zone, which redshift doesn't allow (http://docs.aws.amazon.com/redshift/latest/dg/c_redshift-and-postgres-sql.html)
Reason 3: Even if you change the code in ActiveRecord for the first two errors, you run into additional errors that complain that Redshift is using Postgresql 8.0, at that point I moved on to the adapter, will revisit and update if I find something better later.
I renamed my table to base_aggregate_redshift_tests (notice plural) so ActiveRecord was easily able to connect, if you can't change your table names in redshift use the set_table method I have commented out below
#Gemfile:
gem 'activerecord4-redshift-adapter', github: 'aamine/activerecord4-redshift-adapter'
Option 1
#config/database.yml
redshift_staging:
adapter: redshift
encoding: utf8
database: db03
port: 5439
pool: 5
username: admin
password: supersecretpassword
host: db03.myremotehost.us
timeout: 5000
#app/models/base_aggregates_redshift_test.rb
#Model named to match my tables in Redshift, if you want you can set_table like I have commented out below
class BaseAggregatesRedshiftTest < ActiveRecord::Base
establish_connection "redshift_staging"
self.table_name = "beta.base_aggregates_v2"
end
in console using self.table_name -- notice it queries the right table, so you can name your models whatever you want
[DEV] main:0> redshift = BaseAggregatesRedshiftTest.first
D, [2014-07-17T15:31:58.678103 #43776] DEBUG -- : BaseAggregatesRedshiftTest Load (45.6ms) SELECT "beta"."base_aggregates_v2".* FROM "beta"."base_aggregates_v2" LIMIT 1
Option 2
#app/models/base_aggregates_redshift_test.rb
class BaseAggregatesRedshiftTest < ActiveRecord::Base
set_table "beta.base_aggregates_v2"
ActiveRecord::Base.establish_connection(
adapter: 'redshift',
encoding: 'utf8',
database: 'staging',
port: '5439',
pool: '5',
username: 'admin',
password: 'supersecretpassword',
search_schema: 'beta',
host: 'db03.myremotehost.us',
timeout: '5000'
)
end
#in console, abbreviated example of first record, now it's using the new name for my redshift table, just assuming I've got the record at base_aggregates_redshift_tests because I didn't set the table_name
[DEV] main:0> redshift = BaseAggregatesRedshiftTest.first
D, [2014-07-17T15:09:39.388918 #11537] DEBUG -- : BaseAggregatesRedshiftTest Load (45.3ms) SELECT "base_aggregates_redshift_tests".* FROM "base_aggregates_redshift_tests" LIMIT 1
#<BaseAggregatesRedshiftTest:0x007fd8c4a12580> {
:truncated_month => Thu, 31 Jan 2013 19:00:00 EST -05:00,
:dma => "Cityville",
:group_id => 9712338,
:dma_id => 9999
}
Good luck #johncorser!
This tutorial helps you setup a rails app with a redshift adapter:
https://www.credible.com/code/setting-up-a-data-warehouse-with-aws-redshift-and-ruby/
In a nutshell:
Clone the sample app:
git clone git#github.com:tuesy/redshift-ruby-tutorial.git
cd redshift-ruby-tutorial
Setup ENV Variables via ~/.bashrc (or dotenv):
export REDSHIFT_HOST=redshift-ruby-tutorial.ccmj2nxbsay7.us-east-1.redshift.amazonaws.com
export REDSHIFT_PORT=5439
export REDSHIFT_USER=deploy
export REDSHIFT_PASSWORD=<your password here>
export REDSHIFT_DATABASE=analytics
export REDSHIFT_BUCKET=redshift-ruby-tutorial
Use the gem activerecord4-redshift-adapter, in Gemfile:
'activerecord4-redshift-adapter', '~> 0.2.0' # For Rails 4.2
'activerecord4-redshift-adapter', '~> 0.1.1' # For Rails 4.1
Then you can query into redshift like you would with a normal AR model:
bundle exec rails c
RedshiftUser.count
(Disclosure: I havn't yet given this method a try, but I may soon)
You might want to consider http://www.looker.com/. It's a frontend for exploring your DB, allowing easily savable queries and a GUI that the business guys can also use.

Error connecting external DB to Rails 4 App

I am attempting to add a Postgresql Redshift Database to my Rails 4 app, for use locally and in production. I am testing in Development first.
I have altered my database.yml file to look like this:
development:
adapter: postgresql
encoding: unicode
database: new_db
pool: 5
username: test
password: password
host: test_db.us-east-1.redshift.amazonaws.com
port: 5439
Now, when I hit localhost:3000 I get this error:
permission denied to set parameter "client_min_messages" to "warning"
: SET client_min_messages TO 'warning'
I can't seem to find out what is causing this - It seems like maybe my new DB isn't allow the SET command? I'm not really sure, but any help is appreciated.
Just answered this somewhere else, but I had the same issues today, here's what I did and it's working now:
#app/models/data_warehouse.rb
class DataWarehouse < ActiveRecord::Base
establish_connection "redshift_staging"
#or, if you want to have a db per environment
#establish_connection "redshift_#{Rails.env}"
end
Note that we are connecting on 5439, not the default 5432 so I specify the port
Also, I specify a schema, beta, which is what we use for our unstable aggregates, you could either have a different db per environment as mentioned above, or use various schemas and include them in the search path for ActiveRecord
#config/database.yml
redshift_staging:
adapter: postgresql
encoding: utf8
database: db03
port: 5439
pool: 5
schema_search_path: 'beta'
username: admin
password: supersecretpassword
host: db03.myremotehost.us #your remote host here, might be an aws url from Redshift admin console
###OPTION 2, a direct PG Connection
class DataWarehouse < ActiveRecord::Base
attr_accessor :conn
def initialize
#conn = PG.connect(
database: 'db03',
port: 5439,
pool: 5,
schema_search_path: 'beta',
username: 'admin',
password: 'supersecretpassword',
host: 'db03.myremotehost.us'
)
end
end
[DEV] main:0> redshift = DataWarehouse
E, [2014-07-17T11:09:17.758957 #44535] ERROR -- : PG::InsufficientPrivilege: ERROR: permission denied to set parameter "client_min_messages" to "notice" : SET client_min_messages TO 'notice'
(pry) output error: #<ActiveRecord::StatementInvalid: PG::InsufficientPrivilege: ERROR: permission denied to set parameter "client_min_messages" to "notice" : SET client_min_messages TO 'notice'>
UPDATE:
I ended up going with option 1, but using this adapter for now for multiple reasons:
https://github.com/fiksu/activerecord-redshift-adapter
Reason 1: ActiveRecord postgresql adapter sets client_min_messages
Reason 2: adapter also attempts to set Time Zone, which redshift doesn't allow (http://docs.aws.amazon.com/redshift/latest/dg/c_redshift-and-postgres-sql.html)
Reason 3: Even if you change the code in ActiveRecord for the first two errors, you run into additional errors that complain that Redshift is using Postgresql 8.0, at that point I moved on to the adapter, will revisit and update if I find something better later.
I renamed my table to base_aggregate_redshift_tests (notice plural) so ActiveRecord was easily able to connect, if you can't change your table names in redshift use the set_table method I have commented out below
#Gemfile:
gem 'activerecord4-redshift-adapter', github: 'aamine/activerecord4-redshift-adapter'
Option 1
#config/database.yml
redshift_staging:
adapter: redshift
encoding: utf8
database: db03
port: 5439
pool: 5
username: admin
password: supersecretpassword
host: db03.myremotehost.us
timeout: 5000
#app/models/base_aggregates_redshift_test.rb
#Model named to match my tables in Redshift, if you want you can set_table like I have commented out below
class BaseAggregatesRedshiftTest < ActiveRecord::Base
establish_connection "redshift_staging"
self.table_name = "beta.base_aggregates_v2"
end
in console using self.table_name -- notice it queries the right table, so you can name your models whatever you want
[DEV] main:0> redshift = BaseAggregatesRedshiftTest.first
D, [2014-07-17T15:31:58.678103 #43776] DEBUG -- : BaseAggregatesRedshiftTest Load (45.6ms) SELECT "beta"."base_aggregates_v2".* FROM "beta"."base_aggregates_v2" LIMIT 1
Option 2
#app/models/base_aggregates_redshift_test.rb
class BaseAggregatesRedshiftTest < ActiveRecord::Base
set_table "beta.base_aggregates_v2"
ActiveRecord::Base.establish_connection(
adapter: 'redshift',
encoding: 'utf8',
database: 'staging',
port: '5439',
pool: '5',
username: 'admin',
password: 'supersecretpassword',
search_schema: 'beta',
host: 'db03.myremotehost.us',
timeout: '5000'
)
end
#in console, abbreviated example of first record, now it's using the new name for my redshift table, just assuming I've got the record at base_aggregates_redshift_tests because I didn't set the table_name
[DEV] main:0> redshift = BaseAggregatesRedshiftTest.first
D, [2014-07-17T15:09:39.388918 #11537] DEBUG -- : BaseAggregatesRedshiftTest Load (45.3ms) SELECT "base_aggregates_redshift_tests".* FROM "base_aggregates_redshift_tests" LIMIT 1
#<BaseAggregatesRedshiftTest:0x007fd8c4a12580> {
:truncated_month => Thu, 31 Jan 2013 19:00:00 EST -05:00,
:dma => "Cityville",
:group_id => 9712338,
:dma_id => 9999
}
Good luck!
What's your postgresql version? or middle ware version? it's not a original postgresql version?
you can use \set VERBOSITY verbose
and then do it again, find which function#code raise the error.
and then analyze why you cann't set client_min_messages.
I use postgresql 9.3.3 no this problem.
super and normal user can set client_min_messages correct.
digoal=# \c digoal digoal
You are now connected to database "digoal" as user "digoal".
digoal=> set client_min_messages=debug;
SET
digoal=> set client_min_messages=warning;
SET
digoal=> SET client_min_messages TO 'warning';
SET
digoal=> \c postgres digoal
You are now connected to database "postgres" as user "digoal".
postgres=> SET client_min_messages TO 'warning';
SET
I was able to solve the issue by opening up the postgresql adapter for Activerecord and commenting out all of the SET commands.
However, this is not a best practice at all. At the end of editing the postgresql adapter I was faced with errors due to an outdated postgres version (80002).
The right answer is just updating postgres, which unfortunately isn't possible amongst my shared Redshift DB that lives at Amazon.

Mongoid only working in development. Throws error both is test and production

When I start my rails console as:
$ RAILS_ENV=development rails console
every thing seems to be working fine.
Mongoid is able to connect to mongodb and fetch records.
But with:
$ RAILS_ENV=test rails console
$ RAILS_ENV=production rails console
it's throwing up errors as:
Rack::File headers parameter replaces cache_control after Rack 1.5.
/usr/local/lib/ruby/gems/1.9.1/gems/mongoid-3.0.16/lib/mongoid/criteria.rb:585:in `check_for_missing_documents!': (Mongoid::Errors::DocumentNotFound)
Problem:
Document(s) not found for class Actor with id(s) 50e5259f53c205d815000001.
Summary:
When calling Actor.find with an id or array of ids, each parameter must match a document in the database or this error will be raised. The search was for the id(s): 50e5259f53c205d815000001 ... (1 total) and the following ids were not found: 50e5259f53c205d815000001.
Resolution:
Search for an id that is in the database or set the Mongoid.raise_not_found_error configuration option to false, which will cause a nil to be returned instead of raising this error when searching for a single id, or only the matched documents when searching for multiples.
My config/mongoid.yml has the exact same set of lines for all three environments.
I'm not able to figure out why it isn't able to connect in test and production.
Update:
Mongoid.yml
development:
sessions:
default:
database: tgmd
hosts:
- localhost:27017
test:
sessions:
default:
database: tgmd
hosts:
- localhost:27017
production:
sessions:
default:
uri: <%= ENV['MONGO_URL'] %>
I temporarily solved the issue by placing a:
options:
raise_not_found_error: false
in production:
Also moved out a few scripts from jobs/ folder. It worked then.
Can anyone enlighten me on this?
I think your ENV['MONGO_URL'] contains something that is not correct.
Try to use your development database in production:
production:
sessions:
default:
database: tgmd
hosts:
- localhost:27017
if it work. Check your ENV['MONGO_URL']

how to configure database.yml in windows 7 to use postgis

My database.yml development section looks like this:
development:
adapter: postgis
encoding: unicode
database: openData_development
pool: 5
username: postgres
password: test1234
schema_search_path: "public,postgis"
script_dir: C:\Program Files (x86)\PostgreSQL\9.2\share\contrib\postgis-2.0
rake db:create worked, the openData_development schema was created.
but i cant use postgis...
If i want to create a table like:
CREATE TABLE bezirks (
id integer NOT NULL,
name character varying(255),
beznr integer,
district_code integer,
main_id integer NOT NULL,
latlon geometry,
CONSTRAINT enforce_dims_latlon CHECK ((st_ndims(latlon) = 2)),
CONSTRAINT enforce_geotype_latlon CHECK (((geometrytype(latlon) = 'MULTIPOLYGON'::text) OR (latlon IS NULL))),
CONSTRAINT enforce_srid_latlon CHECK ((st_srid(latlon) = 4326))
);
I get the error:
ERROR: Type »geometry« doesnt exist
LINE 27: latlon geometry,
using the postgis adapter does not automagically install the postgis extensions on your db.
You have to do it yourself ; one solution to help automation of this is to create a 'bare' db on your server named 'template_postgis', install the postgis extensions on it as you would usually do (follow the installation guide on postgis website), and then add this to your database.yml:
development:
template: template_postgis
do it on every dbs so rails will create them as a copy of this template.

Resources