Rails 4.2 Postgres 9.4.4 statement_timeout doesn't work - ruby-on-rails

I am trying to set a statement_timeout. I tried both setting in database.yml file like this
variables:
statement_timeout: 1000
And this
ActiveRecord::Base.connection.execute("SET statement_timeout = 1000")
Tested with
ActiveRecord::Base.connection.execute("select pg_sleep(4)")
And they both don't have any effect.
I am running postgres 10 in my local and the statement_timeouts works just expected. But on my server that is running postgres 9.4.4, it simply doesn't do anything.
I've check Postgres' doc for 9.4 and statement_timeout is available. Anyone can shed some light?

I wasn't able to replicate this issue locally using: Postgresql 9.4.26. But it might be useful to share what I've tried and some thoughts around the server issue.
Here is what I've tried (a useful bit might be a query to verify the PG version from rails):
# Confirming I am executing against 9.4.x PG:
irb(main):002:0> ActiveRecord::Base.connection.execute("select version()")
(10.8ms) select version()
=> #<PG::Result:0x00007ff74782e060 status=PGRES_TUPLES_OK ntuples=1 nfields=1 cmd_tuples=1>
irb(main):003:0> _.first
=> {"version"=>"PostgreSQL 9.4.26 on x86_64-apple-darwin18.7.0, compiled by Apple clang version 11.0.0 (clang-1100.0.33.17), 64-bit"}
# Set timeout:
irb(main):004:0> ActiveRecord::Base.connection.execute("SET statement_timeout = 1000")
(0.4ms) SET statement_timeout = 1000
=> #<PG::Result:0x00007ff7720a3d88 status=PGRES_COMMAND_OK ntuples=0 nfields=0 cmd_tuples=0>
# Confirm it works - it is ~1s and also stacktrace is pretty explicit about it:
irb(main):005:0> ActiveRecord::Base.connection.execute("select pg_sleep(4)")
(1071.2ms) select pg_sleep(4)
.... (stacktrace hidden)
ActiveRecord::StatementInvalid (PG::QueryCanceled: ERROR: canceling statement due to statement timeout)
: select pg_sleep(4)
Here is what to try
Since the issue occurs on server only and since statement_timeout works on other minor version and locally, one thing that comes to mind is the lack of privileges to update statement_timeout from where it is attempted. Perhaps rails pg login used to make db connection is not allowed to update that setting.
The best would be to verify that either via rails console on a server:
irb(main):004:0> ActiveRecord::Base.connection.execute("SET statement_timeout = 1000")
irb(main):004:0> irb(main):003:0> ActiveRecord::Base.connection.execute("show statement_timeout").first
(0.2ms) show statement_timeout
=> {"statement_timeout"=>"1s"}
Or, it can be checked directly via psql console (some deployments allow this too):
psql myserveruser # if this was heroku's pg: heroku pg:psql
postgres=# set statement_timeout = 1000;
SET
postgres=# select pg_sleep(4);
ERROR: canceling statement due to statement timeout
Time: 1068.067 ms (00:01.068)
Other thing to keep in mind (taken from https://dba.stackexchange.com/a/83035/90903):
The way statement_timeout works, the time starts counting when the
server receives a new command from the client...
And if a function does SET statement_timeout = 100; it will have an effect only starting at
the next command from the client.

Related

Configuring Backup gem in Rails 5.2 - Performing backup of PostgreSQL database

I would like to perform a regular backup of a PostgreSQL database, my current intention is to use the Backup and Whenever gems. I am relatively new to Rails and Postgres, so there is every chance I am making a very simple mistake...
I am currently trying to setup the process on my development machine (MAC), but keep getting an error when trying to connect to the database.
In the terminal window, I have performed the following to check the details of my database and connection:
psql -d my_db_name
my_db_name=# \conninfo
You are connected to database "my_db_name" as user "my_MAC_username" via socket in "/tmp" at port "5432".
\q
I have also manually created a backup of the database:
pg_dump -U my_MAC_username -p 5432 my_db_name > name_of_backup_file
However, when I try to repeat this within db_backup.rb (created by the Backup gem) I get the following error:
[2018/10/03 19:59:00][error] Model::Error: Backup for Description for db_backup (db_backup) Failed!
--- Wrapped Exception ---
Database::PostgreSQL::Error: Dump Failed!
Pipeline STDERR Messages:
(Note: may be interleaved if multiple commands returned error messages)
pg_dump: [archiver (db)] connection to database "my_db_name" failed: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/tmp/pg.sock/.s.PGSQL.5432"?
The following system errors were returned:
Errno::EPERM: Operation not permitted - 'pg_dump' returned exit code: 1
The contents of my db_backup.rb:
Model.new(:db_backup, 'Description for db_backup') do
##
# PostgreSQL [Database]
#
database PostgreSQL do |db|
# To dump all databases, set `db.name = :all` (or leave blank)
db.name = "my_db_name"
db.username = "my_MAC_username"
#db.password = ""
db.host = "localhost"
db.port = 5432
db.socket = "/tmp/pg.sock"
# When dumping all databases, `skip_tables` and `only_tables` are ignored.
# db.skip_tables = ["skip", "these", "tables"]
# db.only_tables = ["only", "these", "tables"]
# db.additional_options = ["-xc", "-E=utf8"]
end
end
Please could you suggest what I need to do to resolve this issue and perform the same backup through the db_backup.rb code
In case someone else gets stuck in a similar situation, the key to unlocking this problem was the lines:
psql -d my_db_name
my_db_name=# \conninfo
I realised that I needed to change db.socket = "/tmp/pg.sock" to db.socket = "/tmp", which seems to have resolved the issue.
However, I don't understand why the path on my computer differs to the default as I didn't do anything to customise the installation of any gems or the Postgres App

Rails test DB constantly losing my PostgreSQL functions

My project uses a few custom PostgreSQL stored functions for some features that would be a pain in raw SQL or ActiveRecord. Every now and then I will run the RSpec test suite, and find that all my stored functions have been blown away. Re-running the migrations to create them fixes the problem, but "rake db:structure:load" does NOT.
I am deeply confused. I never drop either the dev or test database unless this happens, but my functions are like Schrodinger's PL/pgSQL. I am REALLY hoping this never happens in production.
Here is an example of a failing test and my attempts to fix it:
ActiveRecord::StatementInvalid:
PG::UndefinedFunction: ERROR: function round_half_down(numeric) does not exist
# Damn. We have to drop the database so we can reload structure.sql:
$ RAILS_ENV=test rake db:drop
$ RAILS_ENV=test rake db:create
# load structure.sql instead of schema.rb:
$ RAILS_ENV=test rake db:structure:load
# Not fixed:
ActiveRecord::StatementInvalid:
PG::UndefinedFunction: ERROR: function round_half_down(numeric) does not exist
$ RAILS_ENV=test rake db:migrate:redo VERSION=20160421184708
== 20171002190107 CreateRoundHalfDownFunction: reverting ======================
-- execute("DROP FUNCTION IF EXISTS round_half_down(numeric)")
-> 0.0004s
== 20171002190107 CreateRoundHalfDownFunction: reverted (0.0005s) =============
== 20171002190107 CreateRoundHalfDownFunction: migrating ======================
-- execute("CREATE OR REPLACE FUNCTION ROUND_HALF_DOWN(NUMERIC)\n RETURNS NUMERIC LANGUAGE SQL AS\n$FUNC$\n SELECT CASE WHEN ($1%1) < 0.6 THEN FLOOR($1) ELSE CEIL($1) END;\n$FUNC$\n")
-> 0.0014s
== 20171002190107 CreateRoundHalfDownFunction: migrated (0.0014s) =============
Now it is fixed!
Yes, I verified that the function is present in structure.sql:
--
-- Name: round_half_down(numeric); Type: FUNCTION; Schema: public; Owner: -
--
CREATE FUNCTION round_half_down(numeric) RETURNS numeric
LANGUAGE sql
AS $_$
SELECT CASE WHEN ($1%1) < 0.6 THEN FLOOR($1) ELSE CEIL($1) END;
$_$;
For the record, this stopped happening to me with newer versions of the pg gem and PostgreSQL itself. I also added config.active_record.schema_format = :sql to application.rb, because my application makes heavy use of Postgres-specific features, and a number of stored functions.

PG::UnableToSend: no connection to the server in Rails

I have a production server running ubuntu 14.04, Rails 4.2.0, postgresql 9.6.1 with gem pg 0.21.0/0.20.0. In last few days, there is constantly error with accessing to a table customer_input_datax_records in psql server.
D, [2017-07-20T18:08:39.166897 #1244] DEBUG -- : CustomerInputDatax::Record Load (0.1ms) SELECT "customer_input_datax_records".* FROM "customer_input_datax_records" WHERE ("customer_input_datax_records"."status" != $1) [["status", "email_sent"]]
E, [2017-07-20T18:08:39.166990 #1244] ERROR -- : PG::UnableToSend: no connection to the server
: SELECT "customer_input_datax_records".* FROM "customer_input_datax_records" WHERE ("customer_input_datax_records"."status" != $1)
The code which call to access the db server is with Rufus scheduler 3.4.2 loop:
s = Rufus::Scheduler.singleton
s.every '2m' do
new_signups = CustomerInputDatax::Record.where.not(:status => 'email_sent').all
.......
end
After restart the server, usually there is with first request (or a few). But after some time (ex, 1 or 2 hours), the issue starts to show up. But the app seems running fine (accessing records with read/write & creating new). There are some online posts about the error. However the problem seems not the one I am having. Before I re-install the psql server, I would like to get some ideas about what causes the no connection.
UPDATE: database.yml
production:
adapter: postgresql
encoding: unicode
host: localhost
database: wb_production
pool: 5
username: postgres
password: xxxxxxx
So, the error is "RAILS: PG::UnableToSend: no connection to the server".
That reminds me of Connection pool issue with ActiveRecord objects in rufus-scheduler
You could do
s = Rufus::Scheduler.singleton
s.every '2m' do
ActiveRecord::Base.connection_pool.with_connection do
new_signups = CustomerInputDatax::Record
.where.not(status: 'email_sent')
.all
# ...
end
end
digging
It would be great to know more about the problem.
I'd suggest trying this code:
s = Rufus::Scheduler.singleton
def s.on_error(job, error)
Rails.logger.error(
"err#{error.object_id} rufus-scheduler intercepted #{error.inspect}" +
" in job #{job.inspect}")
error.backtrace.each_with_index do |line, i|
Rails.logger.error(
"err#{error.object_id} #{i}: #{line}")
end
end
s.every '2m' do
new_signups = CustomerInputDatax::Record.where.not(:status => 'email_sent').all
# .......
end
As soon as the problem manifests itself, I'd look for the on_error full output in the Rails log.
This on_error comes from https://github.com/jmettraux/rufus-scheduler#rufusscheduleron_errorjob-error
As we discuss in the comments, the problem seems related to your rufus version.
I would suggest you to check out whenever gem and to invoke a rake task instead of calling directly the activerecord model.
It could be a good idea, however, to open an issue with the traceback of your error in the rufus-scheduler repo on github (just to let then know...)

Trying to get Postres and Postgis running in Rails app

I am completely new to Postgresql, Postgis, and SQL and have a little bit of Rails knowledge. I am trying to get an existing app going on my local and am currently receiving the following messages when running a rake task to set up my database.
Blane-Cordess-MacBook:ajungo blanecordes$ rake postgres:create_postgis_template --trace
(in /Users/blanecordes/ajungo)
** Invoke postgres:create_postgis_template (first_time)
** Execute postgres:create_postgis_template
Enter your postgres username: blane
Password for user blane:
psql (9.2.1, server 9.1.4)
WARNING: psql version 9.2, server version 9.1.
Some psql features might not work.
You are now connected to database "template1" as user "blane".
psql:assets/sql/postgis_template_osx.sql:2: ERROR: permission denied to create database
psql:assets/sql/postgis_template_osx.sql:6: ERROR: permission denied for relation pg_database
psql:assets/sql/postgis_template_osx.sql:7: \connect: FATAL: database "template_postgis" does not exist
My postgis_template_osx.sql file is the following:
\c template1
CREATE DATABASE template_postgis WITH template = template1;
-- set the 'datistemplate' record in the 'pg_database' table for
-- 'template_postgis' to TRUE indicating its a template
UPDATE pg_database SET datistemplate = TRUE WHERE datname = 'template_postgis';
\c template_postgis
CREATE LANGUAGE plpgsql ;
\i /usr/local/Cellar/postgis/2.0.1/share/postgis/postgis.sql;
\i /usr/local/Cellar/postgis/2.0.1/share/postgis/spatial_ref_sys.sql;
-- 1.5.2
-- in a production environment you may want to
-- give role based permissions, but granting all for now
GRANT ALL ON geometry_columns TO PUBLIC;
GRANT ALL ON spatial_ref_sys TO PUBLIC;
-- vacuum freeze: it will guarantee that all rows in the database are
-- "frozen" and will not be subject to transaction ID wraparound
-- problems.
VACUUM FREEZE;
The PostgreSQL ROLE blane does not have permission to create databases. Either ALTER blane and add the CREATEDB option or use an existing administrator role that has that capability.
postgres=# ALTER ROLE blane WITH CREATEDB;

Texticle fuzzy search not finding anything in Ruby on Rails

I've added the following to my gemfile:
gem 'texticle', "2.0", :require => 'texticle/rails' # search gem
I then ran bundle install and bundle installed the gem
I'm using Rails 3.1.0 and I have am using a Postgres database.
I verify that I actually have the string I am looking for in the database table:
ruby-1.9.2-p290 :004 > Hotel.first
Hotel Load (0.4ms) SELECT "hotels".* FROM "hotels" LIMIT 1
=> #<Hotel id: 1, title: "Marriot Hotel", created_at: "2012-03-01 23:53:16", updated_at: "2012-03-01 23:53:16">
When I run `Hotel.search('e')
ruby-1.9.2-p290 :005 > Hotel.search(:title => 'e')
Hotel Load (1.4ms) SELECT "hotels".*, ts_rank(to_tsvector("hotels"."title"), to_tsquery('e')) AS "rank0.4785527956789428" FROM "hotels" WHERE (to_tsvector('english', "title") ## to_tsquery('e')) ORDER BY "rank0.4785527956789428" DESC
=> []
I get nothing. I tried running Hotel.search('e') and still nothing. If I try Hotel.search(:title => 'Marriot') then it works but the reason I am using Texticle is for the fuzzy search.
Am I missing any other configurations?
Thanks
There are a couple things I would suggest checking:
Verify an extension has been created manually for each database involved. I didn't have luck with 'texticle:install_trigram'. And if you're using Heroku, there may be additional steps, especially if you haven't migrated from your shared database.
Use the fuzzy_search method. In the latest version of Texticle, there are actually 3 new methods; the original #search method has been deprecated.
To manually install the extension:
Local Dev/Test
psql -U <username>
\l
\c db_development
CREATE EXTENSION pg_trgm;
\dx
\c db_test
CREATE EXTENSION pg_trgm;
\dx
\q
Heroku Integration/Production
heroku pg:psql
\dx
CREATE EXTENSION pg_trgm;
\dx
More?
PostgreSQL Trigrams (fuzzy search)
What's new in PostgreSQL 9.1 - PostgreSQL wiki
trigram | Tumblr
Installing PostgreSQL Trigram
Install Trigram Rake Task by benhamill · Pull Request #63 · tenderlove/texticle · GitHub
texticle:install_trigram - on Heroku? · Issue #15 · tenderlove/texticle · GitHub
New Texticle search methods (source code)
texticle/lib/texticle.rb at master · texticle/texticle · GitHub
My understanding of the way Texticle works is that it's going to do full text search on your string columns but not necessarily do fuzzy search by default. If you look at the query that it's generating in Postgres, it's looking for a match on 'e' rather than any word that contains the letter 'e'.
You can read the Postgres docs here:
http://www.postgresql.org/docs/9.1/static/datatype-textsearch.html
That said, I see support for prefix matching but not postfix in the docs, although I might be missing something.
What happens if you do Hotel.search('Marr:*') ?

Resources