I have strange problems with parallel run of tasks using dblink extension on PostgreSQL 9.6.1. Looks like extension in some cases cannot recognize that task in connection already ended and still waits for result. Therefore programs which used to run absolutely perfectly on PG 9.5 sometimes keep hanging indefinitely on PG 9.6.
Procedure:
connection is opened using "perform dblink_connect"
query is send using 'dblink_send_query"
other tasks are send to different connections
program takes first connection and issues "dblink_get_result" and waits for output
after it takes second connection etc. - some of those connections already ended so they should give result immediately after start of "dblink_get_result" but in some cases it does not work on PG 9.6 and check just hangs...
Are there some changes in dblink behavior in PG 9.6 I am not aware of?
Related
We have completed upgrading our app from Rails 4.2 to 5.2. When we run load testing on the 5.2 version it can only handle half the load of the 4.2 version. In looking at NewRelic stats during the load tests it seems to be slower everywhere - pretty much every request, ActiveRecord calls, redis calls, ruby, etc. We have confirmed it is not related to other upgrades that happened in addition - ruby upgrade, upgrading pg gem, or upgrading puma. While researching, the only performance issues I have found related to the upgrade have been fixed.
Has anyone run into something similar or have pointers on where to look?
What we have tried so far:
1. Check non-rails related upgrades that happened at the same time:
- Upgrade 4.2 branch to same version of ruby to see if that has any impact (no impact)
- Downgrade puma and pg gems in Rails 5 branch (no impact)
2. Examine performance traces for slower transactions and DB queries. Remove the slowest interactions from the load test to see if the overall slowest continues (it does).
3. Test if slowness appears in Rails 5.1 (it does).
What we are planning to try:
1. Test if slowness appears in Rails 5.0.
2. See if slowest can be detected in single user use rather than load test.
3. Use https://github.com/tleish/ruby-prof-rails to see if we can get more statistics to examine.
4. Downgrade all gems except the ones we absolutely need for the Rails 5 upgrade and see if problem still exists.
This ended up being a combination of Rack::Timeout, Heroku and Puma. Under heavy load, we would sometimes hit our Rake::Timeout value of 28 seconds. For some reason after the upgrade, the 2 seconds between the Rake::Timeout value and Heroku's 30 second router timeout (H12) was not enough. As a result, processes were getting killed by Heroku before Rake::Timeou which was causing a cascade effect in Puma that made a ton of other requests on the same server also timeout. We fixed it by adjusting our Rake::Timeout value to 25 seconds and everything worked.
I am following Hartl's Rails Tutorial and using Rails 6. I keep getting persistent errors that seem to pop up at random whenever I run tests - random because the consecutive tests sometimes indicate errors in different areas. These tests are also very very slow - > 30 minutes sometimes. Has anyone encountered this? What could I be doing wrong? And now for the red herring: I am using Win 8.1 :)
The common thing about these errors messages is that they all contain a "RuntimeErroer: database is locked" message. Here's one of them:
ERROR["test_email_validation_should_reject_invalid_addresses",
#<Minitest::Reporters::Suite:0x000000000c9b29c0 #name="UserTest">, 608.7059665989946]
test_email_validation_should_reject_invalid_addresses#UserTest (608.71s)
RuntimeError: RuntimeError: database is locked
I've been fighting this same error for quite sometime. It actually go so bad it roadblocked me from really moving forward in Hartl's Rails Tutorial.
Edit: Found a much better answer that simply resolves the problem instead of playing with settings I probably don't really understand.
See -> https://stackoverflow.com/a/62730905/10463184
My only contribution, as a windows user, I found that commenting out the entire line in test/test_help.rb...
parallelize(workers: :number_of_processors, with: :threads)
Resolved the issue. Trying the setting suggested at the link resulted in a "UNIXServer is required (LoadError)" error.
Here's a solution to one of my challenges - the interminablly slow test speed. In the config/database.yml I added (to the test.sqlite part) the following lines:
database: db/test.sqlite3
pool: <%= ENV.fetch("RAILS_MAX_THREADS") { 5 } %>
timeout: 3000
The test duration dropped from minutes to mere seconds. Victory!
Alas, the "RuntimeError: database is locked" messages are still there, however.
In windows you can try this program http://www.nirsoft.net/utils/opened_files_view.html to find out the process is handling db file. Try closed that program for unlock database
In Linux and macOS you can do something similar, for example, if your locked file is development.db:
$ fuser development.db
This command will show what process is locking the file:
> development.db: 5430
Just kill the process...
kill -9 5430
...And your database will be unlocked.
Or
it should be a database's internal problem...
For me it has been manifested after trying to browse database with "SQLite manager"...
So, if you can't find another process connect to database and you just can't fix it, just try this radical solution:
Provide to export your tables (You can use "SQLite manager" on Firefox)
If the migration alter your database scheme delete the last failed migration
Rename your "database.sqlite" file
Execute "rake db:migrate" to make a new working database
Provide to give the right permissions to database for table's importing
Import your backed up tables
Write the new migration
Execute it with "rake db:migrate"
I found myself in the same situation.
In my case this error happened because the tests run in parallel by default on MiniTest, which means that they also run in parallel in rails. When this happens and there are tests that make transactions on the tests db, the db locks itself causing the error.
The solution that worked for me was setting the attribute "use_transactional_tests" to false in the test class.
You can see the proper usage and an example in the docs.
In case anyone else makes the obvious mistake that I did, be sure to exit out of any rails console sessions in your terminals before trying to delete on your live site or in testing.
I am working with a UDP server in rails and I have the server called from my config/initializers folder.
If I run the code thread_a=Thread.new{UDPserver()} then the the code works fine however the code eventually switches off, I presume once my rails script stops running.
If I write
thread_a=Thread.new{UDPserver()}
thread_a.join
then my whole web program doesn't work and I presume this is because my program is waiting till the threads finish (which they don't) until it loads up the rest of the program.
What am I to do?
Notes:
The reason I am running a thread is because I need my program to be constantly waiting for UDP messages and then to react. Also, I need my standard we program to be up at the same time.
ruby 1.9.3p194
Rails 3.2.12
I'm frequently starting up rails console or rails server or using other command line ruby apps. The bootstrap takes several seconds, which becomes tedious after a while.
Is there any way I can either run a compiled version, or keep it loaded using something like spork, so running 'heroku logs' runs instantly, rather than taking 10 seconds to start up?
Run a compiled version of what?
Rails server and rails console take long time to load because they have to load up Ruby on Rails plus whatever your environment requires. If you want to keep instances running then just open up a terminal window, load your console and server, and never close them.
Have you thought of trying vagrant? it will keep your instance running to you want them to stop.
I am working on building a Ruby on Rails v2.3.8 real estate application. I've written a rake task to pull in the property listings and photos. The app uses rets4r (specifically josephholsten-rets4r) to talk to a RETS-based server and Paperclip is used to manage the property photos and push them up to S3.
The problem I'm seeing is the rake task never completes. It will always crash at some point and it's never the same spot. One time it crashed after 45 min, after some minor changes (added some begin/rescue blocks to try and track down the error), it seems to run longer (2-12 hours), but still crashes.
The error I get is always "Rake aborted" and then "thread [memory address] tried to join itself." Looking into the logs provides nothing helpful. Just the standard Paperclip log messages. I'm also not deliberately trying to do anything multi-threaded in the rake task.
Update: I was using the gem version 1.1.17 of rets4r. However, I notice on the Github page for the project, there have been quite a few commits since that version was tagged. One of those commits talks about removing the use of threads.