Ruby: SQLite3::BusyException: database is locked: - ruby-on-rails

Ran into this error message whilst developing tonight: SQLite3::BusyException: database is locked:
I have two models:
Podcasts have many Tracks
Tracks belong to Podcasts.
Podcast files are hosted on mixcloud.
To create a Podcast:
user submits a url for a podcast on mixcloud
rails app grabs json feed associated with url
json is used to set attributes (title, image etc) on the new Podcast object
I'm trying to get my rails app to take advantage of the fact that the json feed also details the names (and artists) of the Tracks that belong to this Podcast.
I thought the following before_validation method would automatically create all associated Tracks whenever we create a new Podcast.
class Podcast < ActiveRecord::Base
attr_accessible :mixcloud_url, :lots, :of, :other, :attrs
has_many :tracks
before_validation :create_tracks
def create_tracks
json = Hashie::Mash.new HTTParty.get(self.json_url)
json.sections.each do |section|
if section.section_type=="track"
Track.create(:name=>section.track.name, :podcast_id=>self.id)
end
end
end
end
How can I get round this? It looks like rails (or sqlite3) doesn't like me creating new instances of an associated model in this way. How else can I do this? I suspect this is as much a rails problem as an sqlite3 one. I can post more code if it's gonna help.

For anyone else encountering this issue with SQLite locking in development when a Rails console is open, try this:
Just run the following:
ActiveRecord::Base.connection.execute("BEGIN TRANSACTION; END;")
For me anyway, it appears to clear any transaction that the console was holding onto and frees up the database.
This is especially a problem for me when running delayed_job, which seems to fail at closing the transaction quite often.

SQLite is not really supposed to be used for concurrent access which is the issue you are running into here. You can try increasing the timeout in your database.yml file which may be a workaround for you in this case. However, I would recommend you switch to another database that supports multiple connections like MySQL or PgSQL.

For me...the issue I had was that it seemed that the Rails Console I had open for a while was locking up a connection with SQLite.
So once I exited that console, and restarted my webserver (Thin) it worked perfectly.
I tried #trisweb's suggestion but it didn't work for me.

I had the same
ActiveRecord::StatementInvalid: SQLite3::BusyException: database is
locked: INSERT INTO "users" ("created_at", "email", "name",
"password_digest", "updated_at") VALUES (?, ?, ?, ?, ?)"
issue. I tried every way around found in Google and I failed. The problem was solved for me when I closed my SQLite Database Browser.

Make sure you don't have 2 guards or several consoles running.
If you want make sure desperately see the "No Name's" answer above.
You can also try increasing pool:
for example:
change test section in your config/database.yml as below
test:
adapter: sqlite3
database: db/test.sqlite3
pool: 50
timeout: 5000

Probably you have a Rails console open on another bash, if so you have to close it (ctrl+D).

actually for me, I found killing rails help to fix this problem.
use "ps aux | grep rails" to find out ongoing rails process id.
then use
"kill -9 [rails-pid]"
to kill processes.
Then it will work

It is most likely not related to rails code. Using at the same time the console with the sandbox option (rails console --sandbox), makes the problem systematic with SQLite, since the console is basically waiting to quit to rollback everything.
The solution above from #trisweb will not work in this case, but quitting the console will.

my trouble is: I opened a database management program named "DB Browser for SQlite". Closed this database management program, and problem solved.

Yes this is an old question and there are many answers on here already. But none of them worked for me, meaning it took me a long time to finally figure out the problem. I found what worked and will share it in case it is what might be causing the problem for you too.
I was using the SQLITE Browser (its a GUI database browser). I will refer to it as "GUI" here (to prevent confusion with the word browser being your localhost::8000 chrome browser or whatever.
http://sqlitebrowser.org/
I was monitoring what was being written to the database and had the GUI open while my rails app ran in my chrome browser. I would refresh the GUI to see if it was adding the data as I expected it to.
As a matter of debugging I had decided to delete a row from the SQLite GUI so I could see if my app would react appropriately to the row being missing now.
As it turns out, the SQLite Browser does not actually delete the row (causing confusion on my end as to why my app was acting like the row was still there, even though it was visually missing on the GUI). Anyway after 30 minutes of frustration I closed the SQLite GUI and then got a notice that asked if i wanted to save any changes to the database that I made. I naively clicked "No" and closed the app.
Apparently what happens is that the GUI then locked the database because there were rows in my database that had been sort of "soft-deleted" without committing to the delete. So the GUI was (for lack of a better term) holding the database in Limbo.
This explains why a) my app wasnt acting like the row was missing, because it hadn't actually been deleted yet, and B) explains why the database locked up. It was still waiting for me to commit the deletes.
So to solve the problem, I simply opened up the GUI once again and deleted the same row and then closed the GUI and this time I clicked "Yes" when asking to save changes to the database. It saved the delete and unlocked the database and now my app works!
I hope this helps someone else that might be having the same issue but was using the SQLite Browser GUI interface. This might be what is locking your database.

I had the same issue. For those with SQLite Database Browser. I did not need to close SQLite Database Browser. I only had to click the "Write Changes" button. It is highlighted and needs to not be highlighted.

SQLite has troubles with concurrency.
I changed sqlite on Postgresql and the issue is gone

This happens when you make any changes manually directly into the SQlite DB Browser (like delete a row or change the value of any column) and forget to save those changes. Any changes made need to be saved (ctrl + s). If not saved, SQLite locks the Database until u save those changes.
I did the same and my issue got resolved!

I was using DB Browser for SQLite and rails console simultaneously. Closing the DB Browser for SQLite fixed the issue for me.

try restarting the server
or closing any running rails console,
worked for me

Your .sqlite3 file must be saved.
Go to DB Browser for SQlite and make ctrl+s or File->Write Changes.

Try wrap cycle in a single transaction:
def create_tracks
json = Hashie::Mash.new HTTParty.get(self.json_url)
Track.transaction do
json.sections.each do |section|
if section.section_type=="track"
Track.create(:name=>section.track.name, :podcast_id=>self.id)
end
end
end
end
(see Track.transaction)

You may also have enabled threads when parallelizing your tests. Remember to disable the option in test/test_helper.rb :
parallelize(workers: :number_of_processors, with: :threads)
to
parallelize(workers: :number_of_processors)
https://edgeguides.rubyonrails.org/testing.html#parallel-testing-with-threads

1. bin/rails console
2. exit
Go into the rails console and type exit, and press enter.
Done!

Sadly, many of these solutions did not work for me.
I was lucky. Since this was happening while I was running tests, I just did a simple DROP and CREATE on the DB.
$ rake db:drop RAILS_ENV=test
Dropped database 'db/test.sqlite3'
$ rake db:create RAILS_ENV=test
Created database 'db/test.sqlite3'
$ rake db:migrate RAILS_ENV=test
== 20220310061725 Tables: migrating ======================================
....
== 20220310061725 Tables: migrated (0.0027s) =============================
...etc
Not the best solution, but it works in a pinch.
If this was in your development environment, and you HAD to do this, I would seriously invest in creating a data seed for your DBs so you can get up to speed again.
rake db:drop
rake db:create
rake db:migrate
rake db:seed

I'd like to talk about the case of rails controller opened.
In my own case, I had opened rails controller with the --sandbox option.
After reading that I had to close the rails console. Closing alone didn't fix it. I had to restart my PC to make sure all processes got terminated. That resolved it.

Related

What does the "database is locked" error message mean in Ruby on Rails testing?

I am following Hartl's Rails Tutorial and using Rails 6. I keep getting persistent errors that seem to pop up at random whenever I run tests - random because the consecutive tests sometimes indicate errors in different areas. These tests are also very very slow - > 30 minutes sometimes. Has anyone encountered this? What could I be doing wrong? And now for the red herring: I am using Win 8.1 :)
The common thing about these errors messages is that they all contain a "RuntimeErroer: database is locked" message. Here's one of them:
ERROR["test_email_validation_should_reject_invalid_addresses",
#<Minitest::Reporters::Suite:0x000000000c9b29c0 #name="UserTest">, 608.7059665989946]
test_email_validation_should_reject_invalid_addresses#UserTest (608.71s)
RuntimeError: RuntimeError: database is locked
I've been fighting this same error for quite sometime. It actually go so bad it roadblocked me from really moving forward in Hartl's Rails Tutorial.
Edit: Found a much better answer that simply resolves the problem instead of playing with settings I probably don't really understand.
See -> https://stackoverflow.com/a/62730905/10463184
My only contribution, as a windows user, I found that commenting out the entire line in test/test_help.rb...
parallelize(workers: :number_of_processors, with: :threads)
Resolved the issue. Trying the setting suggested at the link resulted in a "UNIXServer is required (LoadError)" error.
Here's a solution to one of my challenges - the interminablly slow test speed. In the config/database.yml I added (to the test.sqlite part) the following lines:
database: db/test.sqlite3
pool: <%= ENV.fetch("RAILS_MAX_THREADS") { 5 } %>
timeout: 3000
The test duration dropped from minutes to mere seconds. Victory!
Alas, the "RuntimeError: database is locked" messages are still there, however.
In windows you can try this program http://www.nirsoft.net/utils/opened_files_view.html to find out the process is handling db file. Try closed that program for unlock database
In Linux and macOS you can do something similar, for example, if your locked file is development.db:
$ fuser development.db
This command will show what process is locking the file:
> development.db: 5430
Just kill the process...
kill -9 5430
...And your database will be unlocked.
Or
it should be a database's internal problem...
For me it has been manifested after trying to browse database with "SQLite manager"...
So, if you can't find another process connect to database and you just can't fix it, just try this radical solution:
Provide to export your tables (You can use "SQLite manager" on Firefox)
If the migration alter your database scheme delete the last failed migration
Rename your "database.sqlite" file
Execute "rake db:migrate" to make a new working database
Provide to give the right permissions to database for table's importing
Import your backed up tables
Write the new migration
Execute it with "rake db:migrate"
I found myself in the same situation.
In my case this error happened because the tests run in parallel by default on MiniTest, which means that they also run in parallel in rails. When this happens and there are tests that make transactions on the tests db, the db locks itself causing the error.
The solution that worked for me was setting the attribute "use_transactional_tests" to false in the test class.
You can see the proper usage and an example in the docs.
In case anyone else makes the obvious mistake that I did, be sure to exit out of any rails console sessions in your terminals before trying to delete on your live site or in testing.

Rails -- working with an existing db, cannot run migrations without losing data

I am a beginner working with Rails 4.1. I am trying to integrate an existing sqlite3 db with my app in the development environment.
To try to get this to work, I've followed the steps listed at Joel Berghoff's blog (for MySQL):
Reference the db in config/database.yml
Run “rake db:schema:dump”
Convert schema.rb into db/migrate/001_create_database.rb
The issue I am facing is, whenever I run "rake db:migrate" the entire db refreshes and I lose all the pre-populated data. I got around this for awhile by running migrations first, then replacing the blank db that was generated with my pre-populated copy -- this allowed me to play around with my models in the rails console and see the data. However when I try to boot up the server on my local machine, I get a message that migrations are pending.
I am not quite sure what to do here...I've read that I should be seeding the db from "rake db:seed", but my existing db is quite large -- almost 1mm records, and when I attempted this (in albeit clumsy fashion) it ran for over 3 hours before I gave up. Any guidance on how to proceed is much appreciated!
Migrates should be used to create and changes the tables and fields, not load data, as you can see here Ruby on Rails Guides
If you want to import data you could do it on the seeds, but in your specific case it seems to me that you should create a dump from your origin database and load it on the target database.
Here is a tutorial sqlite3: how to import/export data from/to a file
sqllite3 and MySQL are different things. My guess is you think you are connected to sqllite db but in reality you are connected to an empty MySQL db.
Look into ActiveRecord migrations.
Here is a rails doc on ActiveRecord Migrations: http://guides.rubyonrails.org/migrations.html

SQLite 3 database not being read properly

I had to create a sqlite3 database in my rails 3.1.1 app manually, converting from a mysql database using the mysql2sqlite tool, and then running:
sqlite3 development.sqlite3 < mysql2sqlite_dump.sql
Anyways, the development.sqlite3 file is there, and the server starts up fine and all the pages are loading fine (no database errors about missing tables or anything), but the data isn't being read... just as if it wasn't there. Even running Event.all, for example, in console gives me :
Event Load (0.1ms) SELECT "events".* FROM "events"
=> []
When there should actually be several events. Looking in the sqlite3 file, I can see that all the information is there, but it's just not being read. I will post part of the database file if anybody requests it.
My question is: Why isn't the database being read properly, and how can I make it so?
Thanks.
Well, I seemed to have solved the problem. The issue was that mysqldump (being executed inside of the mysql2sqlite script) was dropping and then creating the tables (which is default), and that was conflicting with the schema file. So, I created the database according the schema file, and then ran the script again with the --no-create-info option added for mysqldump so that it would only insert the information.
Thanks for the help everybody!

Is there a simple way to test a migration before running it?

In other words I'd like to know the best way to make sure that self.down actually rolls back self.up before running the migration in question.
What can I do if I need to rollback a migration but self.down doesn't serve the purpose?
Which is the best practice when dealing with potentially destructive migrations?
Just a database backup?
Thanks,
Duccio.
You should be developing on a development database which should not contain live data. Therefore it should not matter if the data is destroyed as you can easily generate it again?
A database backup might be appropriate if you find yourself in a situation where your development data is important but not ideal.
Typically migrations should contain only schema changes. In that case it should be very safe & easy to run the migrations in the dev/test environment. If something goes wrong you can alway re-create the database and populate it with some test data. But if you have some data related migrations to be tested, things might go wrong when you actually run them on production.
In that case as you mentioned database backup is what you should rely on. Come with a proper & quick restore mechanism before deploying.
To make sure the migrations behave as you want you should experiment in your development environment.
Run the command
rake -T
to show you the available tasks such as
rake db:migrate
or
rake db:rollback
Each migration runs inside a transaction. Keep that in mind. That means that if something would go wrong inside a single migration, the migration is rolled back (and if there are any following they are not executed).
To test migrations, be it up or down I insert a lot of puts-statements, to check everything has worked has supposed to, and then in my last line I raise an exception. This will make rails think the migration has failed, and it will rollback the operation (as if it never happened).
When I am sure everything works as it should, I remove the raise line and let the migration really work.
In your case, you would test with the raise, remove the raise and NOT run it again I assume :)
Hope this helps.

Rails rake db:migrate has no effect

I made a new Rails 3 app today, added a simple migration, and for some reason, nothing happens when I do rake db:migrate. It simply pauses a few seconds, then returns to the command prompt, with no errors or anything. Schema.rb and the database stay empty.
Any ideas what could be going on? I've made many apps and never had this problem. Everything is a totally standard setup too.
There's a few reasons why your migrations won't run, but the most common is that the system is already under the impression that all the migrations you've defined have already run.
Each migration creates an entry in the schema_migrations table with the version column corresponding to the identifier number. If you want to force a migration to re-run you can usually back it out and retry it. For example, if you had 20100421175455_create_things.rb then you would re-run it using:
rake db:migrate:redo VERSION=20100421175455
A common situation is that your migration has failed to run in the first place, that it generated an exception for instance, and yet Rails still considers it complete. To forcibly re-run a migration, delete the corresponding record from the schema_migrations table and run rake db:migrate again.
One way to avoid this kind of problem in the future is to define your migrations with an automatic back-out procedure:
class CreateThings < ActiveRecord::Migration
def self.up
# ... (migration) ...
rescue
# If an exception occurs, back out of this migration, but ignore any
# exceptions generated there. Do the best you can.
self.down rescue nil
# Re-raise this exception for diagnostic purposes.
raise
end
end
If you have a mistake in your migration you will see the exception listed on the console. Since the migration has automatically been rolled back you should be able to run it again and again until you get it right.
I faced the same problem. I did a kind of a short hack that helped me. I am posting it just in case anyone wants a short and sweet solution. I agree with what Tadman is saying
"the system is already under the impression that all the migrations
you've defined have already run"
What I did was to change the name of the migrate file in the /app_folder/db/migrate folder. I think the numeric part in the name of the ruby migrate file is the time at which the file was created.
You can add say 1 , to the filename, every time you want to re-run the migrate. After changing the name drop/delete the table (I used mysql command line tool for deleting) and then run rake db:migrate and the migrations should be done.
Calling spring stop might solve your problems.
Well, I found out what was causing my problem. I'm using the slim_scrooge gem and commenting it out makes everything proceed normally. Don't know why though...
I faced similar problem today while migrating plugin for Redmine using
rake redmine:plugins:migrate RAILS_ENV=production NAME=plugin_name
where plugin_name is actually plugin name defined in init.rb of the plugin.
I struggled for 4 hours and finally figured out that my plugin directory name was not the same as the plugin name (note redmine_ prefix):
~/redmine/plugins/redmine_plugin_name
So, make sure your plugin is placed in folder named after plugin name. I believe it applies to other rails apps as well.

Resources