I am running a rails 4 application with a large amount of data in my development db (postgres with postgis). In order to write tests I need the data from the development db.
Unfortunately I can not uses a loader/seed script since the data are coming from an api with quota limit. So it takes quite some time to collect the data, which is not wanted for test.
Therefore I was wondering how would it be possible to replicate data from the development db for testing?
Best,
Phil
Rails is just a framework - it doesn't have much to do with your actual data:
Every feature built into Rails is designed to help facilitate a more
flexible & smooth input of data into your system. This means that if
you want to transfer a large body of data for a Rails app, it will
likely be more efficient to do it at db level
We've migrated many db's before, and it's just a case of downloading an SQL file & then passing into the new db table. This varies for different SQL variants, but if you want to use Postgres, you'll be best using pg_dump & pgsql to restore:
pg_dump dbname > outfile
psql dbname < infile
I achieved this by creating a Bash script and using a database dump over ssh. The dump file is just text with create or replace statements, my example below is using MySql and I had to use sed to convert the escape characters. I am sure you could do something similar with Postgres. As part of my backup I also have to capture user maintained images which I do using rsync. I am loading the database into development but it could also be test.sqlite3 or any other database.
# Get database
ssh root#mywebsite.com \
'mysqldump -t --replace --compact --extended-insert=FALSE -u root -pDBPassword db_name' | \
sed "s/\\\\'/''/g" | sqlite3 ~/websites/mywebsite/db/development.sqlite3
# Get images
rsync -z root#mywebsite,com:/var/www/mywebsite/public/images/*.jpg \
~/websites/mywebsite/public/images/
Related
I'm trying to move a postgres database between machines as I move from one development platform to another. I have yaml_db gem installed on both machines.
On my old platform I do:
rake db:scheme:dump
rake db:data:dump
When I go to reload the database on my new machine I've discovered that my 2 dozen foreign_keys are preventing me for loading my data. What are my options?
You're copying a database, Rails really shouldn't have anything to do with the process (and as you're seeing, it just gets in the way).
Instead, put on your DBA hat and copy the database without bothering with Rails. Dump the data using pg_dump and then restore the data with pg_restore. The database's backup/restore tools know all about foreign keys, triggers, extensions, and anything else that Railsy tools don't understand.
you can use pg_dump command to dump your database using:
eg:
pg_dump -U <user-name> -h <host> <database> > <file-name>.sql
pg_dump -U postgres -h 127.0.0.1 database1 > database1.sql
Then copy file to other machine and run following command to restore database
psql <database-name> < path/to/sql_dump_file
psql database1 < database1.sql
I am working with a Rails 4 app, I am using PSQL both in my development and production. Due to some reason I have to work with a new computer/laptop so I set up Rails environment in it and cloned my app into it, but what I really require is that, I need my existing databse with data in it, how to do it ?
To do this you need to create a dump on system 1 and restore it on system 2, here are the steps:
sudo -u postgres pg_dump <DB_NAME> > dump - creating a file dump
Copy this file via dropbox or whatever to another system.
sudo -u postgres psql <DB_NAME> < dump - copy the new created dump to new system.
Note:
You should have empty created database on your new system, or you can use dataonly dump passing the --data-only to pg_dump command.
Also you can read documentation for pg_dump to find any other options which you might need.
I am now working on Cloud 9 and need to see my PostgreSQL database. So I hope to know the below two methods.
How to create dump file from PostgreSQL database when you know database name in Cloud 9?
Which tool can I use to see the tables of PostgreSQL database dump?
Use pg_dump. It looks like Cloud9 doesn't permit remote PostgreSQL connections, so you'll need to run it within the workspace, then transfer the dump file to your computer. Use pg_dump -Fc to make a PostgreSQL-custom-format dump, which is the recommended format.
To list tables in the dump use pg_restore -l to list the dump's table of contents. Generally it's easier to work with dumps by restoring them to a local PostgreSQL install (again, using pg_restore) though.
For more detail see the PostgreSQL manual
I have crm rails application deployed in heroku so my database is postgresql. I have an iphone app which is using mysql database. Now what i need to do is to synch my iphone database with my heroku database.
Can anyone please help me on this is it possible to sync two different databases in heroku
Thanks in Advance
The way we did this was to do it manually
Yep, it's a real pain, but at least you know it's done correctly
--
Process
For migrating from Postgres to MYSQL is as follows:
"Backup" the database using pg_backups
Download the backup locally
In MYSQL (we used PHPMyAdmin), create a series of SQL calls to insert the respective data into the correct tables
Migrating from MYSQL to Postgres is, as much as I can determine, a similar process:
Download the SQL dumps of your MYSQL DB (I only know how to do this in PHPMyAdmin)
Configure those files to be "PG" ready
Send the SQL to your Heroku postgres db's
--
Notes
For specifics, I only have the following:
If you dump from Heroku, you'll not get a traditional SQL file; you'll receive a .dump file. To change this, you'll need to use pg_restore in the following way:
$ cd c:/your/postgres/install/pg_restore/location
$ pg_restore path_to_your_dump.dump > path_to_your_sql.sql
This will give you the ability to "read" the file, uploading to your db & importing as you wish. That's when the fun begins :)
Importing MYSQL
The way to import MYSQL into a PG db will have to be done manually
Although we've never done this directly (we only migrated from PG), there are several things you can do. Firstly, you can use the PG2MYSQL converter to give you an idea as to the syntax differences between MYSQL & PG
Then I would perform a "dump" from MYSQL in SQL format. This will give me the data required for PG. This will allow me to insert into my PG database (you might have to do this locally to give you the ability to determine the syntax), from which you can then patch in the MYSQL data
I understand it's not the most elaborate explanation - I hope it gives you some ideas
We have an existing Rails 3 app that has been copied and loaded on a separate server. We've setup the posgres DB for this server; and also configured the database.yaml, pg gems, etc to setup for the port.
However, only the database schema can be migrate...though all the data files has the correct content.
I've tried variations of the db migrate, dump, resets, load, etc. But I'm not success getting the actual data in the database. Again, the server migration is for identical hardware/software config. So, its Rails3.1/Postgres9/Ruby 1.92
I don't get any errors, the data doesn't populate. The ultimate goal is to have an identical app on the 2 servers.
Any ideas? I've already spent 4 days fighting. Many thanks!!
"...the actual data in the database"
If you have an existing database with transactional data - then I think you want to use postgres tools to move the database? maybe I am not understanding the question correctly?
on the source machine
pg_dump DATABASE_NAME > ~/DATABASE_NAME_dump.sql
copy the dump file to the target machine
on the target machine
bundle exec rake db:create
psql DATABASE_NAME < ~/DATABASE_NAME_dump.sql
lots of good information here - http://www.postgresql.org/docs/9.0/static/backup.html
Have you tried the taps gem?
It enables you to transfer schema and data from one instance to another.