mysqldump character set question - character-encoding

I have a database in which the stored data are encoded in GBK. Now I want to make a full dump of it and make the dump file in utf8. This way I can convert the whole database from GBK to utf8. The mysqldump utility seems the right way to go and I just looked into the mysqldump documentation http://dev.mysql.com/doc/refman/5.5/en/mysqldump.html#option_mysqldump_set-charset and it seems it uses the utf8 character set. So my question is can I just go with
mysqldump mydatabase > dump.sql
will that do the work for me, will mysqldump automatically convert my GBK database to utf8 for me? And since all the tables used to have a bgk character set set originally, if I want to import the dump file back into another utf8 database, will that work as well?

I want to import the dump file back into another utf8 database, will
that work as well?
In mysql charset are assigned on many levels (connection, table, field, ecc).
Assuming that all this levels in the old database are with charset GBK, and all this levels in the new database are with charset utf8, you just have to substitute the charset declaration for each level.
Imagine that this is a portion of the dump file dump.sql
CREATE TABLE IF NOT EXISTS `mytable` (
`id` int(11) NOT NULL auto_increment,
`myname` varchar(255) NOT NULL default '',
PRIMARY KEY (`id`)
) ENGINE=MyISAM DEFAULT CHARSET=gbk AUTO_INCREMENT=1065 ;
what you need is to transform it into
CREATE TABLE IF NOT EXISTS `mytable` (
`id` int(11) NOT NULL auto_increment,
`myname` varchar(255) NOT NULL default '',
PRIMARY KEY (`id`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 AUTO_INCREMENT=1065 ;
you could do this with sed
sed -i 's/gbk/utf8/g' dump.sql
now the file is ready to be correctly imported into an utf8 database

Related

utc_current as default value in informix

Use utc_current as default value in a field of an informix database table.
My idea is to do something like this, so that when that record is inserted or updated, the value is automatically increased since datetime or timestamp doesn't work for me.
CREATE TABLE tab1
(
id VARCHAR(128) NOT NULL,
update_ts integer DEFAULT dbinfo('utc_current') ,
modcount BIGINT,
);

Rails Migration Changes Datatype from :bigint to :integer

I recently created a migration to update some data, and after I ran it, the migration changed a bunch of my schema.rb column definitions from :bigint to :integer
I've noticed this a few times in the past. Sometimes a migration will change them from :bigint to :integer and vice versa other times.
Why is this happening and how can I prevent it? I want to keep them as :bigint
I'm running Ruby on Rails 7.0.2.3 and PostgreSQL 14.2
Thanks
I was able to figure it out ....
Issue: the database type was different than what the schema had, so when the migration ran, rails realized the types in the actual db were different so it updated the schema for me.
Why we're the types different though?
I had pulled a database dump down from Heroku in order to replicate a problem locally that I could not replicate (it only occurred on Heroku with that data set). Upon doing so, this problem started.
The issue was that Heroku was running Postgres 13.6 and the column types were integer in that db for whatever reason.
Locally I'm running Postgres 14.2 and the types locally were bigint.
Why was the schema.rb file being updated if the migration didn't change anything?
This happens because the db:schema:dump is run after the migrations automatically. This happens in the database.rake file in Rails.
Source: https://github.com/rails/rails/blob/main/activerecord/lib/active_record/railties/databases.rake#L105
In other words, the migrations get run, then Active Record automatically dumps the schema.
The new schema dump was different than the original schema, due to different data types. Therefore the schema.rb file was updated (and also any model files if you're using the annotate gem).
How was I able to tell that?
Log into the DB with psql and check.
Local: psql -h localhost
Heroku: heroku pg:psql -a your-app-name
Then inspect the table:
\d table_name
And you'll get something like this:
Table "public.account_invitations"
Column | Type | Collation | Nullable | Default
---------------+--------------------------------+-----------+----------+-------------------------------------------------
id | bigint | | not null | nextval('account_invitations_id_seq'::regclass)
account_id | integer | | not null |
invited_by_id | integer | | |
token | character varying | | not null |
In there you can see the column types. Locally they were bigint with Postgres 14.2 but on Heroku they were integer with Postgres 13.6.
When I pulled a dump down, I restored the local db from that dump, which changed the database vs what was in the schema.
Fix:
Restored my original version of my db, ran the migrations, no changes to the schema were made.
How to test this theory ...
Go into your database console and change a column from integer to bigint or vice versa. Then run a migration. You'll notice your schema file gets updated, even though your migration probably had nothing to do with those columns whatsoever.
Not sure about the reason, but as far as I remember, this problem arose back in RoR version 3 or 4.
You need to use a format like this
tbl.column :really_big_int, :bigint

MariaDB, DEFAULT CURRENT_TIMESTAMP not saving

I have a MariaDB version 10.2.13-MariaDB-10.2.13+maria~jessie, with a table ids. The table's create code is:
CREATE TABLE `ids` (
`id` BIGINT(20) UNSIGNED NOT NULL AUTO_INCREMENT,
`lastupdate` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, # for some reason this is being ignored
PRIMARY KEY (`id`)
)
COLLATE='utf8_general_ci'
ENGINE=InnoDB
;
When the table is created, the lastupdate column's default value is not set. In fact, looking at the CREATE code in HeidiSQL, I see ... DEFAULT '' ....
Furthermore, the following query runs without error, but does not affect the table
ALTER TABLE ids
MODIFY lastupdate TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP;
whereas this one works completely fine
ALTER TABLE ids
MODIFY lastupdate TIMESTAMP NOT NULL DEFAULT '0000-00-00 00:00:00';
. What could be going wrong here?
Implementation detail: The database is being run inside a docker container trivially extended from the default mariadb image.
This is probably an issue with HeidiSQL (or this particular version of HeidiSQL), not an issue with MariaDB itself.
You can verify this by using the MariaDB client (mysql) and run your CREATE TABLE query and then:
SHOW CREATE TABLE ids;

Rails 3.1: Is there any way I can view which indexes have been created for my model?

I'm having a problem with my ActiveRecord exception handling and I suspect some of the indexes haven't been created as I thought they did. How can I view the indexes that have been created for my model?
Thanks so much in advance for your wisdom!
You can just look in your db/schema.rb where all the structure of the database is explicitly listed.
Assuming you are using MySQL, from your command line:
$ rails dbconsole
mysql> show create table users;
users | CREATE TABLE `users` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`name` varchar(100) DEFAULT '',
`email` varchar(100) DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `index_users_on_email` (`email`)
) ENGINE=InnoDB AUTO_INCREMENT=61 DEFAULT CHARSET=latin1 |
Any "KEY" lines are your indexes.

string/text storage limits in a sqlite3 database

I have a note field of string type in one of my models in my sqlite3 database, but I realized that I needed to store more text than string would allow.
I just ran a migration changing the type of the field from string to text. Looking at my database, it says that the type is now text(255), whereas before it was varchar(255).
What does the 255 mean? Is that a character limit? If so, would I have the same storage problems as before? How would I fix this?
Here is the migration I used to change the field type
change_column(:posts, :note, :text)
SQLite does not enforce text storage limits. If you declare a column VARCHAR(1) or TEXT(1) you still can store some very long blob of 500 Megabytes inside it. Even though that is not adviseable.
~$ sqlite
SQLite version 3.7.2
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> create table foo ( bar text(3));
sqlite> insert into foo values ('aeiou');
sqlite> select * from foo;
aeiou
You should just make your storage type text and not put any limit on it. The 255 denotes the maximum number of characters allowed in the field.

Resources