I was debugging a Ruby on Rails engine which has problems when running on Sqlite, it has a problem in finding records that the app itself creates. When run on MySQL everything works but the same query on SQLite is failing.
I've tracked down the issue and I found that the problem is in a simple WHERE query which won't find the created record. Essentially the table structure has a column called key which stores some md5 hashes. The failing spec insert a record with a given hash then on the following instruction do a SELECT query for the same hash, but SQLite returns no record for the same key. I've extracted the generated database and the failing query from the app and this is a copy of the app database:
http://dl.dropbox.com/u/2289657/combustion_test.sqlite
Here is a transcript of the queries executed by the software (made with the command line utility):
# Here I'm selecting all the records from the table
# there is a single record in it, the key is the third field
$ sqlite3 combustion_test.sqlite 'SELECT * FROM tr8n_translation_keys'
1||b56c67d10759f8012aff28fc03f26cbf|Hello World|We must start with this sentence!||||en-US|0|2012-03-14 11:49:50.335322|2012-03-14 11:49:50.335322|
# Here I'm selecting the record with that key and it doesn't return anything
$ sqlite3 combustion_test.sqlite "SELECT * FROM tr8n_translation_keys WHERE key = 'b56c67d10759f8012aff28fc03f26cbf'"
# Here I'selecting the record with a LIKE clause and it finds the record
$ sqlite3 combustion_test.sqlite "SELECT * FROM tr8n_translation_keys WHERE key LIKE 'b56c67d10759f8012aff28fc03f26cbf'"
1||b56c67d10759f8012aff28fc03f26cbf|Hello World|We must start with this sentence!||||en-US|0|2012-03-14 11:49:50.335322|2012-03-14 11:49:50.335322|
Should I report this as a bug to SQLite site?
P.S. I've tried also on a different system with a different SQLite version, but the results are the same.
Update
Here is the table schema
sqlite> .schema tr8n_translation_keys
CREATE TABLE "tr8n_translation_keys" (
"id" INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
"type" varchar(255), "key" varchar(255) NOT NULL,
"label" text NOT NULL,
"description" text,
"verified_at" datetime,
"translation_count" integer,
"admin" boolean,
"locale" varchar(255),
"level" integer DEFAULT 0,
"created_at" datetime,
"updated_at" datetime,
"synced_at" datetime
);
CREATE UNIQUE INDEX "index_tr8n_translation_keys_on_key" ON "tr8n_translation_keys" ("key");
CREATE INDEX "index_tr8n_translation_keys_on_synced_at" ON "tr8n_translation_keys" ("synced_at");
Update 2
Here is the rails code which compute the key value inserted into the table (I've removed some code, full method is here)
def self.find_or_create(label, desc = "", options = {})
key = generate_key(label, desc).to_s
# IF I UNCOMMENT THIS LINE EVERYTHING WORKS
#key = 'b56c67d10759f8012aff28fc03f26cbf'
tkey = Tr8n::Cache.fetch("translation_key_#{key}") do
existing_key = where(:key => key).first ### THIS IS THE FAILING WHERE
existing_key ||= begin
new_tkey = create(:key => key.to_s,
:label => label,
:description => desc,
:locale => locale,
:level => level,
:admin => Tr8n::Config.block_options[:admin])
# rest of method...
And here is the generate_key method, the comment about sqlite is from author, not mine)
def self.generate_key(label, desc = "")
# TODO: there is something iffy going on with the strings from the hash
# without the extra ~ = the strings are not seen in the sqlite database - wtf?
"#{Digest::MD5.hexdigest("#{label};;;#{desc}")}"
end
This works:
SELECT * FROM tr8n_translation_keys WHERE LOWER(key)='b56c67d10759f8012aff28fc03f26cbf';
But this doesn't:
SELECT * FROM tr8n_translation_keys WHERE key='b56c67d10759f8012aff28fc03f26cbf' COLLATE NOCASE;
When I examine the database in SQLiteManager, it shows the key as this:
X'6235366336376431303735396638303132616666323866633033663236636266'
which implies it's treating the key as a BLOB (raw binary data) rather than TEXT. This is why the comparison fails. But LOWER(key) causes the field to be cast to text, hence the comparison succeeds.
So, we need to find out why the entry has been stored as a BLOB instead of TEXT. How were these values inserted into the database?
Following your update 2: I'm not a Ruby expert, but the value returned from generate_key is not being converted to a string in the way you expect. Try to_str instead of to_s when calling generate_key.
Based on the following Stack Overflow answer...
https://stackoverflow.com/a/6591427/18064
... you might want to update the generation of your key as follows:
def self.generate_key(label, desc = "")
# TODO: there is something iffy going on with the strings from the hash
# without the extra ~ = the strings are not seen in the sqlite database - wtf?
"#{Digest::MD5.hexdigest("#{label};;;#{desc}").encode('UTF-8')}"
end
Note the addition of .encode('UTF-8').
This worked for me when I had the same problem as yourself.
Related
I need to set a combined index on two fields from a JSONB in my PostgreSQL DB. I can set an index for a single field like so (using ActiveRecord in my Rails 6 application):
add_index :my_table,
"(content->'reference')",
using: :gin,
name: 'index_my_table_on_content_reference'
This one works as expected. However, when I try to set a combined index for two fields, I get the following error:
add_index :my_table,
["(content->'reference')", "(content->'ext_id')"],
using: :gin,
name: 'index_my_table_on_content_ref_and_ext_id'
ActiveRecord::StatementInvalid: PG::UndefinedColumn: ERROR: column "(content->'reference')" does not exist
What am I doing wrong and how can I create a combined index for multiple fields in a JSONB column?
And before you ask: Yes, each JSONB blob has a key named reference.
Using: Ruby 2.6.5, Rails 6.0, PostgreSQL 11
The error PG::UndefinedColumn: ERROR: column "(content->'reference')" does not exist means it's treating "(content->'reference')" as a column name.
Reproducing from SQL:
create index my_table_json_idx on my_table using gin(
(content->'reference'),
"(content->'ext_id')"
);
Note the quotes around the second expression.
It seems there is a problem with your ActiveRecord library and it is escaping your jsonb expression when it isn't desirable.
Either use plain SQL or try to make your ActiveRecord library not escape your expression.
After some Googling I think changing the array to string "(content->'reference'), (content->'ext_id')" might work.
Ya Rails doesn't know how to dump your jsonb index using ruby. So SQL would be the way to go. This was a pretty good article for me when I started messing with jsonb Rails & jsonb
def change
reversible do |dir|
dir.up do
execute <<-SQL
CREATE INDEX IF NOT EXISTS idx_my_table_on_content_references on my_table using gin ((content->>'reference'));
SQL
end
dir.down {execute "DROP INDEX IF EXISTS idx_my_table_on_content_references;"}
end
end
I have a table which contains an integer based column (status) which I'm using for an enum attribute within the Rails model.
At the moment of doing:
Post.select(:id, ..., :status)
Defined as:
enum status: { inactive: 0, active: 1, ... }
It returns everything as expected, but the status column is returned in its string value as inactive, active, etc. But I need it as an integer.
How can I get that?
I'm currently just using ActiveRecord::Base.connection.execute and passing a raw query:
ActiveRecord::Base.connection.execute('select id, ..., status from posts')
It won't be the most elegant way but this is the only way I see possible without looping over your objects.
You can achieve this by creating 2 classes like this:
class Post < ApplicationRecord
end
And the other one:
class PostSave < Post
enum status: { inactive: 0, active: 1, ... }
end
With this when you use Post class you won't get enum value for the "status" column but when you use PostSave class you can use your enum as you were already using.
Now when you do
Post.select(:id, :status)
It will give you integer values for the "status" column as you desire.
Yep. And that's simply how enum field works in Rails. Data is stored as integer, but displayed as string - based on what's defined in enum:
https://api.rubyonrails.org/v5.2.3/classes/ActiveRecord/Enum.html
If you really want to get integer try one of those:
Remove enum declaration. Data is stored as Integer. Without that line query will return Integer
Thats not the most beautiful code but: Post.statuses[post.status] will work
Have you tried this?
# Rails < 5
post = Post.find(123)
post.read_attribute(:status)
# Rails >= 5
post = Post.find(123)
post.read_attribute_before_type_cast(:status)
You can give this a try:
Post.select(:id, ..., 'status as int_status')
I'm not sure this directly applies as I do not know Rails or your exact requirement. But you indicated the desire to "get a collection of records from the DB"; so perhaps it will.
Postgres contains 2 tables in pg_catalog you need to join to get the collection values for a enum: pg_type and pg_enum as follows (where ENUM_NAME is replaced by the appropriate enum type name:
select t.typname,e.enumsortorder, e.enumlabel
from pg_type t
join pg_enum e
on (t.oid = e.enumtypid)
where t.typname = 'ENUM_NAME'
order by e.enumsortorder;
One difference I've notice is that in Postgres the associated numeric value is defined as Real, not as Integer. This is due to how the values are actually maintained. Example:
create type t_ord_stat as enum ('Ordered', 'Delivered', 'Billed', 'Complete', 'Canceled');
select e.enumsortorder, e.enumlabel
from pg_type t
join pg_enum e
on (t.oid = e.enumtypid)
where t.typname = 't_ord_stat'
order by e.enumsortorder;
alter type t_ord_stat add value 'Suppended: Out-of-Stock' after 'Ordered';
alter type t_ord_stat add value 'Suppended: Credit Check' before 'Suppended: Out-of-Stock' ;
alter type t_ord_stat add value 'Suppended: Customer Refused Delivery' before 'Billed';
select e.enumsortorder, e.enumlabel
from pg_type t
join pg_enum e
on (t.oid = e.enumtypid)
where t.typname = 't_ord_stat'
order by e.enumsortorder;
The above gives the actual numeric values underlying the enum string value (as column name indicates it's for sorting purpose). You could get a relative integer value with
select row_number() over() relative_seq, en.enumlabel
from (select e.enumsortorder, e.enumlabel
from pg_type t
join pg_enum e
on (t.oid = e.enumtypid)
where t.typname = 't_ord_stat'
order by e.enumsortorder) en;
I'll leave the actual conversion to Rails to you. Hope it helps.
You could map the column in the SQL statement to another field and read it from there instead, that way Rails passes the value directly as it doesn't know it's an enum (it doesn't know the column at all):
Post.select('id, title, status, status AS status_int').each do |post|
puts post.status # 'inactive'
puts post.status_int # 0
end
If you HAVE to have the same column name, then you're out of luck unless you do a little bit more work:
class Post
STATUSES = %w(inactive active)
end
Post.all.each do |post|
index = Post::STATUSES.index(post.status)
# use 'index' here instead of `post.status`
end
If neither of those are sufficient then this most certainly sounds like the http://xyproblem.info/ ... as these answers should work for 99% of cases. So you should probably explain WHY you cant just use status_int while working with the objects, or why you can't store the int in a variable, or... why you need access to the integer at all, which defeats the purpose of an enum.
The above solutions are working. I have tried with raw sql on mysql version 5.7.28.
the below query is working with mysql only.
I am still looking query for postgres, I will let you know soon.
create table Posts(id integer, title varchar(100), status ENUM ('active', 'inactive') NOT NULL);
insert into Posts(id, title, status) values(1, "Hello", 'inactive')
select title, status+0 from Posts;
I'm writing a migration to change a column type from string to integer.
def change
change_column :emails, :object_id, :integer
end
This migration is failed because this column already contains string values. Before executing this migration I'm trying to remove all letters from this column so that i can get only integer values. Existing values are like
"AB12345"
"A12345X"
"789X26A"
What script i should execute before migration to remove all letters and achieve only integer values like this?
"12345"
"12345"
"78926"
Thanks
If you have more than say 10,000 records do the conversion in the database itself. For postgres, that'd be something like:
select regexp_replace('1231ASDF12', '[^0-9]', '', 'g')
You can use execute to run raw sql in a migration:
update table set col = regexp_replace(col, '[^0-9]', '', 'g')
Keep in mind if you're intending object_id to be a foreign key, you'll need to update whatever table is referenced and also ensure you haven't inadvertently broken anything (e.g., if there was AB123 and BC123 in the dataset).
I think you could use the trim function but the folloing line would do just fine as well.
result = Replace("Some sentence containing Avenue in it.", "Avenue", "Ave")
Example from Access VBA | How to replace parts of a string with another string
You could change "A" into ""
"B" into "" ect.
You would end up whit a code like this
Do While ActiveCell.Value <>""
ActiveCell.Value = Replace(ActiveCell.Value, "A", "")
ActiveCell.Value = Replace(ActiveCell.Value, "B", "")
ActiveCell.Value = Replace(ActiveCell.Value, "C", "")
ect...
ActiveCell.Offset (-1,0).Select
Loop
I'm building a generalized reporting tool for rails, and I'd like to not only get the column names of a raw SQL query, but the converted ruby type as well. I'm going to use the type to make the interface a little better.
The following works, but surely there's a more "rails"-way to approach this? The query is just an example, it could potentially span every table using whatever dynamic SQL the user wants.
sql = "SELECT * FROM SOME_TABLE"
results = ActiveRecord::Base.connection.raw_connection.exec(sql)
results.nfields.times do |i|
puts results.fname(i)
name = results.fname(i)
typename = DataSet.connection.raw_connection.
exec( "SELECT format_type($1,$2)", [results.ftype(i), results.fmod(1)] ).
getvalue( 0, 0 )
column = ActiveRecord::ConnectionAdapters::Column.new(name, nil, typename)
puts column.klass # gives a decent assumption of type
end
ModelX.columns.each do |column|
puts column.sql_type # int(11)
puts column.type # :integer
end
This assumes you have an ActiveRecord model for each table.
In rails, I have a hash that I "sanitized" and then inserted into the database with a direct sql call.
To read the record out, though I need to use the active record. It however is reading the field as a string, though I have the serialize value set for that field.
Here's how I insert the data
def self.update_records(iFileName, iHashArray)
return false if(rows.size == 0)
return false if(killed?)
p "Pushing hash.size = #{rows.size}"
next_id = adjust_sequence( rows.size) #this reserves the record ids. explicit out paces the autosequence
records = iHashArray.each_with_index.map { |hash, index| "\r\n(#{next_id + index}, #{sanitize(hash)})" }
sql = wrapWithSQL(UPLOAD_ID, sanitize(iFileName), records.join(", "));
connection.execute(sql);
adjust_sequence(1);
return true
end
def self.wrapWithSQL(iUploadId, iFileName, iSql)
sql = "WITH new_values (id, hash_value) as ( values #{iSql} )"+
"\r\nINSERT INTO upload_stages (id, hash_value, upload_id, status,file_name, created_at, updated_at)"+
"\r\nSELECT id, hash_value, #{iUploadId}, 1, #{iFileName}, current_timestamp, current_timestamp FROM new_values returning id";
end
My active record class for reading the values
class UploadStage < ActiveRecord::Base
serialize :hash_value, Hash
#...
end
Now on my windows box it works fine. And for the last several months I've seen the hash come out without issue.
On heroku it comes out of the database as a string, and my code can't figure out what to do with a string when it was expecting a hash.
Has anyone any idea how to confront this?
Note... my pushing to the database with raw sql is because I'm pushing 5000 records in one go. I can alter the sql, but I have to be able to push with sql, because active record just doesn't support the speed I need.