I switched to Postgres in development and now following error breaks a test case:
PG::Error: ERROR: column "id" of relation "assets_content_nodes" does not exist
: SELECT pg_get_serial_sequence($1, $2)
Same error happens in development and production environments but it is not a show stopper and does not affect the import.
assets_content_nodes is a join table and I'd rather not add the 'id' column.
Any ideas why this may be happening?
This is fixed in activerecord-import -v0.3.1 (or possibly an earlier version, but it was broken in 0.2.11 and earlier). You no longer need an id column on a table to do an import.
You might also try Upsert.
If you're importing pets:
require 'csv'
require 'upsert'
u = Upsert.new Pet.connection, Pet.table_name
CSV.foreach('pets.csv', headers: :first_row) do |row|
selector = { name: row['name'] }
setter = row
u.row selector, setter
end
My tests show it's 20–30% faster than activerecord-import. Since it's "upserting," it's generally OK if the process gets interrupted and you just start over from the beginning.
Related
I'm trying to run one of my tests, which makes a search, trying to assert the inclusion of records in the search result, but in the meantime, I'm receiving a Elasticsearch::Transport::Transport::Errors::BadRequest error:
SearchTest#test_simple_test_returns_product:
Elasticsearch::Transport::Transport::Errors::BadRequest: [400]
{
"error":{
"root_cause":[
{
"type":"resource_already_exists_exception",
"reason":"index [app_application_test_products/FTt1YC6eQrCw2XwJuqjmDw] already exists",
"index_uuid":"FTt1YC6eQrCw2XwJuqjmDw",
"index":"app_application_test_products"
}
],
"type":"resource_already_exists_exception",
"reason":"index [app_application_test_products/FTt1YC6eQrCw2XwJuqjmDw] already exists",
"index_uuid":"FTt1YC6eQrCw2XwJuqjmDw",
"index":"app_application_test_products"
},
"status":400
}
When I perform a search in development, it works as expected, but in tests is throwing such error, within the test I've added an import and an index refresh, nothing else:
class SearchTest < ActiveSupport::TestCase
setup do
Product.import force: true
Product.__elasticsearch__.refresh_index!
end
test "simple test returns product" do
product = products(:one)
I18n.locale = product.market.lang
search = Search.new(
category: product.category.custom_slug,
page: 1,
market_id: product.market_id,
status: "active",
seed: Date.today.to_time.to_i
)
assert_includes search.results.records, products(:one)
assert_includes search.results.records, products(:two)
assert_not_includes search.results.records, products(:three)
end
end
Any help is appreciated, as any hint to improve the code.
I'm using:
# Gemfile
gem 'minitest', '5.10.1'
# Gemfile.lock
elasticsearch (6.1.0)
elasticsearch-model (6.0.0)
elasticsearch-rails (6.0.0)
minitest (= 5.10.1)
I'm glad you found the root cause for your specific issue.
I ran into a similar issue with ruby-on-rails gem for elasticsearch. While the mappings are all fine, i did get the exact same error message. Leaving my answer here so that anyone else who comes here can get more help.
After a lot of trying and error, eventually figured out that the reason is that it's timing out on the create index.
If you change the client timeout to 60 seconds (it failed with 30 seconds), it was able to create the index successfully without causing this intermittent error.
connection_hash = {
hosts: [ "localhost:9220" ]
reload_connections: true
adapter: :httpclient
retry_on_failure: 2
request_timeout: 60
}
es_connection_client = Elasticsearch::Client.new(connection_hash)
Also, found this issue that is related and was closed after a similar answer.
https://github.com/ankane/searchkick/issues/843#issuecomment-384136164
I was using time freezing in multiple specs, so creating a new object with the same created_at time as the object in the previous spec was causing the resource_already_exists_exception error. Slightly adjusting the timestamp to freeze for each spec fixed the problem.
I had the wrong mappings in my model. Instead using the type option, I was using index, what made ElasticSearch to create a multiple mapping. Which isn't available since the version 6.4 (I guess).
I'm in the process of setting up the backend environment for a mobile app on my local machine and a Postgres function call within a Ruby model errors out the request. Here's the log:
D, [2015-08-05T23:10:06.029187 #1732] DEBUG -- : Game Load (2.0ms) SELECT field1, field2, field3
FROM valuestable
WHERE ARRAY[field1] && (
SELECT custom_function(3, '{}', 'ipad')
);
E, [2015-08-05T23:10:06.029262 #1732] ERROR -- : PG::UndefinedFunction: ERROR: function custom_function(integer, unknown, unknown) does not exist
LINE 4: SELECT custom_function(3, '{}', 'ipad')
^
HINT: No function matches the given name and argument types. You might need to add explicit type casts.
SELECT field1, field2, field3
FROM gamestable
WHERE ARRAY[field1] && (
SELECT custom_function(3, '{}', 'ipad')
);
We use Postgres 9.4.1/Redis/Rails/AWS.
The custom_function in the above sql query is defined in a class that got auto created during a migration. I see it in one of the time-stamped .rb files in the db/migrate folder of the said ruby app. However, I do not know how to "include/import" that into my model so this error can be prevented. I have an sql file called user_functions.sql authored by one of my ex-colleagues in which the custom_function (from the above error) is referenced.
Also, another interesting point is this application runs perfectly on both Staging and Production. It's only on the local machine that it doesn't work. I have no idea if this is some kind of a one-time setup on the backend run directly on AWS or if it's simply a step I'm missing while setting up my local backend.
Meta Information: I'm originally an iOS dev, trying to figure out some backend logic in the absence of a real backend person.
What I tried so far:
Making sure the version number of Postgres matches on server and local backend
Trying to use ActiveRecord::Base.connection.execute(IO.read("/path/to/user_functions.sql")) just before custom_function is first called.
Look what arguments custom function is expecting and cast them.
like this: instead of SELECT custom_function(3, '{}', 'ipad') use SELECT custom_function(3, '{}'::json, 'ipad'::text) or SELECT custom_function(3, '{}'::text, 'ipad'::text) cos many types can go inside the quotes
I have a Rails app running on Heroku which intermittently throws Postgres 'missing FROM-clause' errors from UPDATEs executed by a Rake task.
An example UPDATE statement is:
UPDATE "temp_stats_values"
SET "number" = 66,
"average" = 0,
"percentage" = 0,
"num_out_of" = 0,
"updated_at" = '2013-01-12 11:43:40.067530'
WHERE "temp_stats_values"."id" = 1330
And the error thrown is:
ActiveRecord::StatementInvalid: PG::Error: ERROR: missing FROM-clause entry for table "temp_stats_values"
The thing is, I don't know why it's expecting a FROM clause for this SQL when there clearly isn't another table involved. Also, the fact that it seems to be intermittent is pretty confusing. The errors, when they happen, are being caught by Airbrake.
Anyway, if anyone has any good ideas I'd be very grateful.
UPDATE
Example SQL Fiddle which works as expected
So, without any intervention these errors magically dried up, just like they magically started happening in the first place. Spooky. Anyway, thanks to all who commented.
This morning I switched to postgresql and now my where selects dont work anymore.
What I am trying to do is super simple:
shirt_ids = "1,5,6" # String generated by javascript
Shirt.where("id in (?)", shirt_ids)
This gives me :
PG::Error: ERROR: invalid input syntax for integer: "1,5,6"
LINE 1: SELECT "shirts".* FROM "shirts" WHERE (id in ('1,5,6'))
This works though:
Shirt.where("id in (#{shirt_ids})")
But as everybody knows is unsafe.
Im using:
pg (0.13.2 x86-mingw32)
rails (3.2.2)
Postgresql database is the newest version, I installed it this morning.
Thank you for your help.
^
I believe Postgres is expecting an array, rather than a string for the IN function. If you convert your string to an array, it should work:
shirt_ids = "1,5,6"
Shirt.where('id in (?)', shirt_ids.split(','))
Also, you can do this, which looks a bit cleaner:
Shirt.where(:id => shirt_ids.split(','))
After upgrading to ruby 1.9 we began to notice pages failing to render from the rails template renderer when a user used a non-ASCII character. Specifically "é". I was able to resolve this issue on one of our staging servers, but I have not been able to reproduce the fix on our production server.
The fix that seemed to work the first time:
Converted the database from latin1 to utf8 using the convert_charset tool available here: http://www.mysqlperformanceblog.com/2009/03/17/converting-character-sets/. (including setting default_character_set=utf8 in my.cnf and running SET GLOBAL character_set_server=utf8
Switched to the sam-mysql-ruby adapter (instead of the standard mysql adapter: http://gemcutter.org/gems/sam-mysql-ruby)
Restarted rails
The error is:
"invalid byte sequence in US-ASCII"
Oddly, after following the steps above the error has not changed on our production server. Setting encoding: utf8 in database.yml does not change the error either.
The error raised on the following line of code:
<%= link_to h(question.title), question_path(question) %>
This blog seems to suggest a fix, but it mentions that this should not be a problem in 1.9: http://www.igvita.com/2007/04/11/secure-utf-8-input-in-rails/ (and it's over 2 years old).
I imagine this problem might soon affect a lot of people as more rails developers people switch to 1.9.
I found the solution:
The problem is:
Fetching data from any database (Mysql, Postgresql, Sqlite2 & 3), all configured to have UTF-8 as it's character set, returns the data with ASCII-8BIT in ruby 1.9.1 and rails 2.3.2.1.
(Taken from: https://rails.lighthouseapp.com/projects/8994/tickets/2476)
My attempt to use the patched mysql adapter likely failed because my database was not configured to natively use utf8, so the patched adapter failed to work properly.
The fix ended up being to use the patch file available here: http://gnuu.org/2009/11/06/ruby19-rails-mysql-utf8/
require 'mysql'
class Mysql::Result
def encode(value, encoding = "utf-8")
String === value ? value.force_encoding(encoding) : value
end
def each_utf8(&block)
each_orig do |row|
yield row.map {|col| encode(col) }
end
end
alias each_orig each
alias each each_utf8
def each_hash_utf8(&block)
each_hash_orig do |row|
row.each {|k, v| row[k] = encode(v) }
yield(row)
end
end
alias each_hash_orig each_hash
alias each_hash each_hash_utf8
end
(Placed in lib/mysql_utf8fix.rb and required in enviornment.rb using require 'lib/mysql_utf8fix.rb')
it is only require 'mysql_utf8fix.rb' (rails 2.3.11)
Please user mysql2(gem) adapter instead of mysql adapter in database.yml
and remove the mysql patches(If exists) and add the following lines in environment.rb.
Encoding.default_external = Encoding::UTF_8
Encoding.default_internal = Encoding::UTF_8
Then run in apache and passenger it ll work fine
Thanks,
Ramanavel Selvaraju.