This is a piece of code in place. When I add this to the cron with timeout the entire array gets saved twice. When I remove timeout nothing gets saved
In this scenario we would want to save the array results (coming in from an api) with over 100k records to be saved to the db. I have used bulk insert and TinyTds gems here
ActiveRecord::Base.establish_connection(adapter: 'sqlserver', host: "xxx", username: "xxx", password: "xxx", database: "xxx", azure: true, port: 1433, timeout: 5000)
class Report < ActiveRecord::Base
self.primary_key = 'id'
end
my_array = [] #count of 100000 records
Report.bulk_insert(:account_owner_id) do |worker|
my_array.drop(2).each do |arr|
worker.add account_owner_id: arr[0]
end
end
You can try removing timeout and adding ignore: true to your bulk insert as shown here. There may be an insert that is failing.
Report.bulk_insert(:account_owner_id, ignore: true) do |worker|
Related
I have a task to query in a hive db from ruby code. I am planning to use rbhive gem, but from it documentation, I am not able to get how to pass username, password, db name, etc when connecting to hive server.
Here is my code:
res = RBHive.connect('host_address', 10_000) do |connection|
connection.fetch 'show databases;'
end
It just shows:
Connecting to host_server on port 10000
Executing Hive Query: show databases;
and it hangs there indefinitely.
That may be surprising, but with Hive 1.0.0 I managed to connect using
RBHive.tcli_connect('host', 10000, {transport: :sasl, sasl_params:{}}) do |connection|
Please use
options = { username: 'username', password: 'password' }
RBHive.tcli_connect('host', 'port', { transport: :sasl, sasl_params: options}) do |connection|
results = connection.execute "SHOW DATABASES"
end
database.yml looks like
development:
adapter: mysql2
host: localhost
database: database1
username: root
password:
external:
adapter: mysql2
host: localhost
database: database2
username: root
password:
I have a class ExternalDatabaseConnection
class ExternalDatabaseConnection < ActiveRecord::Base
self.abstract_class = true
establish_connection(:external) #connect to external DB specified in database.yml
attr_protected
end
I'm using AR to query on my external DB.
database2(external DB) is set dynamically from user end and I will not have any information about the tables and attributes.
database1(internal DB) which contains information about tables and some attributes.
So basically I have to handle things dynamically.
Works fine.
ExternalDatabaseConnection.table_name = "set_table1" #dynamic
ExternalDatabaseConnection.create(...) #creates a record.
This doesn't.
ExternalDatabaseConnection.table_name = "set_different_table" #dynamic
ExternalDatabaseConnection.create(...)
throws
unknown attribute: attribute_name
Both the things happen in loop, so every-time it works good for initial table set, for the second table, it throws an error.
Debugging:
Note:
ExternalDatabaseConnection.table_name = "payments"
ExternalDatabaseConnection.column_names
=> ["id", "customer_id", "patent_id", "amount", "currency", "date"]
ExternalDatabaseConnection.table_name = "patents"
=> "patents"
2.0.0-p451 :006 > ExternalDatabaseConnection.column_names
=> ["id", "customer_id", "patent_id", "amount", "currency", "date"]
ExternalDatabaseConnection.all
ExternalDatabaseConnection Load (0.4ms) SELECT `patents`.* FROM `patents`
=> [#<ExternalDatabaseConnection id: 1, number: "1111", description: "https://www.google.co.in", registry_date: "2014-08-20", documnetdir: nil, currency: #<BigDecimal:a8f2adc,'0.1212E2',18(27)>>, #<ExternalDatabaseConnection id: 4, number: "1", description: "jhufgtrdfyuh", registry_date: nil, documnetdir: "", currency: #<BigDecimal:a8f2258,'0.0',9(18)>>, ....]
There is no change in attributes, may be this is causing a problem.
If any information is missing, comment up.
ruby 2.0.0p451
Rails 3.2.17
edit1:
Tries:
Before setting table name
1) ExternalDatabaseConnection.table_name = nil .
2) ExternalDatabaseConnection.clear_active_connections! and clear_all_connections!
3) tried with set_table_name :table1
edit2:
ClassName.column_names is not getting changed even after setting different table_name for the Class
I have tried this following way. It may help
:set_table_name method is used to change the table name dynamically. You can set the table name and get the attributes name instead of columns name.
For payments table
ExternalDatabaseConnection.set_table_name :payments
ExternalDatabaseConnection.first.attribute_names
Then you have to reset the column information
ExternalDatabaseConnection.reset_column_information
For patents:
ExternalDatabaseConnection.set_table_name :patents
ExternalDatabaseConnection.first.attribute_names
I use next settings to connect to remote DB:
class MyModel < ActiveRecord::Base
self.table_name = 'users'
establish_connection(
adapter: "mysql2",
host: "host_ip",
encoding: "koi8u",
username: "custom_name",
password: "password",
database: "db_name")
end
It connects well but when I make a query, I receive something like Address: "п⌠п╟п╪п╟п?п╦п? п░п╩я▄я├п╣я│я┌"
I also try another variant of connection like:
connection = Mysql2::Client.new( host: 'host_name',
username: 'user',
password: 'password',
port: 3306,
database: 'db_name',
encoding: 'koi8u',
reconnect: true
)
In this case I receive connection object, but can't make a query... When I use
connection.query("SELECT * FROM users") it returns me the connection object...
Also when I check encoding it returns:
MyModel.first.Address.encoding.name
MyModel Load (6.8ms) SELECT `users`.* FROM `users` ORDER BY `users`.`login` ASC LIMIT
1
=> "UTF-8"
When in settings I use koi8u !
I was checking charset on MySQL server in this DB - it is set to koi8u !
Any ideas? I need to connect to this DB with normal charset
It seems to me that your database locale is different then you expect it to be
Did you changed it or set it in any stage?
It is not enough to declare about it in the connection string
You can find up here how to check what is your database character set
How do I see what character set a MySQL database / table / column is?
I want to give users the option of uploading files in my Ruby on Rails 3.2 application, with the data going into the db. I wanted to use the COPY FROM command since it is faster than inserting ruby objects.
If I do
User.connection.execute("COPY users (name, taxon_id, created_at, updated_at) FROM 'a.txt'")
I get
ActiveRecord::StatementInvalid: PG::Error: ERROR: must be superuser to COPY to or from a file
HINT: Anyone can COPY to stdout or from stdin. psql's \copy command also works for anyone.
However, \copy does not work in a db connection either. How do you easily upload data from flat files in Rails with PostgreSQL on Heroku? Can you get superuser permissions?
Thanks to #PhilipHallstrom's link, I used COPY FROM STDIN like this:
rc = User.connection.raw_connection
rc.exec("COPY users (name, taxon_id, updated_at, created_at) FROM STDIN")
begin
until rc.put_copy_data( data )
$stderr.puts " waiting for connection to be writable..."
sleep 0.1
end
rescue Errno => err
#errmsg = #errmsg + "%s while reading copy data: %s" % [ err.class.name, err.message ]
error = true
else
rc.put_copy_end
while res = rc.get_result
if (res.result_status != 1)
error = true
#errmsg = #errmsg + "Result of COPY is: %s" % [ res.res_status(res.result_status) ]
end
end
end
You can use the gem https://github.com/diogob/postgres-copy
Assuming data is an IO object.
User.copy_from data, columns: [:name, :taxon_id, :updated_at, :created_at]
Or using the path to the uploaded file directly:
User.copy_from "/tmp/uploaded_file.csv", columns: [:name, :taxon_id, :updated_at, :created_at]
I am having an issue with Rails not inputting values to postgresql. The database itself is connected. When I run db:create:all (snippet from database.yml)
development:
adapter: postgresql
encoding: unicode
database: website_development
username: postgres
password: *******
host: 127.0.0.1
port: 9435
(test: is the same but with database: website_test instead of website_development) all the databases are created for test and development. When I run my db:migration the user table is also created e.g. snippet from migration file "date"_create_user.rb
class CreateUsers < ActiveRecord::Migration
def self.up
create_table :users do |t|
t.string :username
t.string :email
t.timestamps
end
end
def self.down
drop_table :users
end
end
(I have checked in pgAdmin and found the tables that where created)But when I try to insert data from the console e.g.(this was run in sandbox)
irb(main):001:0> User.create!(:username => "John", :email => "john#example.com)
=> #<User id: 1, username: nil, email: nil, created_at: "2011-04-26 22:00:28", u
pdated_at: "2011-04-26 22:00:28">
here is the sql produced on a different create! I had run
[1m[35mSQL (2.0ms)[0m INSERT INTO "users" ("username", "email", "created_at", "updated_at") VALUES (NULL, NULL, '2011-04-26 20:53:43.363908', '2011-04-26 20:53:43.363908') RETURNING "id"
Any help as to why rails is creating the databases and tables fine but can't find the proper username and email to enter into sql.
P.S. I am running Rspec for my tests and have made several tests regarding the values of username and email not being nil to which all succeed.
......................
Finished in 1.62 seconds
22 examples, 0 failures
Notification failed: 201 - The destination server was not reachable
Notification failed: 201 - The destination server was not reachable
As you can see all Rspec tests are green but it to is having trouble connecting to the postgres server
Thank you in advance for any advice.
Update: added user model snippet
class User < ActiveRecord::Base
attr_accessor :username, :email
email_regex = /\A[\w+\-.]+#[a-z\d\-.]+\.[a-z]+\z/i
username_regex = /\A[\w\d]+\z/i
validates :username, :presence => true,
:format => { :with => username_regex },
:length => { :maximum => 30},
:uniqueness => { :case_sensitive => false }
validates :email, :presence => true,
:format => { :with => email_regex },
:uniqueness => { :case_sensitive => false }
end
==Answer==
These were my mistakes:
Part 1: By changing attr_accessor to attr_accessible all my tests worked properly, and everything that needed to went to red, this also allowed me to add :email details but not :username details which leads to part 2.
Part 2: For some reason rails didn't like the fact that my table was named :user and my column was named :username. So I tried changing :username to :loginname which fixed the problem entirely.
Thank you everyone for all your help.
To isolate this you may want to construct a unit test to replicate the problem, then repair it as required. At first I suspected it would be a case of protected attributes, but it appears you have made them accessible, which is the correct thing to do.
Calling create! directly is somewhat hazardous as you are not easily able to capture the object that is half-created in the event of an exception. This is because although the exception contains a reference to a model, it is not clear if the User model or some other model caused the exception in the first place without additional digging.
A more reliable approach is this:
def test_create_example
user = User.new(:username => "John", :email => "john#example.com")
assert_equal 'John', user.username
assert_equal 'john#example.com', email
user.save
assert_equal [ ], user.errors.full_messages
assert_equal false, user.new_record?
end
If an error occurs in the validation stream you will see the error listed alongside what should be an empty array. It also checks that the record has been saved by testing that it is no longer a new record as records can be valid but fail to save if a before_save or before_create filter returns false, something that happens by accident quite often.
If you call new and then save you have an opportunity to inspect the newly prepared object before it is saved, as well as after.