I have a task to query in a hive db from ruby code. I am planning to use rbhive gem, but from it documentation, I am not able to get how to pass username, password, db name, etc when connecting to hive server.
Here is my code:
res = RBHive.connect('host_address', 10_000) do |connection|
connection.fetch 'show databases;'
end
It just shows:
Connecting to host_server on port 10000
Executing Hive Query: show databases;
and it hangs there indefinitely.
That may be surprising, but with Hive 1.0.0 I managed to connect using
RBHive.tcli_connect('host', 10000, {transport: :sasl, sasl_params:{}}) do |connection|
Please use
options = { username: 'username', password: 'password' }
RBHive.tcli_connect('host', 'port', { transport: :sasl, sasl_params: options}) do |connection|
results = connection.execute "SHOW DATABASES"
end
Related
In rails 5, I need to configure the dynamodb feature. I have referred some blogs and tried to implement it. First in localhost it was running without any issue, but when I move to other new system or a server then it is showing an error like,
/home/NICHEPRO/shruthir/.rvm/gems/ruby-2.4.0/gems/aws-sdk-core-2.10.19/lib/aws-sdk-core/plugins/regional_endpoint.rb:34:in `after_initialize': missing region; use :region option or export region name to ENV['AWS_REGION'] (Aws::Errors::MissingRegionError)
from /home/NICHEPRO/shruthir/.rvm/gems/ruby-2.4.0/gems/aws-sdk-core-2.10.19/lib/seahorse/client/base.rb:84:in `block in after_initialize'
AWS gem is,
aws-sdk (2.10.19)
aws-sdk-core (2.10.19)
aws-sdk-resources (2.10.19)
Referred From:
https://assist-software.net/snippets/how-save-data-amazon-dynamodb-using-ruby-on-rails
Also I have tried to fix this by referring other blogs but I will get below error too,
Failed to open TCP connection to localhost:8080 (Connection refused - connect(2) for "localhost" port 8080)
How to solve this issue?
Hope you are using Dynamoid gem. In app/config/initializer add a new config file and add below code.
Dynamoid.configure do |config|
config.adapter = 'aws_sdk_v2' # This adapter establishes a connection to the DynamoDB servers using Amazon's own AWS gem.
config.access_key = (ENV['AWS_ACCESS_KEY_ID'] || APP_CONFIG[:aws_access_key_id])
config.secret_key = (ENV['AWS_SECRET_ACCESS_KEY'] || APP_CONFIG[:aws_secret_access_key])
config.region = (ENV['AWS_REGION'] || 'us-east-1')
config.namespace = nil # To namespace tables created by Dynamoid from other tables you might have. Set to nil to avoid namespacing.
config.warn_on_scan = true # Output a warning to the logger when you perform a scan rather than a query on a table.
config.read_capacity = 100 # Read capacity for your tables
config.write_capacity = 200 # Write capacity for your tables
config.endpoint = (ENV['DYNAMO_ENDPOINT'] || APP_CONFIG[:dynamo_endpoint]) # [Optional]. If provided, it communicates with the DB listening at the endpoint. This is useful for testing with [Amazon Local DB] (http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Tools.DynamoDBLocal.html).
end
Make sure you update your ENV variables. Or if you connect directly to AWS instead of Dynamoid gem just follow...
def client
#client ||= Aws::DynamoDB::Client.new(
access_key_id: (ENV['AWS_ACCESS_KEY_ID'] || APP_CONFIG[:aws_access_key_id]),
secret_access_key: (ENV['AWS_SECRET_ACCESS_KEY'] || APP_CONFIG[:aws_secret_access_key]),
region: (ENV['AWS_REGION'] || 'us-east-1'),
endpoint: (ENV['DYNAMO_ENDPOINT'] || APP_CONFIG[:dynamo_endpoint])
)
end
and do a query like this
client.query(
table_name: table_name,
select: 'COUNT',
expression_attribute_values: {
':v1' => index
},
key_condition_expression: 'user_id = :v1'
).count
For more info http://docs.aws.amazon.com/AWSRubySDK/latest/AWS/DynamoDB.html
This is a piece of code in place. When I add this to the cron with timeout the entire array gets saved twice. When I remove timeout nothing gets saved
In this scenario we would want to save the array results (coming in from an api) with over 100k records to be saved to the db. I have used bulk insert and TinyTds gems here
ActiveRecord::Base.establish_connection(adapter: 'sqlserver', host: "xxx", username: "xxx", password: "xxx", database: "xxx", azure: true, port: 1433, timeout: 5000)
class Report < ActiveRecord::Base
self.primary_key = 'id'
end
my_array = [] #count of 100000 records
Report.bulk_insert(:account_owner_id) do |worker|
my_array.drop(2).each do |arr|
worker.add account_owner_id: arr[0]
end
end
You can try removing timeout and adding ignore: true to your bulk insert as shown here. There may be an insert that is failing.
Report.bulk_insert(:account_owner_id, ignore: true) do |worker|
I used rubycas-server to build an sso systems, using devise to do the user's system, but now my question is, rubycas-server only one field like 'email' , I want to use 'email' / 'tel' / 'nickname' for user login. no more info in wiki
authenticator:
class: CASServer::Authenticators::SQLBcrypt
database:
adapter: mysql2
database: xxxx
username: root
password: xxxx
host: localhost
user_table: users
username_column: email # tel/nickname
password_column: encrypted_password
sorry about my poor English！ help me, thanks very much!
I'm affraid it woulnd't be possible. As can be seen in the official repository, this authenticator just matches given column name with user name:
def matching_users
results = user_model.find(:all, :conditions => ["#{username_column} = ?", #username])
results.select { |user| BCrypt::Password.new(user.send(password_column.to_sym)) == #password }
end
For your case the best idea would be to write custom authenticator that matches email/tel/nickname. This is however very tough login name though, consider some more user friendly.
Modify the following code in rubycas-server/lib/casserver/authenticators/sql_bcrypt.rb
def matching_users
results = user_model.find(:all, :conditions => ["#{username_column} = ?", #username])
results.select { |user| BCrypt::Password.new(user.send(password_column.to_sym)) == #password }
end
to
def matching_users
if username_column.include?(',')
columns = username_column.split(',')
sql = ''
conditions = []
columns.each do |field|
if sql.length != 0
sql += ' or '
end
sql += "#{field} = ?"
conditions << #username
end
conditions.unshift(sql)
else
conditions = ["#{username_column} = ?", #username]
end
results = user_model.find(:all, :conditions => conditions)
results.select { |user| BCrypt::Password.new(user.send(password_column.to_sym)) == #password }
end
then u can use mutli-fieds in config.yml like this:
authenticator:
class: CASServer::Authenticators::SQLBcrypt
database:
adapter: mysql2
database: test
username: root
password: map
host: localhost
user_table: users
username_column: email,nickname,tel
password_column: encrypted_password
I use next settings to connect to remote DB:
class MyModel < ActiveRecord::Base
self.table_name = 'users'
establish_connection(
adapter: "mysql2",
host: "host_ip",
encoding: "koi8u",
username: "custom_name",
password: "password",
database: "db_name")
end
It connects well but when I make a query, I receive something like Address: "п⌠п╟п╪п╟п?п╦п? п░п╩я▄я├п╣я│я┌"
I also try another variant of connection like:
connection = Mysql2::Client.new( host: 'host_name',
username: 'user',
password: 'password',
port: 3306,
database: 'db_name',
encoding: 'koi8u',
reconnect: true
)
In this case I receive connection object, but can't make a query... When I use
connection.query("SELECT * FROM users") it returns me the connection object...
Also when I check encoding it returns:
MyModel.first.Address.encoding.name
MyModel Load (6.8ms) SELECT `users`.* FROM `users` ORDER BY `users`.`login` ASC LIMIT
1
=> "UTF-8"
When in settings I use koi8u !
I was checking charset on MySQL server in this DB - it is set to koi8u !
Any ideas? I need to connect to this DB with normal charset
It seems to me that your database locale is different then you expect it to be
Did you changed it or set it in any stage?
It is not enough to declare about it in the connection string
You can find up here how to check what is your database character set
How do I see what character set a MySQL database / table / column is?
I want to give users the option of uploading files in my Ruby on Rails 3.2 application, with the data going into the db. I wanted to use the COPY FROM command since it is faster than inserting ruby objects.
If I do
User.connection.execute("COPY users (name, taxon_id, created_at, updated_at) FROM 'a.txt'")
I get
ActiveRecord::StatementInvalid: PG::Error: ERROR: must be superuser to COPY to or from a file
HINT: Anyone can COPY to stdout or from stdin. psql's \copy command also works for anyone.
However, \copy does not work in a db connection either. How do you easily upload data from flat files in Rails with PostgreSQL on Heroku? Can you get superuser permissions?
Thanks to #PhilipHallstrom's link, I used COPY FROM STDIN like this:
rc = User.connection.raw_connection
rc.exec("COPY users (name, taxon_id, updated_at, created_at) FROM STDIN")
begin
until rc.put_copy_data( data )
$stderr.puts " waiting for connection to be writable..."
sleep 0.1
end
rescue Errno => err
#errmsg = #errmsg + "%s while reading copy data: %s" % [ err.class.name, err.message ]
error = true
else
rc.put_copy_end
while res = rc.get_result
if (res.result_status != 1)
error = true
#errmsg = #errmsg + "Result of COPY is: %s" % [ res.res_status(res.result_status) ]
end
end
end
You can use the gem https://github.com/diogob/postgres-copy
Assuming data is an IO object.
User.copy_from data, columns: [:name, :taxon_id, :updated_at, :created_at]
Or using the path to the uploaded file directly:
User.copy_from "/tmp/uploaded_file.csv", columns: [:name, :taxon_id, :updated_at, :created_at]