We have a shoryuken worker process setup in rails application (ruby 2.6.6, rails > 5).
this is the shoryuken.yml.erb file
:aws:
:access_key_id: ?
:secret_access_key: ?
:region: <%= Settings.aws.region %>
:concurrency: 10
:daemon: false
:pidfile: tmp/shoryuken.pid
:logfile: log/shoryuken.log.<%= Time.now.strftime('%Y%m%d') %>
:queues:
- [<%= Settings.aws.sqs.use_case_1_queue_name %>, 2]
- [<%= Settings.aws.sqs.use_case_2_queue_name %>, 1]
development:
:verbose: true
And we start the worker using shoryuken -R -C ./shoryuken.yml Please note the -R option since we would want it to run rails initializers in order to access Settings constant in the above shoryuken.yml.erb
It all looked good but then we figured out that the AWS credentials are not being stored in application setting , but in IAM role. So how will I pass the aws-sdk credential (stored using IAM) to shoryuken process.
If there are no solution, then I will need to finally store aws credentials in application setting which might be a security issue according to infra team.
Thank you for this wonderful library . Awaiting a response on this issue
Related
I am trying to connect from a Ruby on Rails application to AWS Keyspaces (AWS Cassandra), but I cannot manage to do it. I use the cequel gem and generated the config/cequel.yml which contains a similar thing to the following:
development:
host: "CONTACT_POINT"
username: "USER"
password: "PASS"
port: 9142
keyspace: key_development
max_retries: 3
retry_delay: 0.5
newrelic: true
ssl: true
server_cert: 'config/certs/AmazonRootCA1.pem'
replication:
class: NetworkTopologyStrategy
datacenter1: 3
datacenter2: 2
durable_writes: false
(Credentials where used in another app and they work which is working as expected.)
when I try to run:
rake cequel:keyspace:create
I get the following errors:
Cassandra::Errors::NoHostsAvailable: All attempted hosts failed: x.xxx.xxx.xxx (Cassandra::Errors::ServerError: Internal Server Error)
Set the dc to us-east-1 . drop the replication definition.
Rails 6
I created the credentials file, as follows:
EDITOR=vi rails credentials:edit
production:
mysql:
db: acme-production
user: deploy
password: xxxxxxxxxxxx
smartagent:
token: lnroftb7sgr8c7f1ogqvij24xl
test:
mysql:
db: acme-test
user: deploy
password: xxxxxxxxxxxx
smartagent:
token: lnroftb7sgr8c7f1ogqvij24xl
secret_key_base: xxxxxxxxxxxxxx
master.key is in the correct place (in the config folder, locally, and as an environment setting on the server).
I am having trouble with the smartagent token.
When I do:
rails c
and
Rails.application.credentials.dig(Rails.env.to_sym, :mysql, :db)
I get:
=> "acme-test"
However, when I do:
Rails.application.credentials.dig(Rails.env.to_sym, :smartagent, :token)
I get:
=> nil
Any idea why this might be happening?
Solution:
I deleted the credentials file, re-populated it, and restarted the Rails app. That fixed the problem, so this must have been either an indentation issue (spaces vs tabs), or that the server had not restarted properly, after earlier changes were made
I have 2 elasticsearch servers. On my rails app, can I change connection to the Elasticsearch servers at run time?
For example,
- If user 1 log in the app, it should connect to elasticsearch server 1
- If user 2 log in the app, it should connect to elasticsearch server 2
Thanks
You can use randomize_hosts when creating connection
args = {
hosts: "https://host1.local:9091,https://host2.local:9091",
adapter: :httpclient,
logger: (Rails.env.development? || Rails.env.test?) ? Rails.logger : nil,
reload_on_failure: false,
randomize_hosts: true,
request_timeout: 5
}
client = Elasticsearch::Client.new(args)
Randomize hosts doc
Here you can read about a different host selection strategy than round robin. You could implement your own ideas.
Working on an Elixir app. There's a Scraper function that copies data from a Google Spreadsheet into a postgres database via the Postgrex driver. The connection through the Google API works fine, but the function always times out after 15 seconds.
01:48:36.654 [info] Running MyApp.Endpoint with Cowboy using http://localhost:80
Interactive Elixir (1.6.4) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> Scraper.update
542
iex(2)> 01:48:55.889 [error] Postgrex.Protocol (#PID<0.324.0>) disconnected: ** (DBConnection.ConnectionError) owner #PID<0.445.0> timed out because it owned the connection for longer than 15000ms
I have tried changing the 15_000 ms timeout setting everywhere in the source, but it seems the setting has been compiled into binary. I am not an erlang/elixir developer, just helping a client install the app for the purposes of demoing. My question is:
How can I recompile the Postgrex driver with the modified timeout setting?
Is there another way to override this setting, or disable the timeout altogether? I have tried find-replace of basically every instance of "15" in the source.
When issuing a query with postgrex, the last argument can be a keyword list of options.
Postgrex.query!(pid, "AN SQL STATEMENT;", [], timeout: 50_000, pool_timeout: 40_000)
https://hexdocs.pm/postgrex/Postgrex.html#query/4
config :my_app, MyApp.Repo,
adapter: Ecto.Adapters.Postgres,
username: "postgres",
password: "postgres",
database: "my_app_dev",
hostname: "localhost",
timeout: 600_000,
ownership_timeout: 600_000,
pool_timeout: 600_000
Look at timeout and ownership_timeout. These values are set to 600 seconds. And probably not of them are necessary.
Also I want to say that once I had to remove everything from _build and recompile an application to have this values actually applied.
I am using rails "Shoryuken" gem but I am getting errors for validation on queues on my development environment when I started rails server below is the error:-
gems/shoryuken-2.0.11/lib/shoryuken/environment_loader.rb:172:in `validate_queues': The specified queue(s) ["development_worker"] do not exist (ArgumentError)
I have used below settings:-
config/shoryuken.yml
aws:
access_key_id: <%= ENV["SQS_IAM"] %>
secret_access_key: <%= ENV["SQS_IAM_SECRET"] %>
region: <%= ENV["SQS_IAM_REGION"] %>
concurrency: 25 # The number of allocated threads to process messages. Default 25
delay: 0 # The delay in seconds to pause a queue when it's empty. Default 0
queues:
- ["<%= Rails.env %>_worker", 1]
initializers/shoryuken.rb
def parse_config(config_file)
if File.exist?(config_file)
YAML.load(ERB.new(IO.read(config_file)).result)
else
raise ArgumentError, "Config file #{config_file} does not exist"
end
end
config = parse_config([Dir.pwd, 'config/shoryuken.yml'].join('/')).deep_symbolize_keys
Shoryuken::EnvironmentLoader.load(config)
I want the queue should be environment specific.
Ravindra, looking at the code of shoryuken, you are getting the error because you do not have created an SQS queue named development_worker, is that rigth?
You will need to create one queue for each developer, because shoryuken will be running in the computers of each developer. If you don't do that, all shoryuken processes of each computer will be polling messages of the same queue.
Imagine that there are two shoryuken processes, sh1 and sh2, that correspond to the dev1 and dev2 machine respectively.
If the dev1 sends a SQS msg to the dev-queue, the sh2 proccess can be able to poll the message that the dev1 sent.
If you wish to avoid to create the queues in AWS, you can take a look at this.