Using Sidekiq with multi APIs, but Sidekiq server executed wrong API code - ruby-on-rails

I've built a rails app with docker-compose like below.
For example,
API A created job A1, that pushed to redis by Sidekiq Client SA.
And API B created job B1, that pushed to redis by Sidekiq Client SB.
But when these jobs executed, It pointed to application code in only API A.
So job B1 was failed because it was executed by API A.
I know that because the uninitialized constant error was raised.
I also used redis-namespace, but it still pointed to wrong API.
Can you help me explain how Sidekiq Server executed jobs.
And how it point to the right API that the job belongs to.
Many thanks.
config_redis = {
url: ENV.fetch('REDIS_URL_SIDEKIQ', 'redis://localhost:6379/0'),
namespace: ENV.fetch('REDIS_NAMESPACE_SIDEKIQ', 'super_admin')
}
Sidekiq.configure_server do |config|
config.redis = config_redis
end
Sidekiq.configure_client do |config|
config.redis = config_redis
end
initializer/sidekiq.rb
config_redis = {
url: ENV.fetch('REDIS_URL_SIDEKIQ', 'redis://localhost:6379/0'),
namespace: ENV.fetch('REDIS_NAMESPACE_SIDEKIQ', 'ignite')
}
Sidekiq.configure_server do |config|
config.redis = config_redis
end
Sidekiq.configure_client do |config|
config.redis = config_redis
end
docker-compose.yml
version: "3.9"
services:
ccp-ignite-api-gmv: # ----------- IGNITE SERVER
build: ../ccp-ignite-api-gmv/.
entrypoint: ./entrypoint.sh
command: WEB 3001
# command: MIGRATE # Uncomment this if you want to run db:migrate only
ports:
- "3001:3001"
volumes:
- ../ccp-ignite-api-gmv/.:/src
depends_on:
- db
- redis
links:
- db
- redis
tty: true
stdin_open: true
environment:
RAILS_ENV: ${RAILS_ENV}
REDIS_URL_SIDEKIQ: redis://redis:6379/ignite
REDIS_NAMESPACE_SIDEKIQ: ignite
ccp-super-admin-api-gmv: # ----------- SUPER ADMIN API SERVER
build: ../ccp-super-admin-api-gmv/.
entrypoint: ./entrypoint.sh
command: WEB 3005
# command: MIGRATE # Uncomment this if you want to run db:migrate only
ports:
- "3005:3005"
volumes:
- ../ccp-super-admin-api-gmv/.:/src
depends_on:
- db
- redis
links:
- db
- redis
tty: true
stdin_open: true
environment:
RAILS_ENV: ${RAILS_ENV}
REDIS_URL_SIDEKIQ: redis://redis:6379/super_admin
REDIS_NAMESPACE_SIDEKIQ: super_admin
db:
image: mysql:8.0.22
volumes:
- ~/docker/mysql:/var/lib/mysql
command: --default-authentication-plugin=mysql_native_password
environment:
MYSQL_ROOT_PASSWORD: password
ports:
- "3307:3306"
redis:
image: redis:5-alpine
command: redis-server
ports:
- 6379:6379
volumes:
- ~/docker/redis:/data
sidekiq_ignite:
depends_on:
- db
- redis
build: .
command: bundle exec sidekiq
volumes:
- .:/src
environment:
- REDIS_URL_SIDEKIQ=redis://redis:6379/0
- REDIS_NAMESPACE_SIDEKIQ=ignite
sidekiq_super_admin:
depends_on:
- db
- redis
build: .
command: bundle exec sidekiq
volumes:
- .:/src
environment:
- REDIS_URL_SIDEKIQ=redis://redis:6379/0
- REDIS_NAMESPACE_SIDEKIQ=super_admin

Thanks st.huber for reminding me,
That confused came from my wrong docker-compose config in 2 sidekiq service.
I've pointed to wrong folder in "build" and "command"
Before:
sidekiq_ignite:
depends_on:
- db
- redis
build: .
command: bundle exec sidekiq
volumes:
- .:/src
environment:
- REDIS_URL_SIDEKIQ=redis://redis:6379/0
- REDIS_NAMESPACE_SIDEKIQ=ignite
sidekiq_super_admin:
depends_on:
- db
- redis
build: .
command: bundle exec sidekiq
volumes:
- .:/src
environment:
- REDIS_URL_SIDEKIQ=redis://redis:6379/0
- REDIS_NAMESPACE_SIDEKIQ=super_admin
Fixed:
sidekiq_ignite:
depends_on:
- db
- redis
build: ../ccp-ignite-api-gmv/.
command: bundle exec sidekiq
volumes:
- ../ccp-ignite-api-gmv/.:/src
environment:
- REDIS_URL_SIDEKIQ=redis://redis:6379/0
- REDIS_NAMESPACE_SIDEKIQ=ignite
sidekiq_super_admin:
depends_on:
- db
- redis
build: ../ccp-super-admin-api-gmv/.
command: bundle exec sidekiq
volumes:
- ../ccp-super-admin-api-gmv/.:/src
environment:
- REDIS_URL_SIDEKIQ=redis://redis:6379/0
- REDIS_NAMESPACE_SIDEKIQ=super_admin

Related

Sidekiq not running enqueued jobs (Rails 7 + Redis + Sidekiq + Docker)

As the title says, i have 3 containers running in docker, 1 for rails, 1 for a postgres db and 1 for redis. I'm able to enqueue jobs doing Job.perform_async but for some reason my jobs stay on the enqueued indefinitely. I checked and my Redis container is up and running.
My Job:
class HardJob
include Sidekiq::Job
def perform(*args)
puts 'HardJob'
end
end
The initializer for sidekiq:
Sidekiq.configure_server do |config|
config.redis = { url: (ENV["REDIS_URL"] || 'redis://localhost:6379') }
end
Sidekiq.configure_client do |config|
config.redis = { url: (ENV["REDIS_URL"] || 'redis://localhost:6379') }
end
My docker-compose:
version: '3.0'
services:
web:
build: .
entrypoint: >
bash -c "
rm -f tmp/pids/server.pid
&& bundle exec rails s -b 0.0.0.0 -p 3000"
ports:
- 3000:3000
volumes:
- .:/src/myapp
depends_on:
- db
- redis
links:
- "db:db"
environment:
REDIS_URL: 'redis://redis:6379'
db:
image: postgres:11
environment:
POSTGRES_PASSWORD: 'postgres'
volumes:
- db_data:/var/lib/postgresql/data
ports:
- 5432:5432
redis:
image: "redis"
volumes:
db_data:
redis:
driver: local
And i also set config.active_job.queue_adapter = :sidekiq in my 3 environments.
Any hint of what could be happening here? Thanks in advance
Update
Seems that running sidekiq -q default in my rails terminal worked. How can i configure Docker to always run sidekiq?
Sidekiq is process on it own and needs to be started on it own, just like the web server process. Add something like the following to docker-compose:
sidekiq:
depends_on:
- 'db'
- 'redis'
build: .
command: bundle exec sidekiq
volumes:
- .:/src/myapp
environment:
- REDIS_URL_SIDEKIQ=redis://redis:6379/1
Or – when you are able to use the latest version of Sidekiq (>= 7.0) – you might want try out the new Sidekiq embedded mode that runs Sidekiq in together with your puma webserver.
Sidekiq is looking for the wrong queue name for some reason. Try adding this
to your config/sidekiq.yml file.
:queues:
- default

Connect to PostgreSQL Database in Docker Container from DBeaver

I can't get the connection between my PostgreSQL database from my Rails app which is running in a Docker container working.
The application just works fine, I just can't connect to the database.
docker-compose.yml:
services:
app:
build:
context: .
dockerfile: app.Dockerfile
container_name: application_instance
command: bash -c "bundle exec puma -C config/puma.rb"
volumes:
- .:/app
- node-modules:/app/node_modules
- public:/app/public
depends_on:
- database
- redis
env_file:
- .env
database:
image: postgres
container_name: database_instance
restart: always
volumes:
- db_data:/var/lib/postgresql/data
ports:
- "5432:5432"
env_file:
- .env
environment:
POSTGRES_USER: ${POSTGRES_USER}
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
POSTGRES_DB: ${POSTGRES_PRODUCTION_DB}
nginx:
build:
context: .
dockerfile: nginx.Dockerfile
depends_on:
- app
volumes:
- public:/app/public
ports:
- "80:80"
redis:
image: redis
container_name: redis_instance
ports:
- "6379:6379"
sidekiq:
container_name: sidekiq_instance
build:
context: .
dockerfile: app.Dockerfile
depends_on:
- redis
- database
command: bundle exec sidekiq
volumes:
- .:/app
env_file:
- .env
volumes:
db_data:
node-modules:
public:
If I try to connect via DBeaver I get the following message:
Any idea what's going wrong here? The port should be exposed on my local machine. I also tried with the IP of the container, but then I get a timeout exception.
This is because you most likely have postgres running locally on your machine (port 5432) and also on a docker (port 5432). Dbeaver wants to connect to database on your local machine, than on docker.
Any solution I figure out is to temporary stop/turn of your local postgres service (on Windows: Task manager -> Services -> (postgres service) -> stop).
I was also struggling with issue.

Connect Rails app and browserless/chrome container

I am trying to use the browserless/chrome docker image and connect to it from a Ruby on Rails (6.0) app running in another docker container. I'm having difficulty getting the two to connect.
The browserless container is running properly (I know because I can access the web debug interface it comes with).
In the Rails app, I am trying to use capybara with selenium-webdriver to drive a headless chrome instance.
I am getting errors saying that either it can't connect or that it cannot post to '/session', which I think means it's properly connecting but can't initialize a session within chrome for some reason.
Below is my current configuration + code:
#docker-compose.yml
version: "3"
services:
web:
build: .
image: sync-plus.web:latest
volumes:
- .:/app
ports:
- "3000:3000"
environment:
RAILS_ENV: development
RACK_ENV: development
REDIS_URL: redis://redis:6379/0
DATABASE_URL_TEST: postgres://postgres:super#postgres:5432/salesync_test
command: "bin/puma -C config/puma.rb"
depends_on:
- postgres
- mailcatcher
- redis
sidekiq:
image: sync-plus.web:latest
volumes:
- .:/app
environment:
RAILS_ENV: development
RACK_ENV: development
REDIS_URL: redis://redis:6379/0
DATABASE_URL_TEST: postgres://postgres:super#postgres:5432/salesync_test
command: "bundle exec sidekiq"
depends_on:
- postgres
- mailcatcher
- web
- redis
guard:
image: sync-plus.web:latest
volumes:
- .:/app
environment:
RAILS_ENV: test
RACK_ENV: test
REDIS_URL: redis://redis:6379/1
DATABASE_URL_TEST: postgres://postgres:super#postgres:5432/salesync_test
command: "bin/rspec"
depends_on:
- postgres
redis:
image: redis:alpine
command: redis-server
volumes:
- 'redis:/data'
postgres:
image: postgres:latest
volumes:
- postgres_volume:/var/lib/postgresql/data
environment:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: super
mailcatcher:
image: schickling/mailcatcher:latest
ports:
- "1080:1080"
browserless:
image: browserless/chrome:latest
ports:
- "4444:3000"
volumes:
postgres_volume:
redis:
#rails_helper.rb
...
require "rspec/rails"
# This require needs to be below the rspec/rails require
require 'capybara/rails'
...
I am using an Rspec test to check if the browserless connection/driving is functional:
#browserless_spec.rb
require "rails_helper"
RSpec.describe 'Browserless', type: :feature do
describe "Connection" do
it "can connect to browserless container" do
#selenium webdriver looks at localhost:4444 by default
driver = Selenium::WebDriver.for :remote
driver.visit("/users/sign_in")
expect(page).to have_selector("btn btn-outline-light my-2")
end
end
end
I can't seem to find any solid solutions or leads elsewhere, so any help would be much appreciated!
Thanks!

Docker/Wallaby - localhost refused to connect

I am trying to run my feature test using Wallaby but I keep getting the localhost refused to connect error.
Here is my compose.yml:
version: '2'
services:
app:
image: hin101/phoenix:1.5.1
build: .
restart: always
ports:
- "4000:4000"
- "4002:4002"
volumes:
- ./src:/app
depends_on:
- db
- selenium
hostname: app
db:
image: postgres:10
restart: always
environment:
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres
- POSTGRES_DB=hinesh_blogs
ports:
- 5432:5432
volumes:
- database:/var/lib/postgresql/data
selenium:
build:
context: .
dockerfile: Dockerfile-selenium
container_name: selenium
image: selenium/standalone-chrome-debug:2.52.0
restart: always
ports:
- "4444:4444"
- "5900:5900"
hostname: selenium
volumes:
database:
test_helper.ex:
ExUnit.start()
Ecto.Adapters.SQL.Sandbox.mode(HineshBlogs.Repo, :auto)
{:ok, _} = Application.ensure_all_started(:ex_machina)
Application.put_env(:wallaby, :base_url, HineshBlogsWeb.Endpoint.url)
Application.put_env(:wallaby, :screenshot_on_failure, true)
{:ok, _} = Application.ensure_all_started(:wallaby)
config/test.exs:
use Mix.Config
# Configure your database
#
# The MIX_TEST_PARTITION environment variable can be used
# to provide built-in test partitioning in CI environment.
# Run `mix help test` for more information.
config :hinesh_blogs, HineshBlogs.Repo,
username: "postgres",
password: "postgres",
database: "hinesh_blogs_test#{System.get_env("MIX_TEST_PARTITION")}",
hostname: "db",
pool: Ecto.Adapters.SQL.Sandbox,
pool_size: 10
# We don't run a server during test. If one is required,
# you can enable the server option below.
config :hinesh_blogs, HineshBlogsWeb.Endpoint,
http: [port: 4002],
server: true
config :hinesh_blogs, :sql_sandbox, true
# Print only warnings and errors during test
config :logger, level: :warn
# Selenium
config :wallaby, otp_app: :hinesh_blogs_web, base_url: "http://localhost:4002/", driver: Wallaby.Selenium, hackney_options: [timeout: :infinity, recv_timeout: :infinity]
I run the tests using the command: docker-compose run app mix test
Do I need to have any additional configurations to run these tests and if not, what is the best way to configure wallaby to use docker containers?

Rails Active Storage in Docker

I'm running a docker compose which consists of a web worker, a postgres database and a redis sidekiq worker. I created a background job to process images after uploading user images. ActiveStorage is used to store images. Normally without docker, in local development, the images are stored in a temporary storage folder to simulate a cloud storage. I'm fairly new to Docker, so I'm not sure how storage works. I believe storage in Docker works a bit differently. The sidekiq worker seems fine, it just seems like it's complaining about not able to find a place to store images. Below is the error that I get from the sidekiq worker.
WARN: Errno::ENOENT: No such file or directory # rb_sysopen - /myapp/storage
And here is my docker-compose.yml
version: '3'
services:
setup:
build: .
depends_on:
- postgres
environment:
- RAILS_ENV=development
command: "bin/rails db:migrate"
postgres:
image: postgres:10-alpine
environment:
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=mysecurepass
- POSTGRES_DB=myapp_development
- PGDATA=/var/lib/postgresql/data
postgres_data:
image: postgres:10-alpine
volumes:
- /var/lib/postgresql/data
command: /bin/true
sidekiq:
build: .
environment:
- REDIS_URL=redis://redis:6379
depends_on:
- redis
command: "bin/bundle exec sidekiq -C config/sidekiq.yml"
redis:
image: redis:4-alpine
ports:
- "6379:6379"
web:
build: .
depends_on:
- redis
- postgres
- setup
command: bundle exec rails s -p 3000 -b '0.0.0.0'
environment:
- REDIS_URL=redis://localhost:6379
volumes:
- .:/myapp
ports:
- "3000:3000"
depends_on:
- postgres
Perhaps you need to add myapp volume for sidekiq as well like this:
sidekiq:
volumes:
- .:/myapp

Resources