Celery + rabbitmq RPC result return failure - docker

Question

I'm using Celery 5.2.7 + rabbitmq 3.9-management for RPC. My applications are separated into 3 different containers: worker, rabbitmq, and API which makes RPC call.
All containers are deployed via single docker-compose and connected to a bridge network. Whole cluster works on a single machine under Ubuntu 20.04.3 LTS.
Work scenario is the following:
API receives request from user
API puts a job for gpu worker to rabbitmq and waits for response.
GPU worker listens to task queue and performs computations when the new task occurs
When computations are complete, GPU worker sends results back (via another rabbitmq queue, I suppose)
My problem is that API never gets its results back. As I can see in logs, my gpu worker gets the job done in <0.1 seconds, but my API never gets results
Setup:
Worker:
#celery_worker.py
app = Celery('celery_worker', broker= f'amqp://{rabbitmq_username}:{rabbitmq_password}#{rabbitmq_host}:{rabbitmq_port}/', backend='rpc://')
#app.task
def remote_procedure(a, b):
#some heavy computations
return res
API main:
#API main.py
from proj.celery_worker import remote_procedure
RESPONSE_TIMEOUT_SECONDS = 1.5
#app.get('/search/')
def search(a,b):
try:
res = remote_procedure.apply_async((a,b), expires = RESPONSE_TIMEOUT_SECONDS-0.3, retry=False).get(timeout=RESPONSE_TIMEOUT_SECONDS)
except celery.exceptions.TimeoutError:
print('timeout')
return {"res":"timeout"}
return {"res":res}
def main():
"""
Main function
"""
uvicorn.run(app, host='127.0.0.1', port=CONFIG['port'])
if __name__ == '__main__':
main()
Compose:
gpu_worker:
build:
context: .
dockerfile: Dockerfile.worker
deploy:
resources:
limits:
memory: 10gb
reservations:
devices:
- capabilities: [ gpu ]
networks:
- search_net
api:
build:
context: .
dockerfile: Dockerfile.api
depends_on:
- gpu_worker
ports:
- "8001:8000"
environment:
MAX_WORKERS: 32
deploy:
resources:
limits:
memory: 10gb
reservations:
devices:
- capabilities: [ gpu ]
networks:
- search_net
rabbitmq:
image: rabbitmq:3.9-management
hostname: rabbitmq
container_name: 'rabbitmq'
ports:
- 5672:5672
- 15672:15672
volumes:
- rabbitmq_data:/var/lib/rabbitmq/
- rabbitmq_log:/var/log/rabbitmq/
restart: on-failure
networks:
- search_net
volumes:
rabbitmq_data:
rabbitmq_log:
networks:
search_net:
driver: bridge
Dockerfile.worker:
...
ENTRYPOINT [ "celery", "-A", "proj.celery_worker", "worker", "--loglevel=DEBUG", "--pool=solo", "-c", "1"]
Dockerfile.api:
...
EXPOSE 80
# Start the app
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]
I'm not setting any other params, so all other parameters or configurations must be default.
I'm not sure, but it seems to me that there is some kind of trouble with returning results. Is it possible that there are networking troubles inside the container?