NGINX vs OpenResty Cache Performance - docker

Question

I have a simple NGINX proxy configured, with some simple caching, and it's performance is behaving oddly in OpenResty vs vanilla NGINX.
Under load testing (300rpm) the vanilla NGINX works just fine, however, the moment I switch the from NGINX over to OpenResty, I get a portion of requests which suddenly hang, unresponsive, taking 20+ seconds to return.
My nginx.conf looks as follows:
events {
worker_connections 1024;
}
http {
proxy_cache_path /var/cache keys_zone=pagecache:10m;
server {
listen 80;
location / {
return 301 https://$host$request_uri;
}
}
server {
listen 443 ssl;
ssl_certificate /etc/ssl/mycert.pem;
ssl_certificate_key /etc/ssl/mycert.key;
location / {
proxy_cache pagecache;
proxy_cache_key $host$request_uri;
proxy_cache_lock on;
proxy_pass http://ssl-proxy-test.s3-website-eu-west-1.amazonaws.com/;
add_header X-Cache-Status $upstream_cache_status;
}
}
}
My Dockerfile for NGINX looks like this:
FROM nginx
COPY certificates /etc/ssl
COPY nginx.conf /etc/nginx/nginx.conf
And for OpenResty looks like this:
FROM openresty/openresty:buster
COPY certificates /etc/ssl
COPY nginx.conf /usr/local/openresty/nginx/conf/nginx.conf
I've tried this on several OpenResty builds (buster, bionic, xenial), and get the same results on each.
The slow requests do however return 304 with a Cache-Status: HIT header, and don't appear to make it through to the upstream server, which makes me think the bottleneck must be while reading the cached data from memory/disk? Rather than coming from upstream.
I'm new to OpenResty, so am not entirely sure how much it differs in respect to vanilla NGINX concerning its cache behaviour.
Any advice on where to start debugging this? Or what might be the cause?

Answer 1

After trying my load tests on some different infrastructure I found that this problem only seemed to occur on AWS Elastic Container Service.
Switching to a Docker image which is based on Centos/Amazon Linux seemed to get things working much more consistently.
Still a little unsure as to the real cause, but at least have something working.