Varnish stops waiting after 1 second - timeout

Question

this problem is gonna make me crazy: my varnish istance stops waiting for a backend response after exactly 1 second.
Every first call to a page is a 503 Backend
Daemon is configured this way:
DAEMON_OPTS="-a :80 \
-T localhost:6082 \
-f /etc/varnish/default.vcl \
-S /etc/varnish/secret \
-p thread_pool_add_delay=2 \
-p thread_pools=4 \
-p thread_pool_min=200 \
-p thread_pool_max=4000 \
-p timeout_linger=50 \
-p connect_timeout=300 \
-p first_byte_timeout=300 \
-p between_bytes_timeout=300 \
-p send_timeout=900 \
-s malloc,3G"
and the VCL backend:
backend default { # Define one backend
.host = "127.0.0.1"; # IP or Hostname of backend
.port = "8080"; # Port Apache or whatever is listening
.probe = {
.url = "/";
.timeout = 1s;
.interval = 1s;
.window = 10;
.threshold = 8;
}
.first_byte_timeout = 60s; # How long to wait before we receive a first byte from our backend?
.connect_timeout = 60s; # How long to wait for a backend connection?
.between_bytes_timeout = 60s; # How long to wait between bytes received from our backend?
}
Here is the one call in the log:
* << Request >> 3440734
- Begin req 3440733 rxreq
- Timestamp Start: 1462781837.623325 0.000000 0.000000
- Timestamp Req: 1462781837.623325 0.000000 0.000000
- ReqStart 10.20.129.118 58572
- ReqMethod GET
- ReqURL xxxxx.html
- ReqProtocol HTTP/1.1
- ReqHeader Accept: image/jpeg, application/x-ms-application, image/gif, application/xaml+xml, image/pjpeg, application/x-ms-xbap, application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, image/pjpeg, application/x-shockwave-flash, */*
- ReqHeader Referer: http://xxxxxx.html
- ReqHeader Accept-Language: it-IT
- ReqHeader User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; Trident/7.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E)
- ReqHeader Accept-Encoding: gzip, deflate
- ReqHeader Host: xxxxxx
- ReqHeader DNT: 1
- ReqHeader Connection: Keep-Alive
- ReqHeader Cookie: fc_uid=p; __utma=127650066.830977012.1423064118.1426582505.1426588086.20; _ga=GA1.3.830977012.1423064118; _gat_UA-13041322-1=1; ZNPCQ003-38303300=71a0f671; ff607e18ab6c715f4bb35b5bbcbe1c56=d82989olp2v0ur3gpl2ouprko6; _ga=GA1.2.830977012.142306411
- ReqHeader X-Forwarded-For: 10.20.129.118
- VCL_call RECV
- ReqUnset Host: xxx
- ReqHeader Host: xxx
- ReqURL /xxxx.html
- ReqUnset Cookie: fc_uid=p; __utma=127650066.830977012.1423064118.1426582505.1426588086.20; _ga=GA1.3.830977012.1423064118; _gat_UA-13041322-1=1; ZNPCQ003-38303300=71a0f671; ff607e18ab6c715f4bb35b5bbcbe1c56=d82989olp2v0ur3gpl2ouprko6; _ga=GA1.2.830977012.142306411
- ReqHeader Cookie: fc_uid=p; __utma=127650066.830977012.1423064118.1426582505.1426588086.20; _ga=GA1.3.830977012.1423064118; _gat_UA-13041322-1=1; ZNPCQ003-38303300=71a0f671; ff607e18ab6c715f4bb35b5bbcbe1c56=d82989olp2v0ur3gpl2ouprko6; _ga=GA1.2.830977012.142306411
- ReqUnset Cookie: fc_uid=p; __utma=127650066.830977012.1423064118.1426582505.1426588086.20; _ga=GA1.3.830977012.1423064118; _gat_UA-13041322-1=1; ZNPCQ003-38303300=71a0f671; ff607e18ab6c715f4bb35b5bbcbe1c56=d82989olp2v0ur3gpl2ouprko6; _ga=GA1.2.830977012.142306411
- ReqHeader Cookie: fc_uid=p; __utma=127650066.830977012.1423064118.1426582505.1426588086.20; _ga=GA1.3.830977012.1423064118; _gat_UA-13041322-1=1; ZNPCQ003-38303300=71a0f671; ff607e18ab6c715f4bb35b5bbcbe1c56=d82989olp2v0ur3gpl2ouprko6; _ga=GA1.2.830977012.142306411
- ReqHeader Surrogate-Capability: key=ESI/1.0
- VCL_return hash
- ReqUnset Accept-Encoding: gzip, deflate
- ReqHeader Accept-Encoding: gzip
- VCL_call HASH
- VCL_return lookup
- VCL_call MISS
- VCL_return fetch
- Link bereq 3440735 fetch
- Timestamp Fetch: 1462781838.492085 0.868760 0.868760
- Timestamp Process: 1462781838.492101 0.868776 0.000016
- RespHeader Date: Mon, 09 May 2016 08:17:18 GMT
- RespHeader Server: Varnish
- RespHeader X-Varnish: 3440734
- RespProtocol HTTP/1.1
- RespStatus 503
- RespReason Service Unavailable
- RespReason Service Unavailable
- VCL_call SYNTH
- VCL_return deliver
- RespHeader Content-Length: 0
- Storage malloc Transient
- Debug "RES_MODE 2"
- RespHeader Connection: keep-alive
- Timestamp Resp: 1462781838.492145 0.868820 0.000044
- ReqAcct 985 0 985 153 0 153
- End
Any precious help? Suggestion?

Answer 1

Pretty sure the issue is here:
.timeout = 1s;
.interval = 1s;
Notice that all of them have a time of 1s = 1 second.
I think what's happening is the backend's probe timeouts so varnish flags it as "unhealthy" until your backend respond in a second or less.
Try modifying those values to check whether thats the issue or not