I have a server running Ubuntu 20.04, nginx and varnish with letsencrypt SSL.
I have installed PHPMyAdmin to the default server block and have another server block for my actual website running magento 2.4.3. This was all working fine until i installed varnish and now when trying to load either site i get the 503 Backend fetch failed error and when i try and load then on port 8080 from the URL i get a "this page cannot be reach" error.
I have configured nginx to listen on port 8080 for both sites, i have set varnish to listen on port 80. I have edited the vcl generated by magento to set the host and port to 127.0.0.1 and 8080 respectively as shown on https://devdocs.magento.com/guides/v2.4/config-guide/varnish/config-varnish-configure.html.
The varnishlog show the backend is unhealthy error but i dont know how to solve this.
The output of netstat -tulpn is:
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22007 0.0.0.0:* LISTEN 48993/sshd: /usr/sb
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 302464/nginx: maste
tcp 0 0 127.0.0.1:6082 0.0.0.0:* LISTEN 304955/varnishd
tcp 0 0 0.0.0.0:25672 0.0.0.0:* LISTEN 42533/beam.smp
tcp 0 0 127.0.0.1:6379 0.0.0.0:* LISTEN 44657/redis-server
tcp 0 0 127.0.0.1:9100 0.0.0.0:* LISTEN 14734/noderig
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 304955/varnishd
tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN 302464/nginx: maste
tcp 0 0 0.0.0.0:21 0.0.0.0:* LISTEN 162503/pure-ftpd (S
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN 843/systemd-resolve
tcp6 0 0 :::22007 :::* LISTEN 48993/sshd: /usr/sb
tcp6 0 0 :::443 :::* LISTEN 302464/nginx: maste
tcp6 0 0 :::5672 :::* LISTEN 42533/beam.smp
tcp6 0 0 :::3306 :::* LISTEN 158505/mysqld
tcp6 0 0 ::1:6379 :::* LISTEN 44657/redis-server
tcp6 0 0 :::80 :::* LISTEN 304955/varnishd
tcp6 0 0 :::8080 :::* LISTEN 302464/nginx: maste
As you can see, varnishd is running on port 80 and nginx on 8080 but its almost as if port 8080 isnt listening. I have ufw enabled and "Nginx Full" is set to allow and i have also tried adding a rule to allow 8080 but this made no difference. Disabling varnish and trying again to load both sites on port 8080 results in the same "page cannot be reached" error so i dont believe this to be a varnish issue, but an nginx one, but why isnt it listen on port 8080?
If i try curl from the local server accessing localhost:8080 for example, it works and loads the default nginx page as expected. My PHPMyAdmin install is in a folder and that also works by using curl to access localhost/phpmyadmin. Does this mean it is then a firewall issue? Disabling ufw didnt help.
The start of my server block for my magento site is:
listen 8080;
listen [::]:8080;
server_name mydomain.com;
return 404; # managed by Certbot
set $MAGE_ROOT /var/www/mydomain.com;
access_log /var/log/nginx/mydomain.com.log;
error_log /var/log/nginx/mydomain.com.err;
include /var/www/mydomain.com/nginx.conf;
EDIT: adding port 8080 to ufw does now allow my phpmyadmin to work when connecting to port 8080, but i still get a 503 error when connecting to anything on port 80 with varnish enabled.
Output of sudo varnishlog -g raw -i backend_health:
varnishlog -g raw -i backend_health
0 Backend_health - default Still sick 4---X-R- 0 5 10 0.000624 0.000000 HTTP/1.1 404 Not Found
0 Backend_health - default Still sick 4---X-R- 0 5 10 0.000621 0.000000 HTTP/1.1 404 Not Found
0 Backend_health - default Still sick 4---X-R- 0 5 10 0.000652 0.000000 HTTP/1.1 404 Not Found
0 Backend_health - default Still sick 4---X-R- 0 5 10 0.000210 0.000000 HTTP/1.1 404 Not Found
0 Backend_health - default Still sick 4---X-R- 0 5 10 0.000214 0.000000 HTTP/1.1 404 Not Found
The output for command sudo varnishlog -g request -q "RespStatus == 503":
sudo varnishlog -g request -q "RespStatus == 503"
* << Request >> 11
- Begin req 10 rxreq
- Timestamp Start: 1641551164.244422 0.000000 0.000000
- Timestamp Req: 1641551164.244422 0.000000 0.000000
- VCL_use boot
- ReqStart 127.0.0.1 39792 a0
- ReqMethod GET
- ReqURL /
- ReqProtocol HTTP/1.0
- ReqHeader Host: staging.clicksaveandprint.com
- ReqHeader X-Forwarded-Host: staging.clicksaveandprint.com
- ReqHeader X-Real-IP: 81.107.192.226
- ReqHeader X-Forwarded-For: 81.107.192.226
- ReqHeader Ssl-Offloaded: 1
- ReqHeader X-Forwarded-Proto: https
- ReqHeader X-Forwarded-Port: 443
- ReqHeader X-Forwarded-Proto: https
- ReqHeader Connection: close
- ReqHeader Cache-Control: max-age=0
- ReqHeader Upgrade-Insecure-Requests: 1
- ReqHeader User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36
- ReqHeader Accept: text/html,application/xhtml xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
- ReqHeader Sec-GPC: 1
- ReqHeader Sec-Fetch-Site: none
- ReqHeader Sec-Fetch-Mode: navigate
- ReqHeader Sec-Fetch-User: ?1
- ReqHeader Sec-Fetch-Dest: document
- ReqHeader Accept-Encoding: gzip, deflate, br
- ReqHeader Accept-Language: en-GB,en-US;q=0.9,en;q=0.8
- ReqHeader Cookie: __tawkuuid=e::clicksaveandprint.com::n8RwZ5uipY2iBtEtSiR wc7TP1jlMwD51gtYez PiWZ4Xr/t3VxVX/BMIhUzfv0U::2; _ga=GA1.2.428376681.1618304758; _ga_01K4WXXY6B=GS1.1.1641392250.103.1.1641392407.60; form_key=VKzAhiJUg3HyT7jg; AMCVS_8F99160E571FC0427F0001
- ReqUnset X-Forwarded-For: 81.107.192.226
- ReqHeader X-Forwarded-For: 81.107.192.226, 127.0.0.1
- VCL_call RECV
- ReqHeader grace: none
- ReqURL /
- ReqUnset Accept-Encoding: gzip, deflate, br
- ReqHeader Accept-Encoding: gzip
- VCL_return hash
- VCL_call HASH
- VCL_return lookup
- VCL_call MISS
- VCL_return fetch
- Link bereq 12 fetch
- Timestamp Fetch: 1641551164.245036 0.000614 0.000614
- RespProtocol HTTP/1.1
- RespStatus 503
- RespReason Backend fetch failed
- RespHeader Date: Fri, 07 Jan 2022 10:26:04 GMT
- RespHeader Server: Varnish
- RespHeader content-type: text/html; charset=utf-8
- RespHeader Retry-After: 5
- RespHeader X-Varnish: 11
- RespHeader Age: 0
- RespHeader Via: 1.1 varnish (Varnish/6.2)
- VCL_call DELIVER
- RespUnset Age: 0
- RespHeader Pragma: no-cache
- RespHeader Expires: -1
- RespHeader Cache-Control: no-store, no-cache, must-revalidate, max-age=0
- RespUnset Server: Varnish
- RespUnset X-Varnish: 11
- RespUnset Via: 1.1 varnish (Varnish/6.2)
- VCL_return deliver
- Timestamp Process: 1641551164.245126 0.000704 0.000090
- Filters
- RespHeader Content-Length: 279
- RespHeader Connection: close
- Timestamp Resp: 1641551164.245228 0.000807 0.000103
- ReqAcct 1233 0 1233 264 279 543
- End
** << BeReq >> 12
-- Begin bereq 11 fetch
-- VCL_use boot
-- Timestamp Start: 1641551164.244718 0.000000 0.000000
-- BereqMethod GET
-- BereqURL /
-- BereqProtocol HTTP/1.0
-- BereqHeader Host: staging.clicksaveandprint.com
-- BereqHeader X-Forwarded-Host: staging.clicksaveandprint.com
-- BereqHeader X-Real-IP: 81.107.192.226
-- BereqHeader Ssl-Offloaded: 1
-- BereqHeader X-Forwarded-Proto: https
-- BereqHeader X-Forwarded-Port: 443
-- BereqHeader X-Forwarded-Proto: https
-- BereqHeader Upgrade-Insecure-Requests: 1
-- BereqHeader User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36
-- BereqHeader Accept: text/html,application/xhtml xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
-- BereqHeader Sec-GPC: 1
-- BereqHeader Sec-Fetch-Site: none
-- BereqHeader Sec-Fetch-Mode: navigate
-- BereqHeader Sec-Fetch-User: ?1
-- BereqHeader Sec-Fetch-Dest: document
-- BereqHeader Accept-Language: en-GB,en-US;q=0.9,en;q=0.8
-- BereqHeader Cookie: __tawkuuid=e::clicksaveandprint.com::n8RwZ5uipY2iBtEtSiR wc7TP1jlMwD51gtYez PiWZ4Xr/t3VxVX/BMIhUzfv0U::2; _ga=GA1.2.428376681.1618304758; _ga_01K4WXXY6B=GS1.1.1641392250.103.1.1641392407.60; form_key=VKzAhiJUg3HyT7jg; AMCVS_8F99160E571FC0427F0001
-- BereqHeader X-Forwarded-For: 81.107.192.226, 127.0.0.1
-- BereqHeader grace: none
-- BereqHeader Accept-Encoding: gzip
-- BereqProtocol HTTP/1.1
-- BereqHeader X-Varnish: 12
-- VCL_call BACKEND_FETCH
-- VCL_return fetch
-- FetchError backend default: unhealthy
-- Timestamp Beresp: 1641551164.244824 0.000107 0.000107
-- Timestamp Error: 1641551164.244834 0.000116 0.000010
-- BerespProtocol HTTP/1.1
-- BerespStatus 503
-- BerespReason Service Unavailable
-- BerespReason Backend fetch failed
-- BerespHeader Date: Fri, 07 Jan 2022 10:26:04 GMT
-- BerespHeader Server: Varnish
-- VCL_call BACKEND_ERROR
-- BerespHeader content-type: text/html; charset=utf-8
-- BerespHeader Retry-After: 5
-- VCL_return deliver
-- Storage malloc Transient
-- Length 279
-- BereqAcct 0 0 0 0 0 0
-- End
CodePudding user response：
Try to increase first_byte_timeout parameter like this:
backend www {
...
.first_byte_timeout = 100s;
...
}
CodePudding user response：
Can you please provide the output of the following command?
sudo varnishlog -g raw -i backend_health
This command will provide logging information on the status of the health probe checks.
The fact that you get a 503 Backend fetch failed probably means the backend is not available.
If the backend health logging is inconclusive, we can always look at the transactions that resulted in that 503 Backend fetch failed error.
You can run the following command for that:
sudo varnishlog -g request -q "RespStatus == 503"
Please provide some logging output:
- A couple of lines from the backend health logging command
- A full log transaction for a request that triggered the
503 Backend fetch failed