After I solvied H13 errors during scale down events I realized I am getting also buch of 502 Bad Gateway HTTP errors during scale up events.
I am running container with Docker image on Heroku containing Django application running with supervisord/nginx/gunicorn/pgbouncer
I am getting these few (e.g. 9) errors during every scale up:
This can be found in Logentries when I search for status>=500:
13 Nov 2022 12:06:49.796373 <158>1 2022-11-13T11:06:49.573611+00:00 heroku router - - at=info method=GET path="/api/v1/enums/rating-types/?show_in_gallery=True&is_user_editable=True" host=www.blenderkit.com request_id=327c9a22-b5ee-4118-90f2-283bbdae5124 fwd="xxxxxxxx,172.68.254.103" dyno=web.2 connect=0ms service=0ms status=502 bytes=697 protocol=httpsException
13 Nov 2022 12:06:49.828483 <158>1 2022-11-13T11:06:49.574134+00:00 heroku router - - at=info method=GET path="/api/v1/search/?query=+asset_type:material+order:-last_upload&dict_parameters=1&page_size=15&addon_version=3.2.0&blender_version=3.3.1&scene_uuid=8c4fa678-e970-44df-bf3d-c4db2f76da33" host=www.blenderkit.com request_id=aff26f77-00c0-4ae3-92a0-47c6caee0cb4 fwd="xxxxxxxx,172.68.210.10" dyno=web.2 connect=0ms service=0ms status=502 bytes=295 protocol=httpsException
13 Nov 2022 12:06:49.903304 <158>1 2022-11-13T11:06:49.672778+00:00 heroku router - - at=info method=HEAD path="/" host=www.blenderkit.com request_id=bad94daa-967d-4fc1-89f1-abf90e935b99 fwd="xxxxxxxx,172.70.82.185" dyno=web.2 connect=0ms service=0ms status=502 bytes=145 protocol=httpsException
13 Nov 2022 12:06:49.958305 <158>1 2022-11-13T11:06:49.748174+00:00 heroku router - - at=info method=HEAD path="/" host=www.blenderkit.com request_id=d73be07c-a9e8-4d9b-8895-5449b7c12376 fwd="xxxxxxx,162.158.170.37" dyno=web.2 connect=0ms service=0ms status=502 bytes=145 protocol=httpsException
13 Nov 2022 12:06:50.083388 <158>1 2022-11-13T11:06:49.931239+00:00 heroku router - - at=info method=GET path="/comments/api/assets-uuidasset/xxxxxxx/count/?_=1668337010" host=www.blenderkit.com request_id=c20cf157-19c5-43cd-95e9-e55b31c1c6d4 fwd="xxxx,172.70.111.52" dyno=web.2 connect=0ms service=0ms status=502 bytes=295 protocol=httpsException
13 Nov 2022 12:06:50.091396 <158>1 2022-11-13T11:06:49.740483+00:00 heroku router - - at=info method=GET path="/api/v1/enums/rating-types/?show_in_gallery=True&is_user_editable=True" host=www.blenderkit.com request_id=84ac1bee-7bdb-43c0-91a6-60a6f901bf64 fwd="xxxxxx,172.70.242.110" dyno=web.2 connect=0ms service=0ms status=502 bytes=697 protocol=httpsException
13 Nov 2022 12:06:50.099396 <158>1 2022-11-13T11:06:49.868807+00:00 heroku router - - at=info method=GET path="/api/v1/enums/rating-types/?show_in_gallery=True&is_user_editable=True" host=www.blenderkit.com request_id=1374d6d0-ca09-4f38-a45d-1d82dbb60a86 fwd="2003:c7:7f21:5f75:c0c9:bbbd:73f1:4975,172.70.242.110" dyno=web.2 connect=0ms service=0ms status=502 bytes=697 protocol=httpsException
13 Nov 2022 12:06:50.127304 <158>1 2022-11-13T11:06:49.961583+00:00 heroku router - - at=info method=HEAD path="/" host=www.blenderkit.com request_id=9a174072-f93f-4220-afb7-8378c84c0937 fwd="xxxxxx,172.71.178.99" dyno=web.2 connect=0ms service=0ms status=502 bytes=145 protocol=httpsException
13 Nov 2022 12:06:50.133395 <158>1 2022-11-13T11:06:49.99942+00:00 heroku router - - at=info method=GET path="/api/v1/enums/rating-types/?show_in_gallery=True&is_user_editable=True" host=www.blenderkit.com request_id=abb84ce4-f4c8-4226-a23f-32d7310007c8 fwd="xxxxx,172.70.242.110" dyno=web.2 connect=0ms service=0ms status=502 bytes=697 protocol=httpsException
13 Nov 2022 12:06:50.151358 <158>1 2022-11-13T11:06:49.783907+00:00 heroku router - - at=info method=GET path="/avatar-redirect/xxxxxxxxx/128/" host=www.blenderkit.com request_id=44e69bb9-2ae4-4d20-a03d-eb71a370f364 fwd="xxxxxxxxxx,172.68.66.87" dyno=web.2 connect=0ms service=0ms status=502 bytes=295 protocol=httpsException
13 Nov 2022 12:06:50.402302 <158>1 2022-11-13T11:06:50.033443+00:00 heroku router - - at=info method=HEAD path="/" host=www.blenderkit.com request_id=810db70f-8b56-404b-befb-456424aacdce fwd="xxxxxxxx,172.70.91.43" dyno=web.2 connect=0ms service=0ms status=502 bytes=145 protocol=httpsException
13 Nov 2022 12:06:55.828359 <158>1 2022-11-13T11:06:55.672729+00:00 heroku router - - at=info method=GET path="/avatar-redirect/xxxxxxxxx/128/" host=www.blenderkit.com request_id=8e090a61-7aae-466a-a22a-d2ed5947a260 fwd="xxxxx,172.71.130.55" dyno=web.4 connect=0ms service=0ms status=502 bytes=295 protocol=httpsException
13 Nov 2022 12:06:55.994399 <158>1 2022-11-13T11:06:55.618393+00:00 heroku router - - at=info method=GET path="/comments/api/assets-uuidasset/xxxxxxxx/count/?_=1668337016" host=www.blenderkit.com request_id=41af736c-c990-42fb-b39b-681a139cce9a fwd="xxxx,172.70.242.61" dyno=web.4 connect=1ms service=0ms status=502 bytes=295 protocol=httpsException
Here is how it looks in Heroku Metrics:
I realized that the problem is exact opposite of the problem with H13 errors during scaledown. The problem is, that Nginx started sooner than rest of the applications.
The solution here is simple, delaying Nginx startup through Supervisord configuration:
[program:nginx]
command=bash -c "sleep 5 && /usr/local/nginx/sbin/nginx -c /etc/nginx/nginx.conf"
Related
I try to run my simple Web-App as Docker Container within Heroku.
When build and run the container locally all works fine.
But on accessing the generated endpoint https://html5-landingpage-buddah.herokuapp.com/ Heroku responses with HTTP 400 Bad Request and the message "Not authoritative"
Logs inside Heroku don't help much:
2020-09-19T14:45:45.269487+00:00 heroku[router]: at=info method=GET path="/" host=html5-landingpage-buddah.herokuapp.com request_id=5f07ea7e-d925-4fbe-b7ba-29387a3284cc fwd="XX.XX.XX.XXX" dyno=web.1 connect=1ms service=1ms status=400 bytes=212 protocol=https
I did a research for the error but found no helpful information.
Also re-creating the App in Heroku, renaming the URL and moving the App to another Region did not help.
Here is the log from restart to the first request on the web app:
2021-06-15T17:11:36.977277+00:00 heroku[web.1]: Starting process with command `./app`
2021-06-15T17:11:39.852345+00:00 app[web.1]: [main] INFO de.clique.westwood.example.html5.landingpage.buddah.App - Server started on port 54225
2021-06-15T17:11:39.852374+00:00 app[web.1]: [main] INFO de.clique.westwood.example.html5.landingpage.buddah.App - Serving albums from ./static/album
2021-06-15T17:11:41.363857+00:00 heroku[web.1]: State changed from starting to up
2021-06-15T17:11:48.162254+00:00 heroku[router]: at=info method=GET path="/" host=html5-landingpage-buddah.herokuapp.com request_id=85c642cb-a48d-41ac-89e1-7a520749e9bc fwd="94.31.82.142" dyno=web.1 connect=1ms service=4ms status=421 bytes=219 protocol=https
2021-06-15T17:11:48.275223+00:00 heroku[router]: at=info method=GET path="/" host=html5-landingpage-buddah.herokuapp.com request_id=406e9e77-f40e-47b9-a33b-af0af5b8f171 fwd="94.31.82.142" dyno=web.1 connect=1ms service=3ms status=421 bytes=219 protocol=https
2021-06-15T17:11:48.520876+00:00 heroku[router]: at=info method=GET path="/favicon.ico" host=html5-landingpage-buddah.herokuapp.com request_id=d3782734-765c-45f1-98a8-04e2098e7267 fwd="94.31.82.142" dyno=web.1 connect=1ms service=4ms status=421 bytes=219 protocol=https
2021-06-15T17:11:48.906097+00:00 heroku[router]: at=info method=GET path="/favicon.ico" host=html5-landingpage-buddah.herokuapp.com request_id=79e4404f-e505-4c16-9778-9f72882d74be fwd="94.31.82.142" dyno=web.1 connect=1ms service=4ms status=421 bytes=219 protocol=https
The error Not Authoritative simply means that Heroku is unable to redirect the traffic from the said domain name to your application.
According to Heroku,
The web process must listen for HTTP traffic on $PORT, which is set by Heroku. EXPOSE in Dockerfile is not respected, but can be used for local testing. Only HTTP requests are supported.
You will need to replace your EXPOSE and ENTRYPOINT as such
CMD [ "sh", "-c", "java -jar ./app --bind 0.0.0.0:$PORT" ]
I have been learning rails for about a month and I have created my first simple static website using ruby on rails. it functions well on localhost:3000.
I have pushed all the files onto GitHub and have followed several Youtube videos on how to deploy the site using GitHub and heroku. I have used them to work through several errors I got and finally managed to "deploy" the site.
However, all I get is a kind of welcome page that says it is my site. it is just a single page with nothing but "Heroku. Welcome to your new app." Written on it. What is the final stage I am missing? why am I not seeing my website?
Below are the application logs:
2017-03-22T20:20:58.057644+00:00 app[api]: Release v1 created by user *******#******.com
2017-03-22T20:20:58.063330+00:00 app[api]: Release v2 created by user *******#******.com
2017-03-22T20:20:58.189041+00:00 app[api]: Release v2 created by user *******#******.com
2017-03-22T20:31:55.915631+00:00 heroku[router]: at=info code=H81 desc="Blank app" method=GET path="/" host=vast-shelf-39597.herokuapp.com request_id=d69fedea-d149-4836-b349-aa490c3b9654 fwd="88.11.201.253" dyno= connect= service= status=502 bytes= protocol=https
2017-03-22T20:31:56.290278+00:00 heroku[router]: at=info code=H81 desc="Blank app" method=GET path="/favicon.ico" host=vast-shelf-39597.herokuapp.com request_id=0c5752c0-b960-443e-8f07-0c2346055ff9 fwd="88.11.201.253" dyno= connect= service= status=502 bytes= protocol=https
2017-03-24T18:44:37.424270+00:00 heroku[router]: at=info code=H81 desc="Blank app" method=GET path="/" host=vast-shelf-39597.herokuapp.com request_id=62e3ddce-34e5-4fb2-892b-2ae9cf5ca93d fwd="88.11.201.253" dyno= connect= service= status=502 bytes= protocol=https
2017-03-24T18:44:37.811237+00:00 heroku[router]: at=info code=H81 desc="Blank app" method=GET path="/favicon.ico" host=vast-shelf-39597.herokuapp.com request_id=7a5b7d99-1b83-406d-bdc2-c68c3d5efa33 fwd="88.11.201.253" dyno= connect= service= status=502 bytes= protocol=https
2017-03-24T18:44:55.237261+00:00 heroku[router]: at=info code=H81 desc="Blank app" method=GET path="/" host=vast-shelf-39597.herokuapp.com request_id=4057726f-e2b9-464b-a970-b37306051e63 fwd="88.11.201.253" dyno= connect= service= status=502 bytes= protocol=https
2017-03-25T20:29:38.837541+00:00 heroku[router]: at=info code=H81 desc="Blank app" method=GET path="/" host=vast-shelf-39597.herokuapp.com request_id=47282fae-e30f-43d1-be52-808c956cad7b fwd="88.11.201.253" dyno= connect= service= status=502 bytes= protocol=https
2017-03-25T21:10:30.120292+00:00 heroku[router]: at=info code=H81 desc="Blank app" method=GET path="/" host=vast-shelf-39597.herokuapp.com request_id=94863f09-962f-49d5-90db-c0fbf7ac0c8a fwd="88.11.201.253" dyno= connect= service= status=502 bytes= protocol=https
2017-03-26T11:07:07.914421+00:00 heroku[router]: at=info code=H81 desc="Blank app" method=GET path="/" host=vast-shelf-39597.herokuapp.com request_id=c3fac4f9-8786-493c-a859-05690842ca89 fwd="88.11.201.253" dyno= connect= service= status=502 bytes= protocol=https
Here is my routes file: I think there is something missing with "root", no?
Rails.application.routes.draw do
get 'handm/home'
get 'handm/about'
get 'handm/contact'
get 'handm/industries'
get 'handm/languages'
get 'handm/services'
Looks like the deploy is fine.
You have to set the "root" route of your application. The root route is the route it will go to when a user first visits you app.
I am assuming home is the route you want for the first one, if not just change the route file as needed.
Change this
get 'handm/home'
to this
root 'handm#home'
Make sure to push to git and then deploy to Heroku.
I have a Rails app deployed to Heroku. It currently has a 2x web dyno, and a 1x worker dyno. No active users at this point other than myself.
When the worker dyno is running, I get frequent H12 errors on app/web.1. When the worker stops, the errors stop.
Within the app, a user can upload a CSV file, and then DelayedJob runs a service on a worker that parses the CSV file, and saves Contact, EmailAddress and PhoneNumber objects to the database. There's some deduplication and some merging logic as well.
The worker runs flawlessly. But while it's running, I have a lot of trouble using the app. Most attempts to load a page end up timing out, and I see entries like this in the log:
Mar 31 10:52:54 abb heroku/router: at=error code=H12 desc="Request timeout" method=GET path="/contacts" host=abb.herokuapp.com request_id=a061e6b1-0233-4be1-9653-5db023d339ac fwd="99.17.86.166" dyno=web.1 connect=17ms service=30000ms status=503 bytes=0
Mar 31 10:53:18 abb heroku/router: at=error code=H12 desc="Request timeout" method=GET path="/" host=abb.herokuapp.com request_id=73d1e9e4-2e93-4e63-a86e-c8f90d22d172 fwd="50.250.42.142" dyno=web.1 connect=1ms service=30000ms status=503 bytes=0
Mar 31 10:53:39 abb heroku/router: at=error code=H12 desc="Request timeout" method=POST path="/contacts" host=abb.herokuapp.com request_id=83753eec-774b-4939-8ad5-3b5876c14151 fwd="99.17.86.166" dyno=web.1 connect=1ms service=30012ms status=503 bytes=0
Mar 31 10:53:49 abb heroku/router: at=error code=H12 desc="Request timeout" method=GET path="/favicon.ico" host=abb.herokuapp.com request_id=575492b9-e997-4be5-8044-1be37ec2eda6 fwd="50.250.42.142" dyno=web.1 connect=1ms service=30000ms status=503 bytes=0
I'm using Ruby 2.3, Rails 4.2.5.1, Postgres for the database, and I have the same problems with both Unicorn and Puma. Any ideas why the worker causing these issues?
Update
I discovered that I'm getting the same problem when I run the app locally in both development and production modes. When the worker is processing a CSV file, the app hangs.
App running Ruby 2.2.2, Rails 4.2.1
When I run my app locally everything is fine. It was running fine on heroku until I made some updates. Now all I get in a browser is "Err too many redirects" from Chrome, or "Too many redirects..." in Safari. Even after I rolled back the version on heroku and reset the database I am still getting this. I have wiped out the cookies and cache on both browsers too. Still the error.
I get a bunch of these one after the other in the heroku log:
2015-06-14T15:09:23.160000+00:00 heroku[router]: at=info method=GET path="/" host=www.domain.com request_id=28455761-bc94-4a27-852f-249f5409a4f2 fwd="50.152.172.74,199.27.128.248" dyno=web.1 connect=1ms service=1ms status=301 bytes=157
Followed by one of these:
2015-06-14T16:11:18.084247+00:00 heroku[router]: at=info method=POST path="/payola/events" host=www.domain.com request_id=2ab49dd7-0392-4261-ae1e-1279998e37e1 fwd="54.241.34.107,199.27.133.164" dyno=web.1 connect=0ms service=1ms status=301 bytes=170
And then this before it shuts down:
2015-06-14T16:39:40.305372+00:00 heroku[router]: at=info method=GET path="/" host=www.domain.com request_id=a54f6089-ca85-428f-ac01-0c02421a83da fwd="50.152.172.74,199.27.128.73" dyno=web.1 connect=1ms service=1ms status=301 bytes=157
2015-06-14T16:39:40.498317+00:00 heroku[router]: at=info method=GET path="/" host=www.domain.com request_id=1f0aa9d4-7ae1-4141-ab32-7103aa0c1ef5 fwd="50.152.172.74,199.27.128.73" dyno=web.1 connect=0ms service=1ms status=301 bytes=157
2015-06-14T16:51:40.945338+00:00 heroku[router]: at=info method=GET path="/robots.txt" host=www.domain.com request_id=6a37a7d4-3277-4f55-9321-bdab79a744ff fwd="68.180.228.91,108.162.246.80" dyno=web.1 connect=3ms service=4ms status=301 bytes=167
2015-06-14T16:51:41.287899+00:00 heroku[router]: at=info method=GET path="/pricing" host=www.domain.com request_id=433b98e3-d445-47f3-b9b3-212dff1dfb57 fwd="68.180.228.91,108.162.246.226" dyno=web.1 connect=2ms service=8ms status=301 bytes=164
2015-06-14T16:56:30.228037+00:00 heroku[router]: at=info method=POST path="/payola/events" host=www.domain.com request_id=9de5611a-4cc4-474e-a4cf-26dfd1de31b3 fwd="54.241.34.107,199.27.133.114" dyno=web.1 connect=6ms service=2ms status=301 bytes=170
2015-06-14T17:58:38.217217+00:00 heroku[router]: at=info method=POST path="/payola/events" host=www.domain.com request_id=48a6417f-69f4-4ed2-b468-bf0edfae0c90 fwd="54.241.34.107,199.27.133.189" dyno=web.1 connect=0ms service=1ms status=301 bytes=170
2015-06-14T18:32:20.867295+00:00 heroku[router]: at=info method=POST path="/payola/events" host=www.domain.com request_id=276fc89e-3e07-441b-ad00-f2b60c77e540 fwd="54.241.34.107,199.27.133.189" dyno=web.1 connect=0ms service=1ms status=301 bytes=170
2015-06-14T19:05:57.170756+00:00 heroku[router]: at=info method=GET path="/robots.txt" host=www.domain.com request_id=22174ed2-db0a-45bb-b9d7-0a6f2ba2ac8c fwd="208.115.113.85,108.162.246.205" dyno=web.1 connect=4ms service=15ms status=301 bytes=167
2015-06-14T19:44:10.474487+00:00 heroku[router]: at=info method=POST path="/payola/events" host=www.domain.com request_id=bb575e60-1017-41c7-869e-31c0f3ec2528 fwd="54.241.34.107,199.27.133.164" dyno=web.1 connect=1ms service=2ms status=301 bytes=170
2015-06-14T20:46:06.583978+00:00 heroku[router]: at=info method=POST path="/payola/events" host=www.domain.com request_id=c348fef6-6ada-4825-a9ca-3631840d8e51 fwd="54.241.31.99,199.27.133.53" dyno=web.1 connect=8ms service=5ms status=301 bytes=170
Does anyone know what is causing this?
I discovered the answer. I am on Cloudflare's network. Turns out that if on their "Crypto" panel, if the SSL is set to "flexible" then you will get the redirect loop error. Had to set it to "Full".
I have an app
http://buzzme.herokuapp.com/
but when I click sign in: http://buzzme.herokuapp.com/clients/sign_in
It gives me an error We're sorry, but something went wrong.
But heroku logs don't say anything:
=0ms service=15ms status=500 bytes=1754
2015-05-03T02:38:27.676212+00:00 heroku[router]: at=info method=GET path="/clients/sign_in" host=buzzme.herokuapp.com request_id=38444765-b318-4719-8017-ff9fc9dd5031 fwd="207.41.179.38" dyno=web.1 connect=0ms service=17ms status=500 bytes=1754
2015-05-03T02:38:27.715718+00:00 heroku[router]: at=info method=GET path="/clients/sign_in" host=buzzme.herokuapp.com request_id=49cba466-6bf5-4413-b25e-aeec7a0bac57 fwd="207.41.179.38" dyno=web.1 connect=4ms service=17ms status=500 bytes=1754
2015-05-03T02:39:06.600569+00:00 heroku[router]: at=info method=POST path="/clients/sign_in" host=buzzme.herokuapp.com request_id=fd6acefb-22ff-4e49-a50b-5a70f21f25de fwd="207.41.179.38" dyno=web.1 connect=1ms service=9ms status=422 bytes=328
How would I debug this? I already changed the log level to debug but it doesn't say anything.