7

A few hours ago we starting seeing a bunch of H10 errors in our logs and on our Heroku dashboard. Heroku says that H10 errors are due to web dyno crashes. Read more here. The strange part is that our 2 web dynos are running completely fine and not crashing at all.

Here's an example of what I'm seeing in the logs:

2021-08-12T20:50:35.934439+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=POST path="/v1/graphql" host=api.rize.io request_id=abcd-ace1-44e8-99f5-03d38d8eaa5a fwd="1.1.1.1" dyno= connect= service= status=503 bytes= protocol=https

When I search for that request ID, abcd-ace1-44e8-99f5-03d38d8eaa5a in this example, in the logs (as Heroku recommends), nothing shows up except the router error listed above.

I've restarted all the dynos and the issue persists.

Anyone have any idea what could be going on and how we can fix this?

TylerH
  • 20,799
  • 66
  • 75
  • 101
Macgill4444
  • 244
  • 1
  • 12
  • 1
    I am seeing the same behavior. Intermittent H10s but app is not crashed. I was losing my mind trying to figure this one out, but I'm glad to see I'm not alone. Hoping this is an outage. – ezekg Aug 13 '21 at 12:06
  • 2
    Same problem here going on 12+ hours. No response from support. `heroku ps` shows app up and running. We've tried maintenance mode, switching dyno types, scaling down to 0 and back, redeploying code, changing environment variables. Nothing helps. – Gary Wright Aug 13 '21 at 12:26
  • Okay cool. Glad to know I'm not the only one seeing this issue. I'm going to file another ticket with Heroku to hopefully put more pressure on them to figure out what's going on. – Macgill4444 Aug 13 '21 at 13:16
  • I'm also having this issue on several of my sites (typically serving ~100 rps, now sitting solidly at 0 rps due to 100% error rate). No word back from Heroku, also seemingly tried everything. Not sure what to do while I wait. – Andrew Brown Aug 13 '21 at 17:39

3 Answers3

4

I don't know if this is an answer for other people but we discovered that our DNS settings were not updated to match the changes documented here: https://devcenter.heroku.com/changelog-items/2151

We also needed to re-establish managed SSL certificates for our custom domains.

Our setup has some complications related to proxies we have in front of Heroku though and the fact that of the three custom domains we have configured on the app, only two of them had been migrated, which complicated the debugging of this situation.

Gary Wright
  • 121
  • 2
  • 4
3

This was due to a Heroku incident. Here is the incident status: https://status.heroku.com/incidents/2311

I asked Heroku support and he rolled back the infrastructure upgrade (https://devcenter.heroku.com/changelog-items/2151) for our site. After that, we don't have H10 errors.

rmlockerd
  • 3,776
  • 2
  • 15
  • 25
Oakbow
  • 31
  • 1
2

Dunno if it helps, but I've cloned my app to a new one (letting all env vars on the old ones) and directed everything to the new Heroku app. It seems to be working just good enough while waiting for the problem to be solved by the Heroku team. No H10 errors but some H31, nothing is broken anymore though.

Raul Coan
  • 21
  • 1
  • 3