Webhook failures with Service Timeout error

Hello,

Recently we migrated GitHub to a new data center(cloud) and seeing 30%-40% increase in the webhook delivery failures. The error message for failed delivery is “We couldn’t deliver this payload: Service Timeout”
“hook_status”: 504,
“hook_response”: “Service Timeout”,

Trying “Redeliver” on the failed delivery fails first time (after about 10 seconds).
Trying “Redelivery” a second time worked (in less than one second),
Sometime it works in the 1st attempt as well.

There are multiple service host destinations and the same time the same delivery can be successful for one service and failed for another.
Sometime PR reviews/comments are generating duplicate events: when there is a failure most of the time this is the first one which fail.
Some seeing the webhook payloads reaching after 20 minutes to their hosts?
Network team is not seeing any issues on their side and curl/ping always shows connection was successful.
Anyone seen this behavior in enterprise version?

Thanks

:wave: @skumardv: Thanks for raising this report here. Since you originally wrote in, are you still experiencing this?

I’d love to get a better understanding of what you mean by “enterprise version”––would that be the type of account or platform? As far as accounts go, we have an Enterprise plan (among several others); the GitHub.com and GitHub Enterprise platforms have their own respective capabilities. More clarity would help us here :smiley_cat:

  • If this is for an account on GitHub.com, could you clarify which one (personal user or organization) and its name?
  • If this is for a GitHub Enterprise instance, we encourage you to head over to the GitHub Enterprise Support portal, signing in and submitting a new ticket. We have a dedicated team of Support Engineers that can help with any of your questions about GitHub Enterprise.

After the upgrade we are not seeing the issue with payload delivery. We are still monitoring the appliance.

1 Like