Recently we migrated GitHub to a new data center(cloud) and seeing 30%-40% increase in the webhook delivery failures. The error message for failed delivery is “We couldn’t deliver this payload: Service Timeout”
“hook_response”: “Service Timeout”,
Trying “Redeliver” on the failed delivery fails first time (after about 10 seconds).
Trying “Redelivery” a second time worked (in less than one second),
Sometime it works in the 1st attempt as well.
There are multiple service host destinations and the same time the same delivery can be successful for one service and failed for another.
Sometime PR reviews/comments are generating duplicate events: when there is a failure most of the time this is the first one which fail.
Some seeing the webhook payloads reaching after 20 minutes to their hosts?
Network team is not seeing any issues on their side and curl/ping always shows connection was successful.
Anyone seen this behavior in enterprise version?