Self-hosted runner jobs Abandoned after 15m

We have a group of self-hosted runners configured to run 2 sets of jobs that complete within 15-25 minutes. They have been executing those jobs and reporting status correctly for months.
The problem seems to show at around 2021-12-09 18:54Z

All jobs sent to any of the runners after this time are Abandoned after 15 minutes.

Below are the runner listener logs of involving two jobs right around the time I mentioned above.
The earlier successful job took 17m to complete successfully. Right after the second job timeout out after 15m. Same for all subsequent jobs after.

Successful job:

[2021-12-09 18:34:56Z INFO JobNotification] Entering JobStarted Notification
[2021-12-09 18:34:56Z INFO JobNotification] Entering StartMonitor
[2021-12-09 18:35:55Z INFO JobDispatcher] Successfully renew job request 5208, job is valid till 12/09/2021 18:45:55
[2021-12-09 18:36:56Z INFO JobDispatcher] Successfully renew job request 5208, job is valid till 12/09/2021 18:46:56
[2021-12-09 18:37:56Z INFO JobDispatcher] Successfully renew job request 5208, job is valid till 12/09/2021 18:47:56
[2021-12-09 18:38:56Z INFO JobDispatcher] Successfully renew job request 5208, job is valid till 12/09/2021 18:48:56
[2021-12-09 18:39:56Z INFO JobDispatcher] Successfully renew job request 5208, job is valid till 12/09/2021 18:49:56
[2021-12-09 18:40:56Z INFO JobDispatcher] Successfully renew job request 5208, job is valid till 12/09/2021 18:50:56
[2021-12-09 18:41:56Z INFO JobDispatcher] Successfully renew job request 5208, job is valid till 12/09/2021 18:51:56
[2021-12-09 18:42:56Z INFO JobDispatcher] Successfully renew job request 5208, job is valid till 12/09/2021 18:52:56
[2021-12-09 18:43:56Z INFO JobDispatcher] Successfully renew job request 5208, job is valid till 12/09/2021 18:53:56
[2021-12-09 18:44:56Z INFO JobDispatcher] Successfully renew job request 5208, job is valid till 12/09/2021 18:54:56
[2021-12-09 18:45:56Z INFO JobDispatcher] Successfully renew job request 5208, job is valid till 12/09/2021 18:55:56
[2021-12-09 18:46:56Z INFO JobDispatcher] Successfully renew job request 5208, job is valid till 12/09/2021 18:56:56
[2021-12-09 18:47:57Z INFO JobDispatcher] Successfully renew job request 5208, job is valid till 12/09/2021 18:57:57
[2021-12-09 18:48:57Z INFO JobDispatcher] Successfully renew job request 5208, job is valid till 12/09/2021 18:58:57
[2021-12-09 18:49:57Z INFO JobDispatcher] Successfully renew job request 5208, job is valid till 12/09/2021 18:59:57
[2021-12-09 18:50:57Z INFO JobDispatcher] Successfully renew job request 5208, job is valid till 12/09/2021 19:00:57
[2021-12-09 18:51:51Z INFO ProcessInvokerWrapper] STDOUT/STDERR stream read finished.
[2021-12-09 18:51:51Z INFO ProcessInvokerWrapper] STDOUT/STDERR stream read finished.
[2021-12-09 18:51:51Z INFO ProcessInvokerWrapper] Finished process 1336523 with exit code 100, and elapsed time 00:16:56.0654095.
[2021-12-09 18:51:51Z INFO JobDispatcher] Worker finished for job ca395085-040a-526b-2ce8-bdc85f692774. Code: 100
[2021-12-09 18:51:51Z INFO JobDispatcher] finish job request for job ca395085-040a-526b-2ce8-bdc85f692774 with result: Succeeded
[2021-12-09 18:51:51Z INFO Terminal] WRITE LINE: 2021-12-09 18:51:51Z: Job yyy Tests completed with result: Succeeded
[2021-12-09 18:51:51Z INFO JobDispatcher] Stop renew job request for job ca395085-040a-526b-2ce8-bdc85f692774.
[2021-12-09 18:51:51Z INFO JobDispatcher] job renew has been canceled, stop renew job request 5208.
[2021-12-09 18:51:51Z INFO JobNotification] Entering JobCompleted Notification
[2021-12-09 18:51:51Z INFO JobNotification] Entering EndMonitor

Timedout Job (Abandoned by Github):

[2021-12-09 18:54:27Z INFO JobNotification] Entering JobStarted Notification
[2021-12-09 18:54:27Z INFO JobNotification] Entering StartMonitor
[2021-12-09 18:55:27Z INFO JobDispatcher] Successfully renew job request 5210, job is valid till 12/09/2021 19:05:26
[2021-12-09 18:56:27Z INFO JobDispatcher] Successfully renew job request 5210, job is valid till 12/09/2021 19:06:27
[2021-12-09 18:57:27Z INFO JobDispatcher] Successfully renew job request 5210, job is valid till 12/09/2021 19:07:27
[2021-12-09 18:58:27Z INFO JobDispatcher] Successfully renew job request 5210, job is valid till 12/09/2021 19:08:27
[2021-12-09 18:59:27Z INFO JobDispatcher] Successfully renew job request 5210, job is valid till 12/09/2021 19:09:27
[2021-12-09 19:00:27Z INFO JobDispatcher] Successfully renew job request 5210, job is valid till 12/09/2021 19:10:27
[2021-12-09 19:01:27Z INFO JobDispatcher] Successfully renew job request 5210, job is valid till 12/09/2021 19:11:27
[2021-12-09 19:02:27Z INFO JobDispatcher] Successfully renew job request 5210, job is valid till 12/09/2021 19:12:27
[2021-12-09 19:03:27Z INFO JobDispatcher] Successfully renew job request 5210, job is valid till 12/09/2021 19:13:27
[2021-12-09 19:04:27Z INFO JobDispatcher] Successfully renew job request 5210, job is valid till 12/09/2021 19:14:27
[2021-12-09 19:05:27Z INFO JobDispatcher] Successfully renew job request 5210, job is valid till 12/09/2021 19:15:27
[2021-12-09 19:06:27Z INFO JobDispatcher] Successfully renew job request 5210, job is valid till 12/09/2021 19:16:27
[2021-12-09 19:07:28Z INFO JobDispatcher] Successfully renew job request 5210, job is valid till 12/09/2021 19:17:28
[2021-12-09 19:08:28Z INFO JobDispatcher] Successfully renew job request 5210, job is valid till 12/09/2021 19:18:28
[2021-12-09 19:09:28Z ERR GitHubActionsService] PATCH request to https://pipelines.actions.githubusercontent.com/xxxxxxxxGSDNFnM6IXCK6v2YLUss643Um/_apis/distributedtask/pools/1/jobrequests/5210?lockToken=00000000-0000-0000-0000-000000000000 failed. HTTP Status: BadRequest, AFD Ref: Ref A: 8C68A2B7AEDB42F78B3C4D90C683CA0F Ref B: CO1EDGE2014 Ref C: 2021-12-09T19:09:28Z
[2021-12-09 19:09:28Z INFO JobDispatcher] TaskAgentJobTokenExpiredException received renew job request 5210, job is no longer valid, stop renew job request.
[2021-12-09 19:09:28Z INFO JobDispatcher] Send job cancellation message to worker for job ea6607b0-6be3-5f64-fab9-b146379c63b3.
[2021-12-09 19:09:28Z INFO ProcessChannel] Sending message of length 0, with hash ‘e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855’
[2021-12-09 19:09:43Z INFO ProcessInvokerWrapper] STDOUT/STDERR stream read finished.
[2021-12-09 19:09:43Z INFO ProcessInvokerWrapper] STDOUT/STDERR stream read finished.
[2021-12-09 19:09:43Z INFO ProcessInvokerWrapper] Finished process 1389420 with exit code 102, and elapsed time 00:15:16.6483272.
[2021-12-09 19:09:43Z INFO JobDispatcher] finish job request for job ea6607b0-6be3-5f64-fab9-b146379c63b3 with result: Abandoned
[2021-12-09 19:09:43Z INFO Terminal] WRITE LINE: 2021-12-09 19:09:43Z: Job xxx Integration Tests completed with result: Abandoned
[2021-12-09 19:09:43Z INFO JobDispatcher] Stop renew job request for job ea6607b0-6be3-5f64-fab9-b146379c63b3.
[2021-12-09 19:09:43Z INFO JobNotification] Entering JobCompleted Notification
[2021-12-09 19:09:43Z INFO JobNotification] Entering EndMonitor

The PATCH request failure looks suspect but I do not know what to make of it.
Please advise.

Cheers,
Vui