Deleted repo was returned in listRepo

Hi Github developers,

my team (CodeStarConnection) have been performing load testing with Github create/list/deleteRepos for integration with the company system.

The initial setup of load test is starting from empty repo
This test consists of

  1. createRepo
  2. listRepo (which should return the fullRepositoryId created in #1
  3. deleteRepo by the fullRepositoryId created in #1
  4. listRepo (which should return empty results)

I noticed that #2 and #4 failed randomly during the load tests from two accounts
The symptom is that #2 (listRepo) returned the deleted repo. also #4(listRepo) returned the deletedRepo which was deleted on #3.
I used repoId as date timestamp to associate API requests to Github events

Below is http request/response. (some of sensitive fields were taken out)

#1 createRepo
05 Oct 2020 01:53:04,720 REQUEST [2298c826-bfc5-4155-b19d-d0771eef02e7]: POST / HTTP/1.1
host: localhost:9980
x-amz-target: com.amazonaws.codestar.connections.CodeStar_connections_20191201.CreateRepository
content-type: application/x-amz-json-1.0
x-amz-date: 20201005T015301Z
content-length: 323

05 Oct 2020 01:53:04,720 REPLY [2298c826-bfc5-4155-b19d-d0771eef02e7]: HTTP/1.1 200
x-amzn-RequestId: 2298c826-bfc5-4155-b19d-d0771eef02e7
Content-Type: application/x-amz-json-1.0
Content-Length: 78


#2 listRepo
05 Oct 2020 01:53:05,707 REQUEST [084c6e5c-452b-4c2b-8497-ae190f85036f]: POST / HTTP/1.1
host: localhost:9980
x-amz-target: com.amazonaws.codestar.connections.CodeStar_connections_20191201.ListRepositories
content-type: application/x-amz-json-1.0
x-amz-date: 20201005T015305Z
content-length: 165

05 Oct 2020 01:53:05,707 REPLY [084c6e5c-452b-4c2b-8497-ae190f85036f]: HTTP/1.1 200
x-amzn-RequestId: 084c6e5c-452b-4c2b-8497-ae190f85036f
Content-Type: application/x-amz-json-1.0
Content-Length: 409

{“Repositories”:[{“Capabilities”:{“ArchiveUpload”:true,“GitClone”:true},“Description”:“kmchang-amazon-description-8-Create”,“FullRepositoryId”:“kmchang-ama zon/kmchang-amazon-repo-2020-10-05T01-52-35Z”,“IsPrivate”:false,“OwnerDisplayName”:“kmchang-amazon”,“OwnerId”:“kmchang-amazon”,“RepositoryDisplayName”:“kmc hang-amazon-repo-2020-10-05T01-52-35Z”,“RepositoryId”:“kmchang-amazon-repo-2020-10-05T01-52-35Z”}]}

As log showed, the kmchang-amazon-repo-2020-10-05T01-52-35Z (which was deleted successfully) was returned when the listRepo of kmchang-amazon-repo-2020-10-05T01-53-01Z was called.

this symptom seems to be similar to Deleted repository shows up in list org repos API

I am wondering if GitHub returns some stale data from caches.
how long would it take for GitHub backend to sync all the data across the system?

Please let me know if you need any information from me.