NuGet Package Registry: publish operations take forever

Pushing PackageName.1.0.6.nupkg to 'https://nuget.pkg.github.com/lol768'...
  PUT https://nuget.pkg.github.com/lol768/
  OK https://nuget.pkg.github.com/lol768/ 1612ms
Your package was pushed.

A few minutes later:

/home/adam/RiderProjects/ProjectName/ProjectName/ProjectName.csproj : error NU1102: Unable to find package PackageName with version (>= 1.0.6) [/home/adam/RiderProjects/ProjectName/ProjectName.sln]
/home/adam/RiderProjects/ProjectName/ProjectName/ProjectName.csproj : error NU1102:   - Found 6 version(s) in github [ Nearest version: 1.0.5 ] [/home/adam/RiderProjects/ProjectName/ProjectName.sln]
/home/adam/RiderProjects/ProjectName/ProjectName/ProjectName.csproj : error NU1102:   - Found 0 version(s) in nuget.org [/home/adam/RiderProjects/ProjectName/ProjectName.sln]

I find myself regularly waiting 5-10 minutes for the dotnet restore to start working. Clearly the HTTP 200 response to the PUT which says Your package was pushed. is a lie because if it had been pushed the restore would work.

Why does it take so long for clients to be able to fetch packages after publishing them? I’d be fine with a few seconds but this is really problematic.