API Search Repositories with no duplicate repos. it's possible?

Hi,
I doing a search on github api repositories with my python program.
the result from this search is limited to 1000. this is correct.
but in this result of 1000 row. the same repositories (same URL) is there.
it’s possible to have a search with 1000 repos with no duplicate?

my query : Query = language:python
here a sample.
"…
“https:///github.com/pennersr/django-allauth.git”
“https:///github.com/jazzband/django-debug-toolbar.git”
“https:///github.com/jazzband/django-debug-toolbar.git <- duplicate”
“https:///github.com/jazzband/django-debug-toolbar.git <- duplicate”
“https:///github.com/django-extensions/django-extensions.git”
“https:///github.com/django-oscar/django-oscar.git”
“https:///github.com/jupyter/docker-stacks.git”
“https:///github.com/jupyter/docker-stacks.git <- duplicate”
“https:///github.com/docopt/docopt.git”
“https:///github.com/shidenggui/easytrader.git”
“https:///github.com/edgedb/edgedb.git”
“https:///github.com/edgedb/edgedb.git <- duplicate”

Hello @StephLang99 and welcome to the community.

Can you give the exact URL you’re making the request to in order to receive these results? That will help us debug what’s going on here better.

Let us know!

Hi thanks to help me.
here https://api.github.com/search/repositories?q=language:python&sort=stars&order=desc&page=10&per_page=100

Thanks for sharing the URL. I tried the following and, at least in the first 100 results, there are no duplicate repositories:

$ http -pb "https://api.github.com/search/repositories?q=language:python&sort=stars&order=desc&per_page=100" | jq .items[].html_url | wc -l
     100
$ http -pb "https://api.github.com/search/repositories?q=language:python&sort=stars&order=desc&per_page=100" | jq .items[].html_url | uniq | wc -l
     100

Are you certain you’re collating or parsing the results correctly? It might also be that, depending on how you paginate, that the sorted results could shift between requests and that might be the cause of the duplicates.

yes, you’re right.
the issue is with the library pygithub when is handling the paginate.
I displayed the stars and the counter is not descending.

1 Like