Help
cancel
Showing results for 
Search instead for 
Did you mean: 
Highlighted
Ground Controller Lvl 1
Message 1 of 1

GraphQL GitHub API: How to get more than 1000 Pull Requests?

Hi,

 

I'm unable to get more than 1000 PRs from the GraphQL API. I'm aware of the resource limitations, but 1000 nodes is far from the limit of 50000 stated in that document. I think I'm incorrectly using pagination, but can't figure out where exactly I'm wrong here.

 

Can anybody help me to understand the reason?

 

Here is a simplified version of my code:

#!/usr/bin/env python

from __future__ import print_function

import os
import requests
import sys
import time

QUERY = """
{
  search(query: "is:pr repo:kubernetes/kubernetes created:>2016-01-01", type: ISSUE, first: 100, %s) {
    issueCount
    pageInfo {
        endCursor
        hasNextPage
    }
    edges {
      cursor
      node {
        ... on PullRequest {
          number
        }
      }
    }
  }
}
"""

def main(argv):
    if len(argv) > 1:
        token = argv[1]
    else:
        token = os.getenv('GITHUB_TOKEN')
    if not token:
        print("Usage: prs <github token>")
        return 1

    after = ''
    while True:
        request = requests.post('https://api.github.com/graphql',
                                json={'query': QUERY % after},
                                headers={"Authorization": "Bearer %s" % token})
        result = request.json()

        if 'errors' in result:
            print(result)
            return 1

        search = result['data']['search']

        print("Issue count:", search['issueCount'], "number of edges:", len(search['edges']))

        print("PageInfo:", search['pageInfo'], "cursor:", search['edges'][-1]['cursor'], "\n")

        if not search['pageInfo']['hasNextPage']:
            print("done")
            break

        after = 'after: "%s"' % search['edges'][-1]['cursor']
        time.sleep(1)

if __name__ == '__main__':
    sys.exit(main(sys.argv))

and its output:

 

$ ./prs-test 
Issue count: 33754 number of edges: 100
PageInfo: {u'endCursor': u'Y3Vyc29yOjEwMA==', u'hasNextPage': True} cursor: Y3Vyc29yOjEwMA== 

Issue count: 33754 number of edges: 100
PageInfo: {u'endCursor': u'Y3Vyc29yOjIwMA==', u'hasNextPage': True} cursor: Y3Vyc29yOjIwMA== 

Issue count: 33754 number of edges: 100
PageInfo: {u'endCursor': u'Y3Vyc29yOjMwMA==', u'hasNextPage': True} cursor: Y3Vyc29yOjMwMA== 

Issue count: 33754 number of edges: 100
PageInfo: {u'endCursor': u'Y3Vyc29yOjQwMA==', u'hasNextPage': True} cursor: Y3Vyc29yOjQwMA== 

Issue count: 33754 number of edges: 100
PageInfo: {u'endCursor': u'Y3Vyc29yOjUwMA==', u'hasNextPage': True} cursor: Y3Vyc29yOjUwMA== 

Issue count: 33754 number of edges: 100
PageInfo: {u'endCursor': u'Y3Vyc29yOjYwMA==', u'hasNextPage': True} cursor: Y3Vyc29yOjYwMA== 

Issue count: 33754 number of edges: 100
PageInfo: {u'endCursor': u'Y3Vyc29yOjcwMA==', u'hasNextPage': True} cursor: Y3Vyc29yOjcwMA== 

Issue count: 33754 number of edges: 100
PageInfo: {u'endCursor': u'Y3Vyc29yOjgwMA==', u'hasNextPage': True} cursor: Y3Vyc29yOjgwMA== 

Issue count: 33754 number of edges: 100
PageInfo: {u'endCursor': u'Y3Vyc29yOjkwMA==', u'hasNextPage': True} cursor: Y3Vyc29yOjkwMA== 

Issue count: 33754 number of edges: 100
PageInfo: {u'endCursor': u'Y3Vyc29yOjEwMDA=', u'hasNextPage': False} cursor: Y3Vyc29yOjEwMDA= 

done

Thank you,

Ed