Help
cancel
Showing results for 
Search instead for 
Did you mean: 
Copilot Lvl 3
Message 1 of 12

[Bug] v4 (GraphQL) API: Trouble retrieving pull request review comments

Solved! Go to Solution.

Howdy! I'm running into a couple of issues using your (wonderful) v4 API. I found a couple of possibly-related threads on your read-only platform forum, but I wanted to mention them here in hopes that there is an available solution.

 

The crux of what I'm trying to do is I'm trying to retrieve all `PullRequestReviewComment` objects associated with a `PullRequestReview`. I've tried to achieve this in three separate ways:

 

  • By querying `PullRequestReviewComment` objects returned as part of the pull request's timeline:
    query($owner: String!, $name: String!, $number: Int!) {
      repository(owner: $owner, name: $name) {
        pullRequest: issueOrPullRequest(number: $number) {
          ... on PullRequest {
            timeline(last: 100) {
              ... on PullRequestReviewComment {
                id
                createdAt
              }
            }
          }
        }
      }
    }
    Surprisingly, this always returns zero `PullRequestReviewComment` objects.
  • By querying `PullRequestReviewComment` objects returned as part of each retrieved pull request review in the pull request's timeline:
    query($owner: String!, $name: String!, $number: Int!) {
      repository(owner: $owner, name: $name) {
        pullRequest: issueOrPullRequest(number: $number) {
          ... on PullRequest {
            timeline(last: 100) {
              ... on PullRequestReview {
                comments(first: 100) {
                  nodes {
                    id
                    createdAt
                  }
                }
              }
            }
          }
        }
      }
    }
    Even more strangely, this returns some of the pull request comments. The response seems to be omitting all but the first comment associated with each commented-on line.
  • By querying `PullRequestReviewComment` objects returned as part of each retrieved pull request thread in the pull request's timeline:
    query($owner: String!, $name: String!, $number: Int!) {
      repository(owner: $owner, name: $name) {
        pullRequest: issueOrPullRequest(number: $number) {
          ... on PullRequest {
            timeline(last: 100) {
              ... on PullRequestReviewThread {
                comments(first: 100) {
                  nodes {
                    id
                    createdAt
                  }
                }
              }
            }
          }
        }
      }
    }
    This solution made a bit of sense given the second solution's incompleteness: comments are likely threaded, given the `replyTo` field in the `PullRequestReviewComment` definition. Alas, this always returned zero threads and thus zero comments.

 

I've considered one possible workaround, which is to use the v3 API (as it apparently returns all review comments related to a pull request), but we are consuming enough data per token from your API that this wouldn't perform at scale. Hoping to find a solution that allows us to continue to query the v4 API for this data.

11 Replies
GitHub Staff
Message 2 of 12

Re: [Bug] v4 (GraphQL) API: Trouble retrieving pull request review comments

Hey @nporteschaikin!

 

I am glad you are finding the v4 API wonderful - I think you've found a less-than-wonderful gap in the information that's available. Looking at it today, I think the way to get this information would be to do a lot of work using the `reviews` connection with a query like this:

 

query ($owner: String!, $name: String!, $number: Int!) {
  repository(owner: $owner, name: $name) {
    pullRequest: issueOrPullRequest(number: $number) {
      ... on PullRequest {
        reviews(last: 100) {
          totalCount
          nodes {
            id
            author {
              login
            }
            url
            comments(first: 100) {
              nodes {
                id
                replyTo {
                  id
                }
                body
                author {
                  login
                }
              }
            }
          }
        }
      }
    }
  }
} 

 

If you look at the id of the `eplyTo field on the comments, you should be able to take a look at what comments and reviews are "top level", and what comments are attributed to them.

 

Unfortunately, this is not a great UX! As you've discovered in those earlier issues, this has been an issue for quite some time (sorry @rnystrom and @yakov116@hsborges and everyone else who we let down this time and others).

 

I've got some GraphQL Schema in development that would allow someone to query something like this:

 

query ($owner: String!, $name: String!, $number: Int!) {
  repository(owner: $owner, name: $name) {
    pullRequest: issueOrPullRequest(number: $number) {
      __typename
      ... on PullRequest {
        reviewThreads(first: 100) {
          edges {
            node {
              __typename
              comments(first: 10) {
                nodes {
                  pullRequestReview {
                    id
                  }
                  body
                }
              }
            }
          }
        }
      }
    }
  }
}

 

Which, for the infamous issue https://github.com/GitHawkApp/GitHawk/pull/555 as mentioned in other threads, would return something like this:

 

{
  "data": {
    "repository": {
      "pullRequest": {
        "__typename": "PullRequest",
        "reviewThreads": {
          "edges": [
            {
              "node": {
                "__typename": "PullRequestReviewThread",
                "comments": {
                  "nodes": [
                    {
                      "pullRequestReview": {
                        "id": "MDE3OlB1bGxSZXF1ZXN0UmV2aWV3NjkyNTI5NzE="
                      },
                      "body": "Can we remove the double space, and if it fails it should default to just returning `fileName`"
                    },
                    {
                      "pullRequestReview": {
                        "id": "MDE3OlB1bGxSZXF1ZXN0UmV2aWV3NjkyNTkxMTk="
                      },
                      "body": "I have tidied up the code and removed the `potentialLanguage`. I was planning to use that for the 'you know what' PR as Highlight.js is not correctly automatically detected the Swift language 😅\r\n\r\nAt GitHub they are using their own Ruby library [github/linguist](https://github.com/github/linguist) can't ask the Github API to send the language?"
                    },
                    {
                      "pullRequestReview": {
                        "id": "MDE3OlB1bGxSZXF1ZXN0UmV2aWV3NjkzOTkwNTE="
                      },
                      "body": "Leaving this reply as a test for PR review threads, demonstrating how broken they are 😞 "
                    }
                  ]
                }
              }
            },
            {
              "node": {
                "__typename": "PullRequestReviewThread",
                "comments": {
                  "nodes": [
                    {
                      "pullRequestReview": {
                        "id": "MDE3OlB1bGxSZXF1ZXN0UmV2aWV3NjkyNTMwMTc="
                      },
                      "body": "Probably want to extract a list of known languages to an enum + separate file? And maybe we can already find a list like that somewhere?\r\n\r\nThen we can `return Language(rawValue: fileExtension)` (or something similar, eg. when there are multiple extensions for one language) instead of this weird `switch`. :D"
                    }
                  ]
                }
              }
            },
            {
              "node": {
                "__typename": "PullRequestReviewThread",
                "comments": {
                  "nodes": [
                    {
                      "pullRequestReview": {
                        "id": "MDE3OlB1bGxSZXF1ZXN0UmV2aWV3NjkyNTI5NzE="
                      },
                      "body": "Not sure this is necessary, at least yet? ~Also it shouldn't **only** support Swift, even if we do keep it~\r\n\r\nEdit// See BasThomas comment"
                    }
                  ]
                }
              }
            }
          ]
        }
      }
    }
  }
}

 How does that sound? I'll keep this thread up to date as we look to release this.

 

Separately, to people I've mentioned in this issue (including you @nporteschaikin): I am sure you've got more schema suggestions, frustrations, or anything of the sort. If you're interested in chatting about them with me, I'd love to sit down over a video chat and go through what's got you down with our public GraphQL API.

Copilot Lvl 3
Message 3 of 12

Re: [Bug] v4 (GraphQL) API: Trouble retrieving pull request review comments

Hi @nickvanw,

 

Thanks for getting back to me and for your detailed response! We're heavy users of your GraphQL API at Code Climate and it's been mostly a pleasant experience :)

 


@nickvanw wrote:

If you look at the id of the `eplyTo field on the comments, you should be able to take a look at what comments and reviews are "top level", and what comments are attributed to them.

 

In my experience, this wouldn't solve the issue we're running into. It looks like the response we receive from GraphQL (mostly) does not include replies to comments - rather, we're receiving all top-level comments, which we can't connect to their replies. Please correct me if I'm misunderstanding something! A `replies` connection would be another (inelegant) option, but so would the `reviewThreads` connection you proposed below. 

 

For now, we're using the v3 API, although - as you could imagine - it's significantly more challenging to extract this data at scale via the traditional REST API. I look forward to seeing the `reviewThreads` connection released!

 

As for additional feedback on the GraphQL API, I'd love to hop on a call and chat! As alluded to above, we're extracting a lot of data, regularly, from the v4 API and I think we've collected a lot of wisdom, best practices, and feedback for GitHub. I'd love to shoot you an email - let me know how to reach you privately!

GitHub Staff
Message 4 of 12

Re: [Bug] v4 (GraphQL) API: Trouble retrieving pull request review comments

Hey @nporteschaikin,

 

I'm happy to hear that! Perhaps we may be speaking about different things - looking at the query I posted earlier that works now, I believe one could re-create the `reviewThreads` connection with the data that's returned using that query. By using the `id` returned in `replyTo`, you should be able to find the parent comment ID that the reply was to, and build the discussion that way. 

 

Taking the output of that query and plugging it into this hastily written python:

 

import json
import collections
# d = Raw GraphQL query output
resp = json.loads(d, strict=False)
reviews = resp['data']['repository']['pullRequest']['reviews']['nodes']
toplevel, replies = {}, collections.defaultdict(list)

for r in reviews:
    for c in r['comments']['nodes']:
        if c['replyTo']:
            replies[c['replyTo']['id']].append(c)
        else:
            toplevel[c['id']] = c
            

for c in toplevel:
    print('Thread:\n')
    print(toplevel[c]['body'] + '\n')
    for r in replies[c]:
        print('  Reply: %s\n' % r['body'])

Using the Issue I used earlier, https://github.com/GitHawkApp/GitHawk/pull/555, I get the following:

 

Thread:

Probably want to extract a list of known languages to an enum + separate file? And maybe we can already find a list like that somewhere?

Then we can `return Language(rawValue: fileExtension)` (or something similar, eg. when there are multiple extensions for one language) instead of this weird `switch`. :D

Thread:

Can we remove the double space, and if it fails it should default to just returning `fileName`

  Reply: I have tidied up the code and removed the `potentialLanguage`. I was planning to use that for the 'you know what' PR as Highlight.js is not correctly automatically detected the Swift language 😅

At GitHub they are using their own Ruby library [github/linguist](https://github.com/github/linguist) can't ask the Github API to send the language?

  Reply: Leaving this reply as a test for PR review threads, demonstrating how broken they are 😞 

Thread:

Not sure this is necessary, at least yet? ~Also it shouldn't **only** support Swift, even if we do keep it~

Edit// See BasThomas comment

As a rough cut, this seems to me to be the equivalent of the reviewThreads connection I talked about releasing earlier - three total review request threads, one of which having three comments and the other only having the top-level comment. I haven't tested it exhaustively, though I think the approach would still work.

 

If I've mischaracterized anything, please let me know! I'm working on getting the `reviewThreads` through code review now, but I want to make sure it will solve your needs before releasing it! 

 

Separately, please reach out to partnerengineering@github.com and mention this thread - we'll be expecting you, and we can set up a chat! 

Copilot Lvl 3
Message 5 of 12

Re: [Bug] v4 (GraphQL) API: Trouble retrieving pull request review comments

@nickvanw:

 

Hmm. The problem for us is that the replies are not being returned, so the `replyTo` field isn't helpful. Take this thread for example:

 

- Review comment A, by me

- Review comment B, by someone else

- Review comment C, by me

 

The only comment being returned by the GraphQL API is comment A, the "parent" comment.

GitHub Staff
Message 6 of 12

Re: [Bug] v4 (GraphQL) API: Trouble retrieving pull request review comments

@nporteschaikin

 

Do you have an example of a public PR where you're experiencing this? 

 

The top level reviews connection will only return the top level comments, but the comments connection inside of that seems to give me all of the information I expect. Specifically, I want to make sure you're trying the `reviews` query that I linked to in my response, and not any of the examples that you gave in your original post. 

 

I'd love to find an example where I can reproduce what you're talking about, but I've been going around to a few PRs and it seems to be returning what I'd expect. 

Highlighted
Copilot Lvl 3
Message 7 of 12

Re: [Bug] v4 (GraphQL) API: Trouble retrieving pull request review comments

Thanks for getting back to me so quickly, @nickvanw. I'm experiencing this exclusively with private repositories, so I'm going to dig into the payload further and see if I'm missing something. Will report back.

Solution
GitHub Staff
Message 8 of 12

Re: [Bug] v4 (GraphQL) API: Trouble retrieving pull request review comments

Hey @nporteschaikin!

 

Remember that reviewThreads connection I was talking about earlier? I have good news!

 

If you use the Pull Requests Preview, you should be able to run this query:

 

query ($owner: String!, $name: String!, $number: Int!) {
  repository(owner: $owner, name: $name) {
    pullRequest: issueOrPullRequest(number: $number) {
      __typename
      ... on PullRequest {
        reviewThreads(first: 100) {
          edges {
            node {
              comments(first: 10) {
                nodes {
                  pullRequestReview {
                    id
                  }
                  body
                }
              }
            }
          }
        }
      }
    }
  }
}

 

This was just shipped, so the documentation should be updating soon.

 

I'd love some feedback on whether this solves your issue(s)! 

Copilot Lvl 3
Message 9 of 12

Re: [Bug] v4 (GraphQL) API: Trouble retrieving pull request review comments

Hi @nickvanw! I'm going to take this for a spin. Are these objects also available via the `PullRequestTimelineConnection` connection?

GitHub Staff
Message 10 of 12

Re: [Bug] v4 (GraphQL) API: Trouble retrieving pull request review comments

At present moment, you can only fetch these threads by looking at this particular connection. I believe re-creating the timeline exactly as it looks on GitHub will take some munging and additional work -- if something looks amiss, please let me know. 

 

If any other issues persist, or something doesn't seem right, please don't hesitate to reach out!