Production Issue: GraphQL API breaks while fetching list of organizations of an authenticated user

IMPORTANT !!!

Since last few days, we are struggling to understand why GraphQL API is failing while fetching list of all organizations of an authenticated user.

We are using access token to fetch authenticated user organizations

Can anybody help us resolve or understand this issue, please?

Thanks in advance!

Hi @ghowner1 could you provide some. more details of your graphql API query you acraylly executed and what failing means in your scenario such as error codea/response etc

Hey @byrneh, below is the query to fetch list of organizations of authenticated user

GraphQL query:
query($limit:Int!, $offset:String = null) {
viewer {
id,
databaseId
name,
url,
login
repositories {
totalCount
}
organizations(first: $limit, after:$offset) {
totalCount
}
}
}

Output:
{“errors”:[{“message”:“Something went wrong while executing your query. Please include 8160:3D62:AD5F7A:C2DFB9:616EC475 when reporting this issue.”}]}

Any help our clue would be real help

Thanks in advance!

This is failing when we use below graphql query. This query was working since last few months.

query($limit:Int!, $offset:String = null) {
                            viewer {
                                id,
                                databaseId
                                name,
                                url,
                                login
                                repositories {
                                    totalCount
                                }
                                organizations(first: $limit, after:$offset) {
                                    totalCount
                                }
                            }
                        }

Above graphql works if I remove orgnizations from query.

What is reason of sudden failing?

Hey @bipinacellere :wave:

I’m wondering if this is similar to the experience in this thread:

I’m going to check the request_id provided in that thread, but would appreciate similar here. Do you have a request_id value you can provide from a failed GQL call?

Edit: oh gosh, I’m silly. Your request_id was in the title. Will review and get back.

Went ahead and merged these threads as they do seem related.

My initial research reveals some internal work to mitigate errors that seem to match. These updates happened within the past 24 hours, and I’m now wondering if these are resolved.

@ghowner1 and @bipinacellere – could each of you let me know if you’re continuing to see these now?

If so, could you provide updated request_id values, please?

Hey @nethgato

We have tried to run same graphQL query (mentioned above) and ended with same error as below:

{
errors: [
{
message: ‘Something went wrong while executing your query. Please include 90CC:063A:1D34FA:203293:61727151 when reporting this issue.’
}
]
}

Is this issue on GitHub end? or something wrong in our graphQL query?

Any clue would be a great help - thanks in advance!

Thanks @ghowner1 !

No, there’s nothing wrong with your query from what I can tell, and I can identify an error on our side.

Yesterday, I had assumed that this was directly due to a recently discovered issue with our Database, but those particular errors have fully resolved, and this is unique.

Beyond that initial assumption, I don’t have more details at the moment, but I’m going to raise this with the team. Will update this thread soon.

Thanks, again!

1 Like

Just replying again to mention a couple things and get clarification…

So far, we’ve run that query ourselves internally, without error. And we can also see that the error count against that query is very low at less than ~20 instances in the past 3 days.

Is this query being run manually? Or is it part of a GitHub App or OAuth app?

Does the query fail for every user, or only specific user(s)?

It seems far more isolated than we originally thought, and while there’s more for us to do on our side, it would be great to get some more clarification on your experience.

@nethgato I’m pretty sure this is the same issue as this one: Possible regression: query which used to work all the time now returns an error I had warned about earlier last week with no response.

From what I gathered this happens with ghu_* tokens as explained in that thread. I can also pinpoint that this started happening somewhere around Sunday 17th of October and Monday 18th of October.

Thank you for clarifying.
Looks like API is working now.

Hey @nethgato

Thanks for prompt response. Issue with GraphQL query seems resolved. Now, we can fetch results as usual.

We are using GitHub App for authentication and running this query as HTTP request programmatically.

Not quite sure, but we think, this query was not failing for every user.

Hope this helps you to further investigate at your end.

Thanks!

Hey there @sashaafm :wave:

Firstly, apologies for not seeing your thread. It can be difficult to identify every new posting that requires GH Staff attention and I’ll try to take a look at that request_id you provided. However, I don’t believe that the experience that @bipinacellere and @ghowner1 have expressed are the same.


Which speaking of, I appreciate the updates and letting me know it seems mostly better? The timeline of the errors does seem to fall in line with work that we had done to resolve a new error that ran for ~2 days or so.

So by the time you both wrote in, the work had already been done and a fix was pushed.


From here I’d say, please update me with an @ mention and a new request_id value to review @ghowner1 and @bipinacellere .

Otherwise, we should be good to go!

And again, I’ll check on: Possible regression: query which used to work all the time now returns an error – which sounds like an issue with our new token formats?

At any rate, thanks all!

Hi @nethgato

Today again GraphQL is not working (check trailing message for query). Looks like still the issue is not resolved.

Can you update on this, please?

Thanks

Hey @ghowner1 :wave:

Could you please provide an updated request_id value from a more recent failure?

Similar questions for you @sashaafm @bipinacellere : is this ongoing? And if so, can you provide an updated request_id or two since you initially wrote in?

Hey @nethgato

This is error request id we got recently when we ran GraphQL query

{
errors: [
{
message: ‘Something went wrong while executing your query. Please include 1E51:099C:1D8F4A:21479C:6178EDC1 when reporting this issue.’
}
]
}

FYI - By the way, @bipinacellere and me both are colleagues and working on same issue :slight_smile:

Thanks!

1 Like

Thanks for the updated request_id value @ghowner1 :bow:

I’ve raised this with our team, but it might take a little while as we’re pretty swamped with Universe right now. So we might not be able to verify if there’s an internal issue that we can solve until next week.

However, some more detail would be useful with regards to:

  • Frequency of failures
  • Patterns of failures relating to specific users

Even just those two points would help better understand the scope, as running the same query internally, hasn’t produced the same error.

Thanks again!

Hey ya’ll :wave:

So we have now identified a SQL problem that we’re working to address, which was elevated from both this thread, and another users submitting a ticket with us.

I’m tagged on those internal issues and when we have anything more substantial to share, I’ll keep you updated!

Hey there @sashaafm @ghowner1 :wave:

Our dev team was able to identify the issue, and push a solution, and we’re no longer seeing these errors.

Could anyone who has experienced these failures try again, and let us know if it’s ongoing?

TYSM! :bow: