Getting a column out of Projects (beta)

I’m very new to GraphQL, so this might just be a learning the ropes thing, but so far the various tools, docs, samples and examples haven’t been enough to get me where I need to be.

For old style projects we have a set of scripts that dump out card details (id, description, story points from labels) to .csv files that can be imported into Planning Poker. It’s all pretty straightforward: find the project id, find the column id, iterate across the cards (issues) in the column to get what’s needed from them.

For beta projects the first hurdle seems to be that columns aren’t really a thing. It’s just a view onto the underlying data. Status (and the Group by Status view) is perhaps equivalent though. And I can see stuff about Status using a query like:

gh api graphql --header 'GraphQL-Features: projects_next_graphql' -f query='
  query{
    node(id: "MDEyOlByb2plY3ROZXh0MzQ1Ng==") {
      ... on ProjectNext {
        fields(first: 20) {
          nodes {
            id
            name
            settings
          }
        }
      }
    }
  }'

Which gets me a response that includes:

{
  "data": {
    "node": {
      "fields": {
        "nodes": [
          ...
          {
            "id": "MTIzOlByb2plY3ROZXh0RmllbGQ0NTY3OA==",
            "name": "Status",
            "settings": "{\"width\":265,\"options\":[{\"id\":\"f75ad846\",\"name\":\"To Do - Yet to Start\",\"name_html\":\"To Do - Yet to Start\"},{\"id\":\"47fc9ee4\",\"name\":\"PR19.5 14 SP's | 15 Velocity (1 over)\",\"name_html\":\"PR19.5 14 SP's | 15 Velocity (1 over)\"},{\"id\":\"98236657\",\"name\":\"Done\",\"name_html\":\"Done\"}]}"
          },

and if I run that settings response through jq I see:

{
  "width": 265,
  "options": [
    {
      "id": "f75ad846",
      "name": "To Do - Yet to Start",
      "name_html": "To Do - Yet to Start"
    },
    {
      "id": "47fc9ee4",
      "name": "PR19.5 14 SP's | 15 Velocity (1 over)",
      "name_html": "PR19.5 14 SP's | 15 Velocity (1 over)"
    },
    {
      "id": "98236657",
      "name": "Done",
      "name_html": "Done"
    }
  ]
}

But what to do with that? How do I then construct a query that gets a bunch of card details out of settings:MTIzOlByb2plY3ROZXh0RmllbGQ0NTY3OA==:47fc9ee4 ?

Of course it doesn’t help that the GitHub GraphQL Explorer doesn’t support the ‘GraphQL-Features: projects_next_graphql’

How about a few more examples please that go beyond trivial first:20 type summaries, and show how to combine and filter across the object relationships?

PS project IDs above have been diddled with (base64 decoded, modified and then re-encoded).

You’re already using the docs from Using the API to manage projects (beta) - GitHub Docs, presumably.

If you’re after details of Issues, looks to me like those are of type ProjectNextItem so I reckon you want to grab ‘items’ rather than ‘fields’ in your initial query with the node ID. The doc entry for ProjectNextItem in the Reference section of that URL tells you where you can ‘walk’ to next in the query.

Thanks @davidMbrooke and yes, already into the Using the API to manage projects (beta) - GitHub Docs docs - they just don’t have suitable examples of what I’m trying to do (and there’s not much more in the Form calls with GraphQL docs where there’s a solitary example that just uses first: and last:).

The problem for me isn’t getting stuff out of issues, but rather filtering on just the things that have a given status.

So I can use a query like this:

gh api graphql --header 'GraphQL-Features: projects_next_graphql' -f query='
  query{
    node(id: "MDEyOlByb2plY3ROZXh0MzQ1Ng==") {
      ... on ProjectNext {
        items(first: 100) {
          nodes{
            title
            fieldValues(first: 8) {
                  nodes{
                    value
                  }
            }
            content{
              ...on Issue {
                number
                labels(first: 50) {
                  nodes{
                    name
                  }
                }
              }
            }
          }
        }
      }
    }
  }'

and it returns a lump of JSON with everything I need - issue IDs, titles, labels. But… there’s a lot of cruft in there. I should be able to do a query that just gets me the issues associated with "PR19.5 14 SP's | 15 Velocity (1 over)" and the Status setting id that corresponds to that of “47fc9ee4”. That’s where I’m presently adrift, as I’ve not seen any examples of how to filter such a query. (And for sure I could just grab that big lump of JSON and carve it down to size in my Python script, which is pretty much what ends up happening with the old REST API, but the promise of GraphQL is to work smarter than that).

You’re right that the promise of GraphQL is to let you filter as you ‘walk’ around the graph data structure. The intent is to do the filtering server-side, to both reduce the amount of data that comes back and to reduce the load on the client.

One good thing about GraphQL is that it’s far from being GitHub-specific, so docs for other GraphQL implementations or more generic docs can often be useful (which I suppose might be an excuse for a light-touch in the GitHub docs): take a look at Understanding GraphQL Filters | Tabnine Blog for example

There’s also the totally generic GraphQL docs at Introduction to GraphQL | GraphQL

Thanks again @davidMbrooke

From Understanding GraphQL Filters | Tabnine Blog it seems that I should be able to do something like:

value(filter: {
value: "47fc9ee4"
})

As part of a full query like this:

gh api graphql --header 'GraphQL-Features: projects_next_graphql' -f query='
  query{
    node(id: "MDEyOlByb2plY3ROZXh0MzQ1Ng==") {
      ... on ProjectNext {
        items(first: 20) {
          nodes{
            title
            id
            fieldValues(first: 8) {
              nodes{
                value(filter: {
                value: "47fc9ee4"
                })
                projectField{
                  name
                }
              }
            }
            content{
              ...on Issue {
                labels(first: 10) {
                  nodes{
                    name
                  }
                }
              }
            }
          }
        }
      }
    }
  }'

But that just gives me gh: Field 'value' doesn't accept argument 'filter', or in it’s full JSON majesty:

{
  "errors": [
    {
      "path": [
        "query",
        "node",
        "... on ProjectNext",
        "items",
        "nodes",
        "fieldValues",
        "nodes",
        "value",
        "filter"
      ],
      "extensions": {
        "code": "argumentNotAccepted",
        "name": "value",
        "typeName": "Field",
        "argumentName": "filter"
      },
      "locations": [
        {
          "line": 11,
          "column": 23
        }
      ],
      "message": "Field 'value' doesn't accept argument 'filter'"
    }
  ]
}

So I’m starting to think that the reason for no filter examples might be that filtering hasn’t been implemented (yet).

I guess for now I’ll be doing my filtering in Python.

I got the filtering working in Python, with some new scripts added to my dump_cards repo.

Most of the action is in the atdumpmemex.py module:

#!/usr/bin/env python3
import base64, json, os, requests

# Color constants
# Reference: https://gist.github.com/chrisopedia/8754917
COLERR="\033[0;31m"
COLINFO="\033[0;35m"
COLRESET="\033[m"

graphqlurl = 'https://api.github.com/graphql'
token = os.environ['GITHUB_API_TOKEN']
headers = {"Content-Type": "application/json", 
    "Accept": "application/json",
    "Authorization": "Bearer " + token,
    "GraphQL-Features": "projects_next_graphql" }



def list_memex_projects(org):
    query = ('query{ organization(login: \\"' + org + '\\") '
        '{ projectsNext(first: 20) { nodes { id title } } } }')
    response = requests.post(graphqlurl, 
        headers=headers, 
        data='{"query": '+'\"' + query + '\"}')
    if response.status_code != 200:
        # An error occured
        print(COLERR + "Error getting project list : "
            + str(response.status_code) + " " + response.text + COLRESET)

    json_projects = json.loads(response.text)
    for node in json_projects["data"]["organization"]["projectsNext"]["nodes"]:
        project_id = base64.b64decode(node["id"]).decode("utf-8")
        print(f'{project_id}  {node["title"]}')


def list_memex_columns(project_id):
    b64id = base64.b64encode(project_id.encode("ascii")).decode("utf-8")
    query = ('query{ node(id: \\"' + b64id + '\\")  '
        '{ ... on ProjectNext { fields(first: 20) '
        '{ nodes { id name settings } } } } }')
    response = requests.post(graphqlurl, 
        headers=headers, 
        data='{"query": '+'\"' + query + '\"}')
    if response.status_code != 200:
        # An error occured
        print(COLERR + "Error getting project columns : "
            + str(response.status_code) + " " + response.text + COLRESET)
    
    json_nodes = json.loads(response.text)
    for node in json_nodes["data"]["node"]["fields"]["nodes"]:
        if node["name"] == "Status":
            json_status = json.loads(node["settings"])
            for options in json_status["options"]:
                print(f'{options["id"]} {options["name"]}')
            


def list_memex_cards(column_id, project_id):
    cards_file = column_id + ".csv"
    b64id = base64.b64encode(project_id.encode("ascii")).decode("utf-8")
    query = ('query{ node(id: \\"' + b64id + '\\") '
        '{ ... on ProjectNext { items(first: 100) '
        '{ nodes{ title fieldValues(first: 8) { nodes{ value } } '
        'content{ ...on Issue { number labels(first: 50) '
        '{ nodes{ name } } } } } } } } }')
    response = requests.post(graphqlurl, 
        headers=headers, 
        data='{"query": '+'\"' + query + '\"}')
    if response.status_code != 200:
        # An error occured
        print(COLERR + "Error getting project column cards : "
            + str(response.status_code) + " " + response.text + COLRESET)

    json_cards = json.loads(response.text)
    f = open(cards_file, 'w')
    f.write("Issue Key,Summary,Description,Acceptance Criteria,Story Points\n")
    for card in json_cards["data"]["node"]["items"]["nodes"]:
        for status in card["fieldValues"]["nodes"]:
            if status["value"] == column_id:
                f.write(f'{card["content"]["number"]},{card["title"]},,,')
                for label in card["content"]["labels"]["nodes"]:
                        if (label["name"][-2:]=="SP"):
                            f.write (f'{label["name"].partition(" ")[0]}')
                            # break loop in case there are multiple SP labels
                            break
                f.write ('\n')
    f.close

This doesn’t feel like the right GraphQL way, but it works.