Searching Github Gists via API

Hey folks,

Is it possible to use the Github API to perform searches on Github Gists? One thing in particularly I’d like to be able to do is to query for all of the most recent gists of a specific language. Here’s an example of performing this exact type of query from the browser: https://gist.github.com/search?q=language%3Apython

Unfortunately, this doesn’t seem possible according to the docs. Specifically, the gists/public endpoint doesn’t have the query param q for search field that the console does: https://docs.github.com/en/rest/reference/gists#list-public-gists

Is this accurate, or am I just looking in the wrong place?

Thanks!
Sean

:wave: hello there @Seancarpenter welcome to the GitHub Support Community!

The REST API’s Search API doesn’t currently an endpoint for searching gists, nor is a gist one of our supported SearchType values for the GraphQL API.

This is accurate; there’s no q parameter (or similar) for that particular endpoint.

Thanks for sharing that context! Since no interface exists for searching Gists today, we encourage you to submit a new request through our official product feedback form so that our product team can track your request. Currently, that’s the best place to share requests like these in consideration for future iterations of GitHub features. It might also be worth following our GitHub public roadmap and our GitHub changelog for upcoming features and updates respectively.

Would you like to know anything more about Gists in relation to the GitHub API? :thought_balloon:

I will have to go look up the code later if you are interested; it’s still possible to programmatically use the console search even if it is not using the rest api. We do this at my org to help monitor gists for users that may accidentally be exposing org resources accidentally. For us it’s just a small ruby app that runs and sends a slack notification when it flags something for the security team to review.

@majormoses - Thanks for being open to share (if @Seancarpenter requests it :+1:); curious to know if that code is specifically tailored for a GitHub Enterprise instance rather than GitHub.com? :thought_balloon:

@francisfuzz it is against public github, it had been a while since I looked at the code, its a fairly light sinatra app and does require redis but much of that is being around:

  • performance
  • flexibility of searching and massaging data
  • running as a service rather than adhoc
  • needing to specify flexible configuration for query expressions
  • needing to filter the data further than what was returned
  • notification/alerting

I will try to include the important bits but apologies that there will be missing functions referenced.

@Seancarpenter since there was no rest endpoint we (ab)used the non rest interfaces by scraping and do all the rest of the processing on our own. Essentially http calls retrieve html which we then parse and store in redis for further searching.

module GistSearch
  class << self
    def search_public_gists(query:, page_limit: 10)
      docs = []
      base_url = "https://gist.github.com"
      next_path = "/search?q=#{query}"

      page_limit.times do
        url = "#{base_url}#{next_path}"
        html = fetch_cached(url)

        doc = Nokogiri(html)
        docs << doc

        if !doc.search("a[rel=next]").empty?
          next_path = doc.search("a[rel=next]").attr("href").value
        else
          break
        end
      end

      docs
    end

    def build_items(queries:)
      queries.flat_map do |query|
        search_public_gists(query: query).flat_map do |doc|
          doc.
            search(".gist-snippet").
            map do |gist_snippet|
              Item.new(
                creator: gist_snippet.search(".creator a:first").text,
                title: gist_snippet.search(".creator a").map(&:text).join(" / ") + " (Matched query: #{query.inspect})",
                link: gist_snippet.search(".link-overlay").attr("href").value,
                guid: gist_snippet.search(".link-overlay").attr("href").value,
                pub_date: Time.iso8601(gist_snippet.search("time-ago").attr("datetime").value),
                content: gist_snippet.search(".js-file-line").map(&:text).join("\n"),
              )
            end
        end
      end
    end

I hope this is helpful in helping you build your own solution until there is a rest api available.