Request Github open up project wiki pages to web indexes

I respectfully request that the Github team consider removing the restriction in their robots.txt file that prevents search engines from indexing github wiki pages.

I put a great deal of effort into providing wiki pages that would assist users of my open source software. I also hoped that they would help potential users find the project by providing meaningful content related to the problems my software addresses. The fact that Bing, Google, and other search engines cannot index my wiki pages limits the effectiveness of these pages for my project and the usefulness of their content for the Internet community as a whole.

For example, I’ve written an article on Natural Neighbor Interpolation, which is a function my software supports. It’s a niche topic and the information I supply is not well-covered elsewhere. Enough people have linked to my article that if your run a Google search on “Natural Neighbor Interpolation” my wiki page comes up as the fourth item in the search. But, disappointingly, the description line on Google’s search page reads “No information is available for this page”. It doesn’t even show up in Bing.

I have thought about this for a few hours now, and I cannot think of a plausible rationale for deterring search engines from indexing Github pages… But I accept that there must have been reason for doing so. However, I believe that project Wiki pages are a special case. Nobody would put content in a wiki page if it were not intended for public consumption. So I ask that the Github managers at least give this issue due consideration.

Thank you for your attention in this matter.

1 Like

Hi there! :wave: Welcome to the Community!

According to our SEO and engineering teams, we originally blocked /wiki in January 2012 to address spam and any risks from wikis being open to anyone adding content. (When wikis were first introduced the default settings meant that anyone could edit them, whether they were a collaborator on the repository or not.)

Some pages had slipped through since it wasn’t written with a proper wild card (*). That was fixed in May 2020 blocking all /wiki/ directories.

I’m afraid this is a deliberate update, and it is not likely to be reversed due to the risk of wikis being used for spammy purposes.

So sorry about that; I completely understand why this is a blocker for your project and others that use wikis. If you feel so inclined, it couldn’t hurt to submit your explanation/request to our product managers through our official product feedback form.

1 Like

Tammy,

Thanks for your reply and the information. While I cannot imagine why opening up a wiki for indexing by a web crawler would be a security issue, I can accept that this is Github policy and not likely to change.

Stay safe.

Gary

Tammy,

Thank you for your email last month. I had a follow up question. Do you know whether the Github pages feature also blocks web crawlers? Do they permit pages to be indexed? I’ve poked around a bit, but have been unable to determine the policy either way.

Since you wrote to me last month, I’ve thought a bit about the potential security vulnerabilities associated with allowing wiki’s to be indexed. It has occurred to me that while there might be some threat from hackers, there might be a more serious threat from patent trolls. So I can certainly understand your SEO’s position.

Thank you again for your attention in this matter.

Gary

Hi Gary,

Yes, we do permit pages to be indexed, and Pages definitely does not block webcrawlers. I think it would be a bit of a waste if nobody’s web page showed up in a Google search. :slight_smile:

1 Like