Access management in GitHub organizations with many teams and many repositories can be challenging. That is why, I prefer to use Terraform to manage some of my GitHub organizations with code. This is the story of how I discovered a bug in GitHub using Terraform.

Terraform is a powerful tool that I daily use. It works through plugins called “providers” and it can help you manage almost anything with an API that provides create, read, update, and destroy actions. GitHub maintains its own provider for Terraform and it supports all actions required to manage team access to repositories.

Prologue

Everything started when I started to manage another organization of mine with Terraform. Using code, I was able to invite members to my organization, assign them to teams, and grant these teams access to some of the repositories. I was using an API token with admin:org scope and it was working perfectly, except for the last step.

It was completing the operation (granting team permission for a repository) but Terraform was failing to update its state. At first, I assumed it was a bug in the provider. I have visited the repository to check if anyone has reported the same issue, but there was none. I decided to run Terraform in debug mode and see if I could find anything useful in the logs. I noticed the last API call returned HTTP 404 Not Found, and this is exactly where it got interesting!

Inconsistent behavior in GitHub API

After creating a resource (it can be anything, in this case, “team permissions for a repository”) Terraform performs a read request to verify the resource has been created successfully. In my case, it was failing after a successful creation.

The first thing I checked was the GitHub API documentation. This was what it says:

If the repository is private, you must have at least read permission for that repository and your token must have the repo scope. Otherwise, you will receive a 404 Not Found response status.

The paragraph was clear enough and my token didn’t have repo scope. Most people would have granted scope to the token and continue happily with their lives, but not me. It made no sense, I had a token with a powerful scope that could grant team permission but it wasn’t powerful enough to check it, really?

I dug deeper and noticed GitHub API sends a header of allowed scopes: x-accepted-oauth-scopes: admin:org, read:org, repo, user, write:org. According to this header, a token with admin:org scope should be sufficient. It was clearly a bug.

Convincing GitHub Support

At this point, I decided to contact GitHub support and report the bug. In the report, I have included both the debug logs from Terraform and the corresponding cURL command as an easy way to reproduce.

Even though the bug was very clear to me, it wasn’t clear to the support engineer handling my ticket. We discussed the issue back and forth for two days and I had to try several different approaches to make my case, but since the documentation was mentioning repo scope, they just assumed I was doing something wrong and didn’t pay any attention to the inconsistency at all.

In the end, I paraphrased the situation and made it sound like a security issue. I told them that I could grant team permissions for a repository without the repo scope and demanded they check in with the API team and make sure the repo scope is really required. The word “security” convinced the support team, they made sure repo scope shouldn’t be mandatory if the token has another accepted scope, and they said they would fix the documentation.

Once again, I had to explain that the issue was not just a misleading paragraph in the documentation, and the API was not working as expected with admin:org scope. I have also included four cURL commands in my report, for both granting and checking team permission with two different tokens each, one with admin:org and the other with repo. Outputs were clear that all of them were working as expected except checking team permissions with admin:org scope.

Resolution

On November 2nd, I have received the following message:

Hi Onur,

Thank you for sharing those details.

Apologies, my initial assessment of the issue was wrong. I see what you mean now and I agree this is a bug. If the admin:org permission can add/update team repository permission then it should be able to check the team repository permission.

I have filed an internal bug issue with the engineering team. I don’t have a timeline on when they’d get back to me but will keep a close eye on the issue and let you know when I have any news.

Regards,

The next update arrived on November 21st:

Hi Onur,

The Engineering team deployed a fix for this behind a feature flag and enabled it for your organization. Please can you confirm this on your end?

Regards,

I have tried all four variations of my previous cURL commands and confirmed the fix is working as expected. Furthermore, I asked when the fix would be released to the public, however GitHub support engineer said that the timeline wasn’t clear. That’s why I was surprised when I got this message on the next day:

Hi,

I just thought to mention that the fix has been made available to the public. Update to the relevant documentation section is also in progress and would be publish as soon as the underlying PR is approved.

Regards,

Within a few days, the documentation was also updated to include admin:org scope. Now the paragraph says:

If the repository is private, you must have at least read permission for that repository, and your token must have the repo or admin:org scope. Otherwise, you will receive a 404 Not Found response status.

Finally, the happy ending.