Introduction
In today’s digital landscape, securing code repositories from the start is more critical than ever. The rise of sophisticated cyber threats and the growing emphasis on Software Supply Chain Security demand proactive security measures across the development pipeline. As the number of repositories grows, the risks of manual change management become apparent, as highlighted by breaches like those at Toyota and MedData. Configuring GitHub settings as code has become a key strategy for standardizing our repository setups, enforcing security, and streamlining compliance.
Our initiative to centralize and standardize GitHub repository configurations began with a heavy lifting technical effort, importing all existing repositories and permissions into Terraform that consumed quite some time for our small team. However, the most significant change was cultural: shifting from a decentralized model where users had admin rights to a centralized approach where all configurations and new repositories are managed as code. This move not only strengthened our security but also enhanced collaboration and consistency across the organization.
Far from hindering developer experience, this shift has streamlined the process. Creating a new repository now takes just minutes, with hardened configurations, branch protection (this is a great article explaining it), and integrated security and Quality Engineering tools built-in, allowing developers to focus on coding rather than setup. Configuration changes that once took hours can now be implemented in minutes expanding multiple repositories at once.
This blog post details Apheris's journey to managing all GitHub repositories as code, the strategies employed, and lessons learned. We’ll explore how integrating DevSecOps practices, managing configurations as code, and automating change management have fortified our GitHub repositories against potential threats, establishing a scalable, robust approach to security across our Github organization. The post is designed for IT, Security or Compliance teams who are responsible for maintaining and securing their organization's GitHub repositories and platform and have some exposure to Terraform and Infrastrucure-as-Code.
Apheris uses Github Enterprise Single-Sign On (SSO) with conditional access policies, enforcing user and machine authentication with location aware conditions and phishing-resistant MFA keys. Those security configurations will not be covered on this blog post.
Foundations
This project was initiated by our Engineering Infrastructure team, who started managing their infrastructure-related repositories as code with the Terraform GitHub provider. Seeing the substantial advantages of this approach, the Apheris Security team quickly recognized its potential and spearheaded the effort to standardize and adopt it organization-wide.
Building on this initiative, we embraced the concept of innersource, which brings open-source principles into the organization to encourage collaboration, transparency, and shared ownership of code across teams. By adopting innersource practices, we aimed to break down silos and foster a culture where everyone can contribute to and learn from each other's work. As part of this approach, we balanced security with openness by setting a base permission level in our GitHub organization that grants read access to all our Github employee users by default. This ensures that everyone can easily review code across the company, promoting knowledge sharing and cross-team collaboration, while still maintaining the necessary controls to protect sensitive information.
Before implementing any of the changes discussed in this article, one key piece of advice we recommend is to first establish a clear agreement with your Engineering teams on default settings and standards. At Apheris, we utilize a Request For Comments (RFC) process to facilitate discussions and achieve consensus across all teams. This collaborative approach has been invaluable in reaching agreements on topics such as:
Enforcing branch protection on all production repositories.
Defining the number of approvals required per pull request, which can vary based on team size.
Requiring all checks to pass before a pull request can be merged.
Ensuring branches are up to date and that no comments are left unresolved before merging.
Mandating signed commits.
Enforcing code owners to maintain clear ownership of code areas.
Automatically deleting branches after they’ve been merged.
Require conversation resolution before merging.
While teams can request fine-tuning for specific repositories, the default settings established through our RFC process serve as the baseline. Each repository also has a designated team responsible for maintaining it. Although anyone in the organization can contribute, maintainers are primarily responsible for reviewing pull requests.
For those looking for a solid foundation for these configurations, the CIS Software Supply Chain Security Guide, particularly the section on source code, is an excellent starting point.
Technical Detail
Terraform Module
To scale these security practices across the entire organization, we leveraged Infrastructure as Code (IaC) principles. By using Terraform and the GitHub provider, we were able to manage our GitHub repository settings as code. Our Terraform module has resources for the repository the branch protection using variables for the fine tuning of each repository:
Github Teams & Permissions
For every team in the Apheris organisation that needs access to Github, there will be two Github teams created as code:
One that has all the repository owning team members.
Another called “Breakglass“ that will contain the Security team and the Security Champion of the owning team.
Our decision to set up a base permission level in our GitHub organization that grants read access to all our Github employee users by default has simplified our permissions configuration as code. We have created two additional Github custom roles:
Releaser: Inherits from the Github repository role write permission but with the additional permission to create tags.
Breakglass: Inherits from the Github repository role write permission but with the additional permissions to delete tags and to bypass branch protection.
For every new repository created, the following team repository permissions will be created (additional “write“ permission for other teams are granted as needed):
Owning team will have a Releaser role.
Breakglass team will inherit the Breakglass role.
With those permissions we ensure that teams can operate normally on their day to day tasks, with the minimum set of permissions and can respond themselves to the exceptional cases such as to bypass a branch protection or to delete a tag.
Future work will be to build monitoring and alerting on bypass branch protection events. Also we will be looking at Github organization roles.
Pull Request Checks
Branch protection is a crucial tool to ensure quality and security during software development. We can configure continuous integrations checks to block the pull request from merging as long as they fail. We want these checks to be flexible, yet consistent across repositories, which we achieved with a combination of 1) ensuring all jobs triggered in the pull request context are green (for that we use all-greens) and 2) requiring the status checks to pass.
Apheris uses Aikido security to scan all our code and third party dependencies for vulnerabilities, secrets and malware. For us, mandating that those pull request checks are enforced to pass on each repository is as simple as adding an extra line to our required status checks within our infrastructure as code which uses Aikido’s provided Github App to do the integration with their platform. That way we ensure that all our code is scanned on each pull request and failures will block the code from being merged into the main branch. Switching all our security scanners to Aikido took only a few days, thanks to having our entire infrastructure configuration managed as code.
To also allow teams (Quality Engineering and the relevant Engineering teams) to mandate the checks that they consider mandatory to be passing on each of their pull requests without the need to any change on our Github infrastructure as code, we use the all-greens functionality. On our side, we mandate that quality-checks workflow needs to pass on the branch protection for the pull request to be merged and the specific details of what that means are controlled by the teams owning the repository and our Quality Engineers team:
Additionally:
We enable Dependabot vulnerability alerts across the organization.
All our repositories utilize the open-source Renovate tool to uplift third-party dependencies and container base images. We also take advantage of Renovate's automerge capabilities under specific conditions but this requires the Renovate App to bypass the repository's branch protection settings that mandate pull request reviews. However, it still ensures that all required checks, such as Quality Engineering and Security, pass before merging the pull request.
The repository containing all the Github infrastructure as code is owned by the security team and follows the same standard configuration as any other repository and it is also scanned for security misconfigurations.
Code Owners
Ownership of a GitHub repository is a key responsibility assigned to specific teams within our organization. To ensure that the team responsible for a repository actively participates in reviewing code proposals, we’ve implemented branch protection rules that mandate pull request reviews from designated code owners. This approach guarantees that any changes to the repository are reviewed by those who are most familiar with the codebase, ensuring consistency, adherence to best practices, and alignment with the team's standards. By requiring code owners to approve pull requests, we reinforce accountability and maintain the integrity of our repositories.
To work around the limitations of GitHub's search feature and ensure that every repository includes a code owners file that meets our standards, we’ve used an open-source tool called Steampipe. It allows us to query the GitHub APIs using SQL, making it easier to check that all repositories have the correct code owners files in place (something that might not be the case for the imported repositories). An example query to search for non-archived repositories that are missing a CODEOWNERS file looks like the following:
Repository Templates & Archival
We’ve implemented GitHub templates (minimal setup, python projects etc) to streamline the creation of new repositories and ensure consistency across our projects with the bootstrapping of repositories. These templates come pre-configured with essential settings needed for our development:
The selection of the template is part of our Github infrastructure as code. We colllect all information to spin up a new repository from our Helpdesk portal:
To be able to archive repositories also as code without breaking our Terraform state, we use the archive_on_destroy variable so the repositories are actually archived when we remove the configuration from our infrastructure as code.
Organizational Secrets
At Apheris, we manage organization-level GitHub secrets using the github_actions_organization_secret Terraform resource. This resource allows us to create secrets at the organization level that can be accessed by specific repositories within the organization.
In line with our commitment to the principle of least privilege, we ensure that these secrets are only visible to the repositories that absolutely need them. To enforce this, we set the visibility of the secret to selected and specify a list of repositories that should have access. The secret itself is encrypted using the encrypted_value property, ensuring that sensitive information is securely stored.
Here's an example of how we configure a GitHub organization secret:
When an organization secret is created, its encrypted value is generated using the organization's public key. GitHub Actions then decrypt this value when the secret is used in workflows. This encryption can be achieved through various methods, including using the GitHub CLI or Python scripts, ensuring that the process is secure and efficient.
Using the Github CLI is the easiest way to encrypt the secret, but it requires admin:org permissions in order to retrieve the public key. The following command will generate a base64 encoded encrypted value for a secret:
We have developed an internal python script to overcome the limitation of the admin:org permissions as it already has access to the organization public key.
Challenges & Future Work
One of the most significant challenges we faced was the initial effort required to import our existing repositories into the new system. Due to the complexity and variety of configurations across different repositories, this process was both time-consuming and resource-intensive. However, the relatively small size of our startup allowed us to manage this transition more efficiently than a larger organization might have.
As our Terraform state has grown, particularly with the management of environment variables and deploy keys as code, we've encountered slower Terraform plans and applies. To address this, we plan to split our Terraform state to improve execution speed. We're also exploring tools like Atlantis to automate Terraform Pull Requests, further streamlining our workflow.
Enabling certain settings can sometimes impact the developer experience, so it's important to approach these configurations with caution. For instance, requiring a specific number of approvals per pull request, depending on team size, or ensuring branches are up to date and that no comments are left unresolved before merging, can introduce additional steps in the workflow. While these measures can enhance security and code quality, they can also slow down the development process if not carefully managed.
Similarly, implementing Git commit signing brought its own set of challenges. Once enabled on a repository, all open pull requests with unsigned commits required a branch protection bypass. Moreover, the "Rebase and Merge" option cannot be signed by a key, adding further complexity to the process. These examples underscore the need to balance security with developer efficiency, ensuring that new settings improve overall practices without causing unnecessary disruption.
Another important consideration is how we manage bypass branch protection permissions. Currently, we have a small group (Security and Security Champions) with these permissions. While GitHub is extending functionality with delegated bypass, which we haven't yet explored, we recognize that there will be situations where this feature is necessary. We plan to monitor and alert when these permissions are used to ensure proper oversight.
Finally, while we are grateful for the open-source contributions of the GitHub Terraform provider, it's important to note that newly released GitHub features (ie pre-defined organization roles) may not immediately be supported by the provider. This can result in certain tasks needing to be performed manually until the functionality is incorporated into the provider or a contribution can be done to the provider to pay back to the open source community.
Conclusion
Our commitment to securing GitHub repositories has fundamentally transformed how we protect and manage our codebase. By treating GitHub as a critical infrastructure component, we've implemented rigorous security measures and adopted Infrastructure as Code to ensure consistent protection across all teams. What began as an initiative by our Infrastructure team has become a cornerstone of our DevSecOps strategy, significantly reducing our exposure to potential threats and supply chain attacks.
The success of this project is clear through several key metrics. We've maintained a positive developer experience, ensuring efficient workflows without disruption. Setting up and configuring new repositories now takes just a few minutes, and collaboration has increased, with teams outside of security and infrastructure actively contributing to our GitHub configuration as code. Additionally, making changes across all repositories—such as responding to deprecations—now requires only a few lines of code, greatly enhancing our productivity and responsiveness. These outcomes underscore the long-term value of our approach in building a more agile, secure, and collaborative development environment.
References
https://www.cisecurity.org/insights/white-papers/cis-software-supply-chain-security-guide
https://www.arnica.io/blog/what-every-developer-needs-to-know-about-github-branch-protection
https://attack.mitre.org/techniques/T1602/
