Hard-fail Security checks in your CI/CD pipelines

security
Author

Christophe Beaucé

Published

June 16, 2023

Infrastructure as Code is an important tool in the toolbox of Cloud Architects and developers.

It is also widely used by enterprises.

I am a big fan of Security checks performed on Infrastructure as Code.

I frequently scan code with the open source checkov, for example, when using Terraform.

As a Cloud Security practioner, I deploy checkov scans within CI/CD pipelines to make sure the terraform code deployed by application teams is compliant with our security policies.

However, with tools such as checkov, developers can always skip security checks. In some cases, skipping security test is legitimate. But in other cases, this is a way to escape from Security and a road to future troubles.

I have come up with an idea to solve this issue neatly I believe.

Within my CI/CD pipelines, I introduce 2 stages of Security checks.

flowchart LR
  A(tf fmt) --> B(tf validate)
  B --> C(checkov scan on tf code)
  C --> D(tf plan)
  D --> E(checkov scan on tf plan)

The first stage scans the terraform code itself. At this stage, checkov parses the comments in the code, and some policy checks can be skipped by developers.

The second stage happens after the “terraform plan” action: another checkov security scan is performed on the terraform plan itself.

The benefit of this scan is that the checkov skip comments are actually not present in the terraform plan. Therefore, none of the Security policies can be skipped.

Of course blindingly enforcing all Security checks would be too strict.

That’s why in the second phase, the terraform scan has a soft-fail by default, with a carefully selected list of hard-fail policies. It can be easily set in a configuration file where I define the list of hard-fail policies, in addition with any extra custom policies I may add to the checkov default policies. All of this in a Git repository maintained centrally for multiple projects.

I have used this approach to enforce over 30 security policies for one of my customer with close to one hundred terraform projects. I incrementally added more and more ‘hard-fail’ security policies after communicating in advance to let development teams prepare. As the organization Security matures, more Security policies are added to the list of “hard-fail” ones.

This approach, although gradual, nonetheless avoided multiple security misconfigurations.

Shifting left Security works.