Create an integrated STO/CI pipeline
This tutorial builds on the Create a standalone STO pipeline tutorial. You need to complete the standalone STO pipeline, which scans a test target and reports on the vulnerabilities, but doesn't do anything else.
In this tutorial, you'll learn how to integrate STO functionality into CI and CD pipelines. The core benefit of STO in an integrated pipeline is to fail the pipeline if a scanner finds any "show-stopper" vulnerabilities. The following sections describe the different failure strategies you can implement.
For the list of supported scanners, go to Security step settings reference.
Objectives
This tutorial guides you through the following workflows:
- Set up a security step to fail automatically if it detects an issue with the specified severity or higher.
- Request an exemption ("ignore rule") for a specific issue.
- Approve the exception. Once approved, the exemption won't fail the pipeline even if it equals or exceeds the severity threshold.
Fail pipelines on severity
Key concept: fail_on_severity
Every STO scan step has a fail_on_severity setting. If any vulnerability with the specified severity or higher is found, the pipeline fails.
It is good practice to set fail_on_severity in every scan step in an integrated pipeline.
In the Pipeline Studio, open the pipeline that you created in the Standalone pipeline tutorial.
Open the Bandit step.
Set Fail on Severity to Critical.
Select Apply Changes, save the updated pipeline, and run a new build with the DEMO-001 branch.
The pipeline now fails because the Bandit step is now configured to fail on any critical vulnerability . The last log message in the Bandit step log is:
Exited with message: fail_on_severity is set to critical and that threshold was reached.
Exemptions for specific issues
Key concept: Exemptions, requests, and approvals
You can exempt known issues from fail_on_severity so that they don't stop the pipeline even when a scan detects them. The following steps outline the workflow:
A developer requests an exemption for a specific issue and forwards the request to a SecOps user.
The SecOps user approves the request or rejects it. Developer users can request exemptions, but only SecOps users can approve them.
If the exemption is approved, and a future scan detects the exempted issue, the pipeline execution will not fail even if the issue meets the
fail_on_severitythreshold.
In this section, you'll create an exemption as a developer and then approve it as a SecOps user. (In many real-world scenarios, two separate people will be performing the workflow.)
Make sure that you have the SecOps role assigned to yourself:
Select Account Settings (left menu) > Access Control.
In the Users table, select your user profile.
Under Role Bindings, select +Role.
Make sure that you have the Security Testing SecOps role assigned to yourself.
Go to the Security Tests page for the build you ran previously: In the pipeline Studio, select Execution History (top right) and then select the last successful build you ran before the failed build.
In the following step, you will create an exemption for each of the two critical issues found:
subprocess_popen_with_shell_equals_true(only in the current scan) andhashlib(common to the baseline scan).In the Security Tests tab, do the following steps for each critical issue:
Select the critical issue in the issues table (bottom left) to open Issue Details.
Select Request Exemption.
In Request Exemption for Issue, configure the exemption request as follows:
- Where do you want this issue to be exempted? This pipeline
- For how long? 1 Day (if available)
- Reason this issue should be exempted: Other
- Further describe the reason this issue should be exempted: Tutorial example pipeline, not for use in QA or Prod environments
- Select Create Request.
Select Security Tests (left menu) and then Exemptions (second-from-left menu).
In the Security Review page, select the "thumbs-up" buttons to approve both exemptions.
Go back to your pipeline and run another build with the DEMO-001 branch. When the build finishes, go to the Security Tests page.
Select Exempted (far right, under Security Executions). Note that this button, like the Critical, High, and other buttons, acts as a toggle to show and hide specific issues in the issues table. If you select and unselect Exempted, the exempted issues switch between visible and hidden.
Go to Security Tests > Exemptions. Then select Approved to show the Ignore rules you created and approved.
Select the Delete (X) buttons on the right to delete both rules.
Next steps
You've now learned the core STO features and workflows. Here are the next steps you can take.
Add more scanner steps
STO supports an extensive set of external scanners for repos, images, and artifacts. Go to What's supported.
Add steps or stages for CI/CD workflows
You know how to implement pipelines when scanners detect security issues, and how to create Ignore Rules for specific issues. Once you set up your Security steps, baselines, and exemptions, you can add more stages and steps to implement your CI/CD workflows.
Add governance policies
You can use the Harness Policy Engine to create policies based on the Open Policy Agent (OPA) standard. For example, you can create a rule like the following to ensure that all pipelines include a Security stage.
package pipeline_required
# Deny pipelines that are missing required steps
deny[sprintf("CI stage '%s' is missing required step '%s'", [stage.name, existing_steps])] {
stage = input.pipeline.stages[i].stage # Find all stages ...
stage.type == "CI" # ... that are CI stages
existing_steps := [ s | s = stage.spec.execution.steps[_].step.type ] # ... and create a list of all step types in use
required_step := required_steps[_] # For each required step ...
not contains(existing_steps, required_step) # ... check if it's present in the existing steps
}
# steps that must be present in every CI stage - try to create a CI stage without a Security step to see the policy fail
required_steps = ["Security"]
contains(arr, elem) {
arr[_] = elem
}
Add failure strategies to a CI/CD stage
You can implement Failure Strategies to bypass the failure policies in previous security steps. One use case for this would be to enable manual interventions when a Security step generates a failure. You can set up a workflow like this:
- A Build step is downstream from the Security step. It has a failure strategy that's set to run on All Errors.
- The scanner detects issues and the Security step generates an error.
- The Failure Strategy in the Build step initiates a 30-minute pause before proceeding.
- The developer and security team evaluate the issues and then abort the pipeline execution or allow it to proceed.
Integrated STO/CI Workflow Example
The following pipeline extends the example workflow described above. After it scans the repo, it builds a container image, scans the image, and fails the pipeline if the image scan fails. The YAML of this pipeline is provided below.
This pipeline works as follows:
The SecurityTestStage consists of the following steps:
- backgroundDinD runs Docker-in-Docker as a background service. This is required to scan the container image.
- banditScan scans a GitHub repo used to build the container image.
In this case,
fail_on_severityis set tohigh. - buildAndPush_PRIVATE builds a local container image from the repository and pushes it to a private registry.
- The aquaTrivyScan step uses the open-source tool Aqua Trivy to scan the local image. It has
fail_on_severityset tohigh. - If the container image has no issues with high or critical severity, buildAndPush_PUBLIC pushes the image to a public registry.
The sendEmail stage includes an step that sends an email if the previous stage succeeded.
After the pipeline executes, you can view all issues from all scanners in the Security Tests tab, and also filter the issue list by scanner.
Congratulations!
In this tutorial, you learned how to:
- Configure
fail_on_severityto fail a pipeline execution if a scan detects a vulnerability with the specified severity or higher. - Request a exemption for a specific vulnerability (if you're a developer) and approve an exemption (if you're a SecOps person).
Integrated Workflow YAML
Here's the YAML of the integrated workflow example we examined in this tutorial.
Integrated Workflow YAML
pipeline:
projectIdentifier: STO_tutorial_test_2023_09_29
orgIdentifier: foobar
tags: {}
properties:
ci:
codebase:
connectorRef: account.GitHub_STO_Tutorial
repoName: dvpwa
build: <+input>
stages:
- stage:
name: securityTestStage
identifier: securityTestStage
type: CI
spec:
cloneCodebase: true
sharedPaths:
- /var/run
infrastructure:
type: KubernetesDirect
spec:
connectorRef: account.k8sdelegateconnector
namespace: harness-delegate-ng
automountServiceAccountToken: true
nodeSelector: {}
harnessImageConnectorRef: account.harnessImage
os: Linux
execution:
steps:
- step:
type: Background
name: backgroundDinD
identifier: Background_1
spec:
connectorRef: account.harnessImage
image: docker:dind
shell: Sh
privileged: true
- step:
type: Bandit
name: banditScan
identifier: Bandit_1
spec:
mode: orchestration
config: default
target:
name: dvpwa
type: repository
variant: " <+codebase.branch> "
advanced:
log:
level: info
fail_on_severity: high
failureStrategies:
- onFailure:
errors:
- AllErrors
action:
type: Ignore
- step:
type: BuildAndPushDockerRegistry
name: buildAndPush_PRIVATE
identifier: BuildAndPushDockerRegistry_1
spec:
connectorRef: account.foobardockerhubsto
repo: foobar/sto-tutorial-test-private
tags:
- <+pipeline.sequenceId>
- step:
type: AquaTrivy
name: aquaTrivyScan
identifier: AquaTrivy_1
spec:
mode: orchestration
config: default
target:
name: sto-tutorial-test-private
type: container
variant: <+pipeline.sequenceId>
advanced:
log:
level: info
fail_on_severity: high
privileged: true
image:
type: docker_v2
name: foobar/sto-tutorial-test-private
tag: <+pipeline.sequenceId>
- step:
type: BuildAndPushDockerRegistry
name: buildAndPush_PUBLIC
identifier: buildAndPush_PUBLIC
spec:
connectorRef: account.foobardockerhubsto
repo: foobar/sto-tutorial-test
tags:
- <+pipeline.sequenceId>
variables: []
- stage:
name: sendEmail
identifier: sendEmail
description: ""
type: Custom
spec:
execution:
steps:
- step:
type: Email
name: Email_1
identifier: Email_1
spec:
to: my.email@myorg.org
cc: ""
subject: NEW IMAGE! Scan results for <+pipeline.name>
body: "STO scan of <+pipeline.name> found the following issues: <br> --------------------------------------------------------------<br> Bandit repo scan:<br> Critical : <+pipeline.stages.securityTestStage.spec.execution.steps.Bandit_1.output.outputVariables.CRITICAL> <br> New Critical : <+pipeline.stages.securityTestStage.spec.execution.steps.Bandit_1.output.outputVariables.NEW_CRITICAL> <br> High: <+pipeline.stages.securityTestStage.spec.execution.steps.Bandit_1.output.outputVariables.HIGH> <br> New High: <+pipeline.stages.securityTestStage.spec.execution.steps.Bandit_1.output.outputVariables.NEW_HIGH> <br> Medium: <+pipeline.stages.securityTestStage.spec.execution.steps.Bandit_1.output.outputVariables.MEDIUM> <br> New Medium: <+pipeline.stages.securityTestStage.spec.execution.steps.Bandit_1.output.outputVariables.NEW_MEDIUM> <br> --------------------------------------------------------------<br> Aqua Trivy image scan:<br> Critical : <+pipeline.stages.securityTestStage.spec.execution.steps.AquaTrivy_1.output.outputVariables.CRITICAL> <br> New Critical : <+pipeline.stages.securityTestStage.spec.execution.steps.AquaTrivy_1.output.outputVariables.NEW_CRITICAL> <br> High: <+pipeline.stages.securityTestStage.spec.execution.steps.AquaTrivy_1.output.outputVariables.HIGH> <br> New High: <+pipeline.stages.securityTestStage.spec.execution.steps.AquaTrivy_1.output.outputVariables.NEW_HIGH> <br> Medium: <+pipeline.stages.securityTestStage.spec.execution.steps.AquaTrivy_1.output.outputVariables.MEDIUM> <br> New Medium: <+pipeline.stages.securityTestStage.spec.execution.steps.AquaTrivy_1.output.outputVariables.NEW_MEDIUM> <br> See https://app.harness.io/ng/#/account/MY_ACCOUNT_ID/sto/orgs/default/"
timeout: 10m
tags: {}
identifier: tutorial4integratedstoci
name: 2023-tutorial-4-integrated-sto-ci