Writing & Testing Policies

The way we write the policies for Mequal follow the guidelines provided in Conforma, just in a more opionated manner and with some additional extensions to it. Even though Mequal is using OPA, this is to make sure our policies remain compatible with Conforma when it comes to validation and gating. For more detail on these guidelines, visit the Conforma documentation page.

1. Basic Concepts for Policy Evaluation

In this section we’ll dive into the basic concepts of policy evaluation at a high-level.

a. Policies

A policy is a set of rules that are evaluated against a specific input. In our case that input would be the SBOM we pass into Mequal for evaluation.

b. Rules

A rule is a self-contained condition that belongs to a policy. Rules are what make up each policy and can be considered the smallest unit of evaluation.

c. Prerequisites

Prerequisites is a set of conditions that an input is supposed to meet before a policy can be evaluated on said input. For example, if a policy is related to only 1.6 and higher versions of a CycloneDX SBOM, the prerequisites to check before running this policy evaluation on an SBOM is:

  • A CycloneDX SBOM

  • With version is higher than 1.6

d. Policy Repository

We store a set of policies packed into a certain hierarchical directory inside of an SCM repository. Both the written policies and the code for the policies would live in this repository and can be fetched when needed for evaluation. This enables us to maintain the policies and their changes in a systematic, version-controlled manner, where all changes and version releases can be easily tracked and managed.

e. Policy Metadata (WIP)

In order to represent a policy as code that is compatible for OPA evaluation, we need to include a certain amount of metadata related to a policy. A standardized way of representing this required information is a policy metadata schema that each written policy should comply with. (For example, each policy should have a title and description)

f. Evaluation Results

The evaluation results are the output produced by OPA after running a set of policies against a specific input. They can are in JSON format and contain what denials, warnings or guidance messages have been returned against these policies for a given manifest as input. The results can be context-specific depending on the policies we run. (For example different policy checks for different SBOM versions, SBOM types, etc.)

The denial and warning messages help with the validation of the SBOMs to make sure they comply with the policy and fail if there are any policies the SBOM doesn’t comply with. These are more related to the basic important rules the SBOM must comply with to be considered valid.

The guidance messages help give feedback against certain policies without necessarily rejecting the SBOM input. They anticipate certain information that every single SBOM may not necessarily need to represent, and provide guidance on how to include this information on the SBOM. It also tries to gauge the quality of the SBOM based on what it expected vs. what is actually found within the SBOM and gives a grading in the form of levels or points.

2. How to Write Policies

Resources on Rego

The policies evaluated against SBOMs using Mequal will need to be written using the Rego language, which is a language that is specialized in writing policies as code. So having a basic understanding of it will be necessary to write policies.

Folder Structures and Corresponding Package Hierarchy

The first practice we should make sure to follow is to keep a directory of policies consistent with the rego package structure, very similar to how it’s done in Java. So given we’d like to organize our policy like below:

example
-> sbom
---> cyclonedx
------> policy1.rego
------> policy2.rego
---> spdx
------> policy1.rego
------> policy2.rego
-> main.rego
-> .manifest

and a .rego file representing example/sbom/cyclonedx/policy1.rego should have

package example.sbom.cyclonedx.policy1

defined on top of the file.

Starting from the files on the top level, .manifest is a file that defines the metadata as well as the root of the policy bundle, which allows for uniquely identifying the policy bundle so it can be conjoined with others later down the line. With a simple definition within in as below:

{
    "revision": "v1.0.0",
    "roots": ["example"]
}

And as for the main.rego file, it is the high-level code that will allow to cleanly return the policy evaluation results, without returning everything at once. It can be defined like below:

package example.main

import data.example.sbom
import rego.v1

violations contains result if {
	# goes 2 layers deep to get messages for all policies for both cyclonedx and spdx formats.
	result := sbom[_][_].deny[_]
}

warnings contains result if {
	result := sbom[_][_].warn[_]
}

guidance contains result if {
	result := sbom[_][_].guide[_]
}

It’s important that we define one policy per file, with the rules related to that policies residing within that file.

Writing Policies in Rego

Let’s get started by writing a nice and simple example policy, and then explaining all the important parts of the code.

Let’s say we’d like to create a a policy for an SPDX SBOM that checks if the SBOM contains any packages. To do this we can create a packages.rego file within the directory example above, as example/sbom/spdx/packages.rego

As rules for this policy, we can define two of them:

  • The SBOM includes a packages field

  • The SBOM has a non-empty packages field

Our file would then look like below:

# METADATA
# title: SPDX Contains Packages (1)
# description: >-
#   Check if the SPDX SBOM contains any packages. (2)
package example.sbom.spdx.packages (3)

import data.ec.lib (4)
import data.example.sbom.is_spdx (5)
import rego.v1 (6)

# Define the prerequisites to check for each policy (i.e. what SBOMs should these policies run on?)
prerequisite if {
	is_spdx (7)
}

# METADATA
# title: SPDX SBOM has a packages field (8)
# description: The SPDX SBOM has a packages field. (9)
# custom:
#   short_name: spdx_sbom_has_packages_field (10)
#   failure_msg: SPDX SBOM does not have a packages field (11)
deny contains result if {
	prerequisite (12)
	not input.packages
	result := object.union( (13)
		lib.result_helper(rego.metadata.chain(), []), (14)
		{"custom_data": "example_value"}, (15)
	)
}

# METADATA
# title: SPDX SBOM packages field not empty
# description: The SPDX SBOM has a non-empty packages field.
# custom:
#   short_name: spdx_sbom_packages_field_not_empty
#   failure_msg: SPDX SBOM does not have a packages field
deny contains result if {
	prerequisite
	count(input.packages) == 0
	result := object.union(
		lib.result_helper(rego.metadata.chain(), []),
		{"custom_data": "example_value"},
	)
}
1 Each policy requires a title as metadata
2 Each policy requires a description as metadata
3 Package import in line with the directory structure
4 Import the helper functions from EC/Conforma
5 Can import functions defined in the policies
6 Can add to ensure rego v1 compatibility
7 All prerequisite conditions needed for the evaluation to be performed go into this function
8 Each rule within a policy requires a title as metadata
9 Each rule within a policy requires a description as metadata
10 Each rule within a policy requires an ID as metadata
11 Each rule within a policy requires a failure message as metadata. '%s' can be used within the string in order to pass variables to it.
12 Before each rule, the prerequisite conditions functions should be called to ensure evaluation is only done on inputs that satisfy the prerequisites.
13 Helper function for passing information as output.
14 String values can be passed into the empty array parameter of the helper function to populate the '%s' variables in the failure_msg metadata
15 A custom object can be passed as part of the output. In this case it would be "{"custom_data": "example_value"}"

Following the example above, we were able to write a policy with two rules, where the policy is represented as a file, and the rules are represented as separate conditionals in the file.

In the rego rules, each conditional line by line form an AND statement. The example below:

deny contains result if {
	condition1
	condition2
}

would represent condition1 AND condition2

To make an OR statement, the cleanest way to do it is to keep them in separate rules. The example below:

deny contains result if {
	condition1
}
deny contains result if {
	condition2
}

would represent condition1 OR condition2

3. How to Write Unit Tests Policies

Folder Structures and Corresponding Package Hierarchy

When writing unit tests for policies, we create a corresponding test file next to the policy we would like to test in whichever folder it’s in, as shown below:

example
-> sbom
---> cyclonedx
------> policy1.rego
------> policy1_test.rego
---> spdx
------> policy1.rego
------> policy1_test.rego
-> main.rego
-> .manifest

a .rego file representing example/sbom/cyclonedx/policy1_test.rego should have

package example.sbom.cyclonedx.policy1_test

defined on top of the file.

Writing Unit Tests in Rego

Now let’s prepare some unit tests for the example policy we defined in the section above.

package example.sbom.spdx.packages_test (1)

import data.ec.lib (2)
import data.example.sbom.spdx.packages (3)
import data.ec.lib.assert_passes_rules (4)
import data.ec.lib.assert_violates_rules
import rego.v1

# Rule IDs we would like to test (5)
_rule_spdx_sbom_has_packages_field := "example.sbom.spdx.packages.spdx_sbom_has_packages_field"
_rule_spdx_sbom_packages_field_not_empty := "example.sbom.spdx.packages.spdx_sbom_packages_field_not_empty"

# Prerequisites (6)

# If not an SPDX, make sure no rules in this policy are evaluated (i.e. don't return violations)
test_prerequisite if {
	sbom := {"name": "John", "surname": "Smith"}
	results := packages.deny with input as sbom
	lib.assert_equal(count(results), 0)
}

# Packages

test_spdx_sbom_has_packages_field if {
	sbom := {"SPDXID": "SPDXRef-DOCUMENT", "packages": [{"SPDXID": "test", "versionInfo": "1.3"}]}
	results := packages.deny with input as sbom (7)
	assert_passes_rules(results, [_rule_spdx_sbom_has_packages_field]) (8)
}

test_spdx_sbom_does_not_have_packages_field if {
	sbom := {"SPDXID": "SPDXRef-DOCUMENT"}
	results := packages.deny with input as sbom
	assert_violates_rules(results, [_rule_spdx_sbom_has_packages_field]) (9)
}

test_spdx_sbom_has_nonempty_packages_field if {
	sbom := {"SPDXID": "SPDXRef-DOCUMENT", "packages": [{"SPDXID": "test", "versionInfo": "1.3"}]}
	results := packages.deny with input as sbom
	assert_passes_rules(results, [_rule_spdx_sbom_packages_field_not_empty])
}

test_spdx_sbom_has_empty_packages_field if {
	sbom := {"SPDXID": "SPDXRef-DOCUMENT", "packages": []}
	results := packages.deny with input as sbom
	assert_violates_rules(results, [_rule_spdx_sbom_packages_field_not_empty])
}
1 Package import in line with the directory structure
2 Import the helper functions used to make the output compatible with Conforma
3 Import the policy package that we would like to test
4 Import the test assertion functions to validate if a set of rules pass or fail
5 Define the rules within the package we want to test. (package directory.short_name)
6 Test prerequisites to make sure no rules are evaluated for an example input that doesn’t follow the prerequisite conditions.
7 Define a mock SBOM and pass it as input into the policy for evaluation. Our example policy is named packages, so we get the deny violations of this policy by fetching packages.deny. If there were policies returning warn messages, we can run our mock input against those by fetching packages.warn
8 Assert that a given list of rules pass for the mock SBOM input given to the policy. Takes an array of rules to verify that the input complies with the policy.
9 Assert that a given list rules contains violations for the mock SBOM input given to the policy. Takes an array of rules to verify that the input violates the policy.

Following the example above, we’ve now written some unit tests to make sure that the mock SBOM inputs we’ve prepared either expectedly pass or fail the policy we are testing.

4. Conclusion

Following the practices above, we can now build and run Mequal with our own policies following the instructions in the Getting Started page.