Testing

Features, segments, and global variables can grow into complex configuration very fast, and it is important that you have confidence they work as expected.

We can write test specs in the same expressive way as we defined our features, segments, and global variables to test them in great detail.

For a practical CI workflow, start with testing feature flags before deployment.

Running tests#

Use the Featurevisor CLI to run your tests:

$ npx featurevisor test

If any of your assertions fail in any test specs, it will terminate with a non-zero exit code.

Testing features#

Assuming we already have a foo feature in features/foo.yml:

description: Foo feature
tags:
  - all
bucketBy: userId
variablesSchema:
  someKey:
    type: string
    defaultValue: someValue
variations:
  - value: control
    weight: 50
  - value: treatment
    weight: 50
rules:
  production:
    - key: everyone
      segments: '*'
      percentage: 100

We can create a new test spec for it in tests directory:

feature: foo # your feature key
assertions:
  # asserting evaluated variation
  # against bucketed value and context
  - description: Testing variation at 40% in NL
    environment: production
    at: 40
    context:
      country: nl
    expectedToBeEnabled: true
    # if testing variations
    expectedVariation: control
  # asserting evaluated variables
  - description: Testing variables at 90% in NL
    environment: production
    at: 90
    context:
      country: nl
    expectedToBeEnabled: true
    # if testing variables
    expectedVariables:
      someKey: someValue

The at property is the bucketed value (in percentage form ranging from 0 to 100) that assertions will be run against. Read more in Bucketing.

If your project has no environments, you can omit the environment property in your assertions.

File names of test specs are not important, but we recommend using the same name as the feature key.

Testing global variables#

Place variable test specs under tests/variables/ and identify the variable with the variable field:

variable: supportEmail
assertions:
  - description: Uses the Netherlands override
    environment: production
    context:
      country: nl
    expectedValue: support-[email protected]
    expectedEvaluation:
      reason: variable_override_rule
      variableOverrideKey: netherlands
      variableOverrideIndex: 0

Variable assertions support the following fields:

Global variables do not use percentage bucketing themselves. When a variable or one of its overrides declares requiredFeatures, use at to select the bucket position for those feature evaluations:

variable: signupMessage
assertions:
  - description: Required feature is in its treatment allocation
    environment: production
    at: 75
    expectedValue: Sign up with your preferred provider

at must be a number from 0 to 100. It can also be a complete matrix placeholder when every resulting value is in that range. The same position applies throughout a transitive required feature chain, and it has no effect when no required feature is evaluated.

Use stickyFeatures when the exact upstream result should be supplied instead of evaluated:

variable: signupMessage
assertions:
  - description: Required feature has a sticky treatment result
    environment: production
    stickyFeatures:
      allowSignup:
        enabled: true
        variation: treatment
    expectedValue: Sign up with your preferred provider

Sticky feature values take precedence over at for their own feature keys. Other required features continue to use at. Test linting checks sticky feature keys, variation values, feature variable keys, and global variable keys before the test runs. stickyVariables remains separate and fixes the global variable's own value before normal evaluation.

Use children to evaluate the same variable through child SDK instances. Children inherit the parent context and then apply their own context. They continue to use the parent's at, but do not inherit the parent's sticky feature or variable maps:

variable: campaignBanner
assertions:
  - description: Uses progressively more specific context
    environment: production
    context:
      country: nl
    expectedValue: Welkom
    children:
      - context:
          city: amsterdam
        expectedValue: Welkom Amsterdam
      - stickyVariables:
          campaignBanner: Preview
        expectedValue: Preview

Every child must declare at least one of expectedValue or expectedEvaluation. Matrix expansion also applies recursively inside children and their sticky values.

Read Global variables for the complete authoring and evaluation model.

Testing segments#

Similar to features, we can write test specs to test our segments as well.

Assuming we already have a netherlands segment:

description: The Netherlands
conditions:
  - attribute: country
    operator: equals
    value: nl

We can create a new test spec in tests directory:

segment: netherlands # your segment key
assertions:
  - description: Testing segment in NL
    context:
      country: nl
    expectedToMatch: true
  - description: Testing segment in DE
    context:
      country: de
    expectedToMatch: false

Matrix#

Use the optional matrix property when an assertion needs to cover many combinations of values.

For example, in a feature test spec:

feature: foo
assertions:
    # define a matrix
  - matrix:
      at: [40, 60]
      environment: [production]
      country: [nl, de, us]
      plan: [free, premium]
    # make use of the matrix values everywhere
    description: At ${{ at }}%, in ${{ country }} against ${{ plan }}
    environment: ${{ environment }}
    at: ${{ at }}
    context:
      country: ${{ country }}
      plan: ${{ plan }}
    # match expectations as usual
    expectedToBeEnabled: true

This runs the assertion against every combination of values in the matrix. Placeholders are replaced recursively inside nested objects and arrays, including context, sticky values, defaults, expected values, detailed expected evaluations, and child assertions. When a placeholder is the complete value, its original type is preserved, so booleans, numbers, arrays, and objects do not become strings.

Every matrix must define at least one key, and every key must contain at least one value. Every ${{ name }} placeholder must refer to a key in that assertion's matrix. An environment or target selected through a matrix must use a complete placeholder, and every expanded value must name an environment or Target that exists in the project.

Assertions must not be empty. A detailed expectedEvaluation must also contain at least one field. These checks prevent a matrix from producing no cases or an expectation from passing without testing anything.

To inspect the expanded assertions as JSON, use npx featurevisor list --tests --apply-matrix --json. The result contains one final assertion for each matrix combination and omits the original matrix property.

You can do the same for segment test specs as well:

segment: netherlands # your segment key
assertions:
  - matrix:
      country: [nl]
      city: [amsterdam, rotterdam]
    description: Testing in ${{ city }}, ${{ country }}
    context:
      country: ${{ country }}
      city: ${{ city }}
    expectedToMatch: true

This helps us cover more scenarios by having to write less code in our specs.

Promotable test specs and assertions#

In a project that uses sets, a feature, segment, or global variable test spec can protect its existing destination version from later promotions by setting promotable: false at the top level:

feature: foo
promotable: false
assertions:
  - description: Keep the production rollout disabled
    at: 50
    context:
      userId: production-user
    expectedToBeEnabled: false

If the destination test spec exists, it remains unchanged when either the source or destination spec has this field. A missing destination spec is still created and retains promotable: false. Matching is based on the test spec's file path.

You can also protect an individual assertion by giving every assertion in the spec a unique, stable key:

feature: foo
assertions:
  - key: production-rollout
    promotable: false
    description: Keep the production rollout disabled
    at: 50
    context:
      userId: production-user
    expectedToBeEnabled: false
  - key: general-rollout
    description: Test the general rollout
    at: 80
    context:
      userId: general-user
    expectedToBeEnabled: true

A source assertion with promotable: false is omitted. A protected destination assertion is preserved when the source contains an assertion with the same key. If a test spec has any keyed assertion, all of its assertions must have unique keys. Setting promotable on an assertion also requires a key.

When assertion protection is involved, both source and destination specs must use assertion keys. Existing unkeyed specs continue to promote their assertion arrays as a whole.

Matrix cases and child assertions belong to their parent assertion. Set promotable: false on that parent to protect all of them together. They cannot be protected individually.

Assertion keys are also used as stable labels and permalinks in the Catalog. Expanded matrix cases receive labels such as production-rollout.1 and production-rollout.2.

Testing against datafile#

When running tests, Featurevisor CLI will produce a datafile in memory containing your entire project's features. This is handy to make the tests run quickly by default.

But to gain more confidence like a real end user, we may also want to execute individual assertions against target datafiles.

Learn more in:

• Targets

Against a target#

If testing a feature against a particular target, the test spec can be written as follows:

feature: my_feature
assertions:
  - environment: production
    at: 90
    context:
      country: nl
    target: web
    expectedToBeEnabled: true

And run:

$ npx featurevisor test

This makes sure the assertion is run against the datafile for target web in production environment. A Target assertion requires that exact Target datafile. The runner reports a missing datafile instead of falling back to the base environment datafile.

CLI options#

entityType#

If you want to run tests for a specific type of entity, use feature, segment, or variable:

$ npx featurevisor test --entityType=feature
$ npx featurevisor test --entityType=segment
$ npx featurevisor test --entityType=variable

keyPattern#

You can also filter tests by feature, segment, or global variable keys using regex patterns:

$ npx featurevisor test --keyPattern="myKeyHere"

assertionPattern#

If you are writing assertion descriptions, then you can filter them further using regex patterns:

$ npx featurevisor test \
    --keyPattern="myKeyHere" \
    --assertionPattern="text..."

verbose#

For debugging purposes, you can enable verbose mode to see more details of your assertion evaluations

$ npx featurevisor test --verbose

quiet#

You can disable all log output coming from SDK (including errors and warnings):

$ npx featurevisor test --quiet

showDatafile#

For more advanced debugging, you can print the datafile content used by test runner:

$ npx featurevisor test --showDatafile

Printing datafile content for every tested feature or global variable can be very verbose, so we recommend using this option with --keyPattern to filter tests.

onlyFailures#

If you are interested to see only the test specs that fail:

$ npx featurevisor test --onlyFailures

set#

In a project with sets, you can run tests for a single set:

$ npx featurevisor test --set=storefront

Target assertions do not need extra CLI options. The test runner builds target datafiles in memory automatically.

NPM scripts#

If you are using npm scripts for testing your Featurevisor project like this:

{
  "scripts": {
    "test": "featurevisor test"
  }
}

You can then pass your options in CLI after --:

$ npm test -- --keyPattern="myKeyHere"