Test Plans in CI/CD

This guide shows how to wire Mockarty test plans into the three most common continuous integration systems: GitHub Actions, GitLab CI and Jenkins. The goal is a single CI step that triggers a plan, streams progress into the build log, fails the pipeline on regressions and uploads the Allure report as a build artefact.

About URLs in examples: all examples use https://mockarty.company.com as a placeholder for your Mockarty admin node. Replace it with the actual address of your instance. See Tips & Useful Features for details.

Related pages: Test Plans · Test Plans API Cookbook · CLI User Guide

What you get

• One exit code — mockarty-cli testplan run --wait returns a distinct code per terminal state.
• Five report formats — mockarty-cli testplan report <runID> --plan <plan> --format <fmt> returns Allure JSON, Allure ZIP, JUnit XML, Markdown or Unified JSON from the same run. Pick the format your CI expects; no re-run needed to switch.
• One command for multi-protocol verification — functional, fuzz, chaos, load and contract checks bundled together.

Prerequisites

1. Running Mockarty admin node reachable from the CI runners.
2. An API token with at least Developer rights in the target namespace. Generate it in Settings → API Tokens or via POST /api/v1/auth/tokens.
3. The mockarty-cli binary available on the runner (installed via OS package, Docker image, or downloaded from the releases page). In examples we use Linux x86-64 archives; adjust the URL for your runner OS/arch.
4. A pre-created plan in the target namespace. If your pipeline needs the plan assembled dynamically, use the ad-hoc endpoint (see Ad-hoc master runs).

Exit code contract

mockarty-cli testplan run --wait --timeout <d> blocks until the run reaches a terminal state and then exits with:

Treat non-zero as “the pipeline must fail”. Codes 2 and 3 typically warrant alerting the on-call (they indicate human intervention or infrastructure pressure), not just a failing test.

Authentication

All CLI calls authenticate with the MOCKARTY_API_TOKEN environment variable. Store it as an encrypted secret in your CI system — never commit tokens to the repository.

export MOCKARTY_URL=https://mockarty.company.com
export MOCKARTY_API_TOKEN=mk_7_...
export MOCKARTY_NAMESPACE=default

MOCKARTY_NAMESPACE sets the default namespace so you don’t have to pass --namespace on every subcommand.

GitHub Actions

Minimal workflow

name: mockarty-regression

on:
  push:
    branches: [main]
  pull_request:

jobs:
  regression:
    runs-on: ubuntu-latest
    env:
      MOCKARTY_URL: https://mockarty.company.com
      MOCKARTY_API_TOKEN: ${{ secrets.MOCKARTY_API_TOKEN }}
      MOCKARTY_NAMESPACE: default
    steps:
      - name: Install mockarty-cli
        run: |
          curl -sSL -o /tmp/cli.tar.gz https://mockarty.company.com/downloads/mockarty-cli_linux_amd64.tar.gz
          tar -xzf /tmp/cli.tar.gz -C /usr/local/bin mockarty-cli
          mockarty-cli version

      - name: Trigger plan and wait
        id: run
        run: |
          RUN_ID=$(mockarty-cli testplan run '#42' \
              --wait --timeout 15m \
              --output json | jq -r .run_id)
          echo "run_id=$RUN_ID" >> "$GITHUB_OUTPUT"

      - name: Download Allure archive
        if: always() && steps.run.outputs.run_id != ''
        run: |
          mockarty-cli testplan report "${{ steps.run.outputs.run_id }}" \
              --plan '#42' --zip ./allure.zip

      - name: Publish Allure artefact
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: allure-report
          path: allure.zip
          retention-days: 30

The if: always() guards ensure the Allure archive is captured even when the run fails, so developers can inspect the failing items. The steps.run.outputs.run_id hop survives the failure because the step sets the output before the exit code propagates.

Fan-in pattern

For a monorepo with several plans, trigger them as a matrix:

strategy:
  fail-fast: false
  matrix:
    plan: ['#42', '#43', '#44']
steps:
  - name: Trigger
    run: mockarty-cli testplan run "${{ matrix.plan }}" --wait --timeout 10m

fail-fast: false lets every plan finish so you can see every regression in one PR review.

GitLab CI

stages:
  - test

mockarty:
  stage: test
  image: curlimages/curl:8.6.0
  variables:
    MOCKARTY_URL: "https://mockarty.company.com"
    MOCKARTY_NAMESPACE: "default"
  before_script:
    - curl -sSL -o /tmp/cli.tar.gz "$MOCKARTY_URL/downloads/mockarty-cli_linux_amd64.tar.gz"
    - tar -xzf /tmp/cli.tar.gz -C /usr/local/bin mockarty-cli
  script:
    - RUN_JSON=$(mockarty-cli testplan run '#42' --wait --timeout 15m --output json)
    - echo "$RUN_JSON" | tee run.json
    - RUN_ID=$(echo "$RUN_JSON" | sed -n 's/.*"run_id":"\([^"]*\).*/\1/p')
    - mockarty-cli testplan report "$RUN_ID" --plan '#42' --zip allure.zip
    - mockarty-cli testplan report "$RUN_ID" --plan '#42' --format junit -o report.junit.xml
  artifacts:
    when: always
    paths:
      - allure.zip
      - run.json
      - report.junit.xml
    expire_in: 30 days
    reports:
      # GitLab surfaces this file natively in the merge-request widget.
      junit: report.junit.xml
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == "main"

Store MOCKARTY_API_TOKEN as a masked/protected CI/CD variable in Settings → CI/CD → Variables.

Multi-project pipelines

If the plan verifies an upstream service, trigger Mockarty from a downstream needs: stage with MOCKARTY_URL pinned per environment:

mockarty:staging:
  extends: .mockarty-template
  variables:
    MOCKARTY_URL: "https://mockarty-staging.company.com"
  environment: staging

Jenkins (declarative pipeline)

pipeline {
    agent any
    environment {
        MOCKARTY_URL       = 'https://mockarty.company.com'
        MOCKARTY_NAMESPACE = 'default'
    }
    stages {
        stage('Install CLI') {
            steps {
                sh '''
                    curl -sSL -o /tmp/cli.tar.gz "$MOCKARTY_URL/downloads/mockarty-cli_linux_amd64.tar.gz"
                    tar -xzf /tmp/cli.tar.gz -C /usr/local/bin mockarty-cli
                '''
            }
        }
        stage('Run plan') {
            steps {
                withCredentials([string(credentialsId: 'MOCKARTY_API_TOKEN',
                                        variable: 'MOCKARTY_API_TOKEN')]) {
                    script {
                        env.RUN_ID = sh(returnStdout: true, script: '''
                            mockarty-cli testplan run '#42' \
                                --wait --timeout 15m \
                                --output json | jq -r .run_id
                        ''').trim()
                    }
                }
            }
        }
    }
    post {
        always {
            withCredentials([string(credentialsId: 'MOCKARTY_API_TOKEN',
                                    variable: 'MOCKARTY_API_TOKEN')]) {
                sh '''
                    if [ -n "$RUN_ID" ]; then
                      mockarty-cli testplan report "$RUN_ID" --plan '#42' --zip allure.zip
                    fi
                '''
            }
            archiveArtifacts artifacts: 'allure.zip', allowEmptyArchive: true, fingerprint: true
            // With the Allure Jenkins plugin:
            // allure includeProperties: false, jdk: '', results: [[path: 'allure']]
        }
    }
}

Store the API token with Jenkins Credentials (ID MOCKARTY_API_TOKEN, type “Secret text”) so it is masked in the build log.

Dynamic ad-hoc runs

If the plan contents depend on what the pull request changed, assemble the items on the fly and hit the ad-hoc endpoint instead of pre-registering a plan. The response already carries run_id; poll it to completion with testplan run-status (or the SDK WaitForRun):

RUN_ID=$(curl -s -X POST \
    "$MOCKARTY_URL/api/v1/namespaces/$MOCKARTY_NAMESPACE/test-runs/ad-hoc" \
    -H "X-API-Key: $MOCKARTY_API_TOKEN" \
    -H "Content-Type: application/json" \
    -d @adhoc.json | jq -r .run_id)

# poll the run id until it reaches a terminal status:
until mockarty-cli testplan run-status "$RUN_ID" | grep -qiE 'completed|failed|cancelled'; do sleep 5; done

adhoc.json is a file your CI script assembles from the diff: each changed component maps to one item (functional, contract, fuzz, …) pointing at the right resource UUID. See the API cookbook for a full payload.

Uploading to Allure TestOps

The ZIP Mockarty returns is an Allure results archive (result-*.json + attachments). Feed it to any Allure tool:

# Local HTML report
allure generate allure.zip -o allure-html --clean

# Allure TestOps CLI
allurectl upload allure.zip \
    --project-id $ALLURE_PROJECT_ID \
    --launch-name "mockarty #42"

allurectl is the reference client from the Allure team; use the same pattern for Report Portal, XRay or any other tool that consumes Allure results.

Picking the right report format

Every Mockarty run can be exported in six formats without re-running the plan. Pick what your pipeline already knows how to consume:

You can also grab the raw bytes straight from the REST endpoints — see the API cookbook.

Common pitfalls

• Timeout too low — a flaky external dependency can push p95 over 10 minutes. Set --timeout to at least 1.5x the historical p95; budget longer for load/fuzz items.
• CLI downloaded on every build — cache /usr/local/bin/mockarty-cli between runs or bake it into your CI base image.
• Ephemeral runners lose the artefact — always upload Allure after triggering and before the job cleans up. Use if: always() (GitHub) / when: always (GitLab) / post { always {...} } (Jenkins).
• Token leaks — rotate MOCKARTY_API_TOKEN on schedule and scope it to the narrowest namespace and role that still permits testplan run.
• Wrong namespace — MOCKARTY_NAMESPACE determines where the plan lookup happens; mismatches surface as 404 from the trigger endpoint.
• Cancelled runs (exit code 2) — someone hit Cancel in the UI or another CI job is stepping on the same plan. Give each environment its own plan or serialise the stage with a mutex.

Where to go next

• Test Plans concepts — plan structure, schedules, webhooks and reports explained in depth.
• Test Plans API Cookbook — copy-pasteable curl and SDK snippets for every endpoint.
• CLI User Guide — every mockarty-cli testplan subcommand with flags.