Smart Regression — run only the tests a change can affect

Smart Regression analyzes the files changed in a commit and tells you
which test cases actually need to run — instead of re-running the whole
suite on every push. It is a Test Impact Analysis built for CI: you
send the changed-file list, Mockarty returns the impacted test set plus
a safety verdict, and your pipeline runs just that subset.

It is deterministic — no AI, no training, no model. The selection is
a direct match between your changed files and each test’s recorded source
location, combined with two always-run safety buckets.

What gets selected

The selected set is the union of up to four buckets:

Risk profiles

Choose how aggressively to narrow the set:

• conservative (default) — all four buckets, including unmappable
  tests. Safest; never skips a test the analysis can’t reason about.
• standard — affected ∪ previously-failed ∪ never-run.
• fast — affected by path only. Smallest set, highest risk; use when
  your tests have complete source references.

Fail-safe

If the change can’t be reasoned about — no changed files, no source-ref
coverage, or no test cases — the verdict is inconclusive and you
should run the full plan. Smart Regression never silently skips a
test on uncertainty.

Use it from CI (CLI)

The CLI collects the diff and queries the server:

mockarty-cli util ci impact \
  --server https://mockarty.example.com \
  --base origin/main --head HEAD \
  --namespace my-team \
  --risk conservative \
  --allure-out testplan.json

Output:

✓ 12 test case(s) selected from 7 changed file(s)
  risk=conservative buckets: affected-by-path=4 previously-failed=2 never-run=5 unmappable=1 (union=12)
  wrote Allure testplan (12 tests) → testplan.json

--allure-out writes a standard test-plan file. Point your test runner
at it (ALLURE_TESTPLAN_PATH=testplan.json) and it executes only the
selected subset. When the verdict is inconclusive, the file is not
written — your pipeline should run the full plan instead.

The Mockarty Go, Python and Java SDKs read this file out of the box, and
refuse to run instead of silently falling back to a full run when it is
empty or broken. See
Selective runs (test plans)
for the exact behaviour and the selector shapes each language accepts.

Assemble a Mockarty Test Plan

Instead of exporting a file for an external runner, you can have Mockarty
build a Test Plan over exactly the selected cases and (optionally) run it:

mockarty-cli util ci impact \
  --base origin/main --head HEAD --namespace my-team \
  --assemble-plan \
  --plan-name "PR #482 regression" \
  --run

--assemble-plan creates a plan and prints its id (advisory — a human can
review and run it from the UI). Add --run to start an orchestrated run
immediately and get the run id. When the verdict is inconclusive the plan is
not assembled (run the full plan instead).

GitHub Actions

name: smart-regression
on: pull_request
jobs:
  impact:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0          # full history so the base ref is available
      - name: Compute impacted tests
        env:
          MOCKARTY_SERVER: ${{ secrets.MOCKARTY_SERVER }}
          MOCKARTY_API_TOKEN: ${{ secrets.MOCKARTY_API_TOKEN }}
        run: |
          mockarty-cli util ci impact \
            --base "origin/${{ github.base_ref }}" --head HEAD \
            --namespace my-team --risk conservative \
            --allure-out testplan.json
      - name: Run selected tests
        env:
          ALLURE_TESTPLAN_PATH: testplan.json
        run: ./run-tests.sh    # your existing test command

Consumer-impact (did the change break a downstream service?)

If the change touches a service you publish a contract for, pass its
registry entry id(s) and Smart Regression also checks whether the change
would break any consumer of that service:

mockarty-cli util ci impact --base origin/main --head HEAD \
  --provider-entry <registry-entry-id> \
  --fail-on-consumer-break

With --fail-on-consumer-break the command exits non-zero when a
consumer would break, gating the pipeline. Without it the check is
advisory (reported, not enforced).

API

POST /api/v1/ci/impact?namespace=my-team
{
  "changedFiles": ["src/api/user.go", "src/login.py"],
  "base": "origin/main", "head": "HEAD",
  "risk": "conservative",
  "providerRegistryEntryIds": ["<id>"],
  "assemblePlan": true,
  "runAfterAssemble": false,
  "planName": "PR #482 regression"
}

The response carries selectedCaseIds, selectedCases (id + selector),
per-bucket buckets, the inconclusive verdict + reason, and an
optional consumerImpact block. When assemblePlan is set (and the verdict
is conclusive), the response also carries assembledPlanId (and
assembledRunId when runAfterAssemble is set).

For AI agents

The same analysis is available to the agent network as the
analyze_regression_impact tool (Test Plans group). An autonomous agent
reasoning about “what should I re-test after this change?” gets the
identical selection, fail-safe verdict, and consumer-impact a human or CI
pipeline gets.

Notes

• Selection is scoped to your namespace; you never see another team’s tests.
• Accuracy depends on how complete your tests’ source references are —
  tests discovered from your code or imported with a source location map
  precisely; tests without one fall into the safe “unmappable” bucket
  under the conservative profile.