Chaos Engineering

Status: BETA – This feature is in active development. API and behavior may change.

Mockarty provides a built-in chaos engineering platform for Kubernetes. Run controlled fault injection experiments to test system resilience. Supports 12 fault types including pod kills, network partitions/delay/loss, resource stress, DNS disruption, IO chaos, and time chaos.

Key capabilities:

• 8 fault injection types (pod kill, network partition, deployment restart, scale, node drain, resource stress, DNS disruption)
• Safety guardrails (namespace deny lists, blast radius limits, auto-rollback)
• Steady-state validation (HTTP probes, Prometheus metrics, pod counts)
• Scheduling and automation (cron-based recurring experiments)
• Reports (HTML, JSON)
• 6 pre-built experiment presets (Chaos Monkey, Network Partition, Scale to Zero, etc.)
• Web UI with cluster topology visualization
• Integration with contract testing for validating resilience alongside spec compliance

Requirements:

• PRO license (chaos engineering is a PRO feature)
• Kubernetes cluster (1.20+)
• kubectl access or kubeconfig file

About URLs in examples: All examples use localhost:5770 as the default Mockarty address. If your instance runs on a remote server, replace localhost:5770 with its actual address (e.g. https://mockarty.company.com or http://192.168.1.50:5770). See Tips & Useful Features for details.

Quick Start

Step 1. Connect to a cluster

cURL

curl -X POST http://localhost:5770/api/v1/chaos/profiles \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -d '{"name": "staging", "namespace": "default", "kubeconfig": "/path/to/.kube/config"}'

CLI

mockarty-cli chaos cluster add --name staging --kubeconfig ~/.kube/config
mockarty-cli chaos cluster test

Go

// go get github.com/mockarty/mockarty-go
client := mockarty.NewClient("http://localhost:5770", "YOUR_TOKEN")

profile, err := client.Chaos().CreateProfile(ctx, &mockarty.ChaosProfile{
    Name:       "staging",
    Namespace:  "default",
    Kubeconfig: "/path/to/.kube/config",
})

// Test connectivity
result, err := client.Chaos().TestProfile(ctx, profile.ID)
fmt.Println("Connected:", result)

Python

# pip install mockarty
from mockarty import MockartyClient

client = MockartyClient("http://localhost:5770", token="YOUR_TOKEN")

profile = client.chaos.create_profile({
    "name": "staging",
    "namespace": "default",
    "kubeconfig": "/path/to/.kube/config",
})

# Test connectivity
result = client.chaos.test_profile(profile["id"])
print("Connected:", result)

Java

// Gradle: implementation "ru.mockarty:mockarty-java:1.0.0"
MockartyClient client = new MockartyClient("http://localhost:5770", "YOUR_TOKEN");

ChaosProfile profile = new ChaosProfile();
profile.setName("staging");
profile.setNamespace("default");
profile.setKubeconfig("/path/to/.kube/config");

ChaosProfile created = client.chaos().createProfile(profile);

// Test connectivity
Map<String, Object> result = client.chaos().testProfile(created.getId());
System.out.println("Connected: " + result);

Step 2. Run a preset experiment

cURL

# Create experiment from preset, then run it
EXPERIMENT_ID=$(curl -s -X POST http://localhost:5770/api/v1/chaos/experiments \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -d '{
    "name": "frontend-chaos-monkey",
    "namespace": "my-app",
    "presetName": "chaos-monkey",
    "durationSec": 300,
    "target": {"selector": {"app": "frontend"}, "namespace": "my-app"}
  }' | jq -r '.id')

curl -X POST "http://localhost:5770/api/v1/chaos/experiments/$EXPERIMENT_ID/run" \
  -H "Authorization: Bearer YOUR_TOKEN"

CLI

mockarty-cli chaos preset run --preset chaos-monkey \
  --k8s-namespace my-app --selector app=frontend --duration 5m

Go

// List available presets
presets, err := client.Chaos().ListPresets(ctx)

// Create and run an experiment from a preset definition
exp, err := client.Chaos().Create(ctx, &mockarty.ChaosExperiment{
    Name:        "frontend-chaos-monkey",
    Namespace:   "my-app",
    PresetName:  "chaos-monkey",
    DurationSec: 300,
    Target: mockarty.TargetConfig{
        Selector:  map[string]string{"app": "frontend"},
        Namespace: "my-app",
    },
})
err = client.Chaos().Run(ctx, exp.ID)

Python

# List available presets
presets = client.chaos.list_presets()

# Create and run an experiment from a preset definition
exp = client.chaos.create({
    "name": "frontend-chaos-monkey",
    "namespace": "my-app",
    "presetName": "chaos-monkey",
    "durationSec": 300,
    "target": {
        "selector": {"app": "frontend"},
        "namespace": "my-app",
    },
})
client.chaos.run(exp["id"])

Java

// List available presets
List<ChaosPreset> presets = client.chaos().listPresets();

// Create and run an experiment from a preset definition
ChaosExperiment exp = new ChaosExperiment();
exp.setName("frontend-chaos-monkey");
exp.setNamespace("my-app");
exp.setPresetName("chaos-monkey");
exp.setDurationSec(300);

ChaosExperiment created = client.chaos().create(exp);
client.chaos().run(created.getId());

Step 3. View results

cURL

curl http://localhost:5770/api/v1/chaos/experiments/EXPERIMENT_ID/report \
  -H "Authorization: Bearer YOUR_TOKEN"

CLI

mockarty-cli chaos get EXPERIMENT_ID
mockarty-cli chaos report EXPERIMENT_ID --format html --output report.html

Go

report, err := client.Chaos().GetReport(ctx, "EXPERIMENT_ID")
fmt.Println("Status:", report.Status)

// Download HTML report
htmlBytes, err := client.Chaos().DownloadReport(ctx, "EXPERIMENT_ID", "html")
os.WriteFile("report.html", htmlBytes, 0644)

Python

report = client.chaos.get_report("EXPERIMENT_ID")
print("Status:", report["status"])

# Download HTML report
html_bytes = client.chaos.download_report("EXPERIMENT_ID", format="html")
with open("report.html", "wb") as f:
    f.write(html_bytes)

Java

Map<String, Object> report = client.chaos().getReport("EXPERIMENT_ID");
System.out.println("Status: " + report.get("status"));

// Download HTML report
byte[] htmlBytes = client.chaos().downloadReport("EXPERIMENT_ID", "html");
Files.write(Path.of("report.html"), htmlBytes);

Cluster Setup

Adding a Cluster Profile

cURL

curl -X POST http://localhost:5770/api/v1/chaos/profiles \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -d '{
    "name": "production",
    "namespace": "default",
    "kubeconfig": "/home/user/.kube/config",
    "context": "prod-context"
  }'

CLI

mockarty-cli chaos cluster add --name production --kubeconfig ~/.kube/config

# With context and namespace restrictions
mockarty-cli chaos cluster add --name production \
  --kubeconfig ~/.kube/config --context prod-context \
  --allowed-namespaces app,test --denied-namespaces kube-system,monitoring

Go

profile, err := client.Chaos().CreateProfile(ctx, &mockarty.ChaosProfile{
    Name:       "production",
    Namespace:  "default",
    Kubeconfig: "/home/user/.kube/config",
    Context:    "prod-context",
})
fmt.Println("Created profile:", profile.ID)

Python

profile = client.chaos.create_profile({
    "name": "production",
    "namespace": "default",
    "kubeconfig": "/home/user/.kube/config",
    "context": "prod-context",
})
print("Created profile:", profile["id"])

Java

ChaosProfile profile = new ChaosProfile();
profile.setName("production");
profile.setNamespace("default");
profile.setKubeconfig("/home/user/.kube/config");
profile.setContext("prod-context");

ChaosProfile created = client.chaos().createProfile(profile);
System.out.println("Created profile: " + created.getId());

Testing Connectivity

cURL

curl -X POST http://localhost:5770/api/v1/chaos/profiles/PROFILE_ID/test \
  -H "Authorization: Bearer YOUR_TOKEN"

CLI

mockarty-cli chaos cluster test

Go

result, err := client.Chaos().TestProfile(ctx, "PROFILE_ID")
fmt.Println("Connection status:", result)

Python

result = client.chaos.test_profile("PROFILE_ID")
print("Connection status:", result)

Java

Map<String, Object> result = client.chaos().testProfile("PROFILE_ID");
System.out.println("Connection status: " + result);

Viewing Cluster Info

mockarty-cli chaos cluster info    # Namespaces, nodes, capabilities
mockarty-cli chaos cluster list    # All configured clusters
mockarty-cli chaos cluster use staging  # Switch active cluster

Fault Types

Target modes: one (specific pod), random_one (default), all, fixed_percent.

Running Experiments

From CLI Flags

mockarty-cli chaos run \
  --type pod_kill_random \
  --k8s-namespace my-app \
  --selector app=frontend \
  --duration 5m \
  --interval 30s \
  --auto-rollback

mockarty-cli chaos run \
  --type network_partition \
  --deployment my-service \
  --k8s-namespace my-app \
  --duration 2m \
  --auto-rollback

From YAML File

# experiment.yaml
name: frontend-resilience-test
namespace: my-app
durationSec: 600

faults:
  - type: pod_kill_random
    intervalSec: 60

target:
  mode: random_one
  selector:
    app: frontend
  namespace: my-app

safety:
  autoRollback: true
  haltOnSteadyFail: true
  denyNamespaces: [kube-system, monitoring]
  maxPodsAffected: 3
  blastRadiusPercent: 30

steadyState:
  checks:
    - name: frontend-health
      type: http
      endpoint: "http://frontend.my-app.svc:8080/health"
      method: GET
      expected: 200
      intervalSec: 10
    - name: min-frontend-pods
      type: pod_count
      query: "app=frontend"
      expected: 2

cURL

curl -X POST http://localhost:5770/api/v1/chaos/experiments \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -d @experiment.json

# Then run it
curl -X POST http://localhost:5770/api/v1/chaos/experiments/EXPERIMENT_ID/run \
  -H "Authorization: Bearer YOUR_TOKEN"

CLI

mockarty-cli chaos run -f experiment.yaml

Go

exp, err := client.Chaos().Create(ctx, &mockarty.ChaosExperiment{
    Name:        "frontend-resilience-test",
    Namespace:   "my-app",
    DurationSec: 600,
    Faults: []mockarty.FaultConfig{
        {Type: "pod_kill_random", IntervalSec: 60},
    },
    Target: mockarty.TargetConfig{
        Mode:      "random_one",
        Selector:  map[string]string{"app": "frontend"},
        Namespace: "my-app",
    },
    Safety: mockarty.SafetyConfig{
        AutoRollback:      true,
        HaltOnSteadyFail:  true,
        DenyNamespaces:    []string{"kube-system", "monitoring"},
        MaxPodsAffected:   3,
        BlastRadiusPercent: 30,
    },
})

// Run the experiment
err = client.Chaos().Run(ctx, exp.ID)

Python

exp = client.chaos.create({
    "name": "frontend-resilience-test",
    "namespace": "my-app",
    "durationSec": 600,
    "faults": [
        {"type": "pod_kill_random", "intervalSec": 60},
    ],
    "target": {
        "mode": "random_one",
        "selector": {"app": "frontend"},
        "namespace": "my-app",
    },
    "safety": {
        "autoRollback": True,
        "haltOnSteadyFail": True,
        "denyNamespaces": ["kube-system", "monitoring"],
        "maxPodsAffected": 3,
        "blastRadiusPercent": 30,
    },
})

# Run the experiment
client.chaos.run(exp["id"])

Java

ChaosExperiment exp = new ChaosExperiment();
exp.setName("frontend-resilience-test");
exp.setNamespace("my-app");
exp.setDurationSec(600);

ChaosExperiment created = client.chaos().create(exp);

// Run the experiment
client.chaos().run(created.getId());

Using Presets

Mockarty ships with 6 pre-built experiment presets:

cURL

# Create an experiment from a preset and run it
EXPERIMENT_ID=$(curl -s -X POST http://localhost:5770/api/v1/chaos/experiments \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -d '{
    "name": "chaos-monkey-test",
    "namespace": "my-app",
    "presetName": "chaos-monkey",
    "durationSec": 300,
    "target": {"mode": "random_one", "selector": {"app": "web"}, "namespace": "my-app"}
  }' | jq -r '.id')

curl -X POST "http://localhost:5770/api/v1/chaos/experiments/$EXPERIMENT_ID/run" \
  -H "Authorization: Bearer YOUR_TOKEN"

CLI

# List all presets
mockarty-cli chaos preset list

# Run a preset
mockarty-cli chaos preset run --preset chaos-monkey \
  --k8s-namespace my-app --selector app=web --duration 5m

mockarty-cli chaos preset run --preset network-partition \
  --k8s-namespace my-app --selector app=api --duration 3m

mockarty-cli chaos preset run --preset dns-disruption \
  --k8s-namespace my-app --selector app=gateway --duration 2m

Go

// List all presets
presets, err := client.Chaos().ListPresets(ctx)
for _, p := range presets {
    fmt.Printf("Preset: %s (%s)\n", p.Name, p.Description)
}

// Create experiment from a preset and run it
exp, err := client.Chaos().Create(ctx, &mockarty.ChaosExperiment{
    Name:        "chaos-monkey-test",
    Namespace:   "my-app",
    PresetName:  "chaos-monkey",
    DurationSec: 300,
    Target: mockarty.TargetConfig{
        Selector:  map[string]string{"app": "web"},
        Namespace: "my-app",
    },
})
err = client.Chaos().Run(ctx, exp.ID)

Python

# List all presets
presets = client.chaos.list_presets()
for p in presets:
    print(f"Preset: {p['name']} ({p.get('description', '')})")

# Create experiment from a preset and run it
exp = client.chaos.create({
    "name": "chaos-monkey-test",
    "namespace": "my-app",
    "presetName": "chaos-monkey",
    "durationSec": 300,
    "target": {
        "selector": {"app": "web"},
        "namespace": "my-app",
    },
})
client.chaos.run(exp["id"])

Java

// List all presets
List<ChaosPreset> presets = client.chaos().listPresets();
for (ChaosPreset p : presets) {
    System.out.printf("Preset: %s (%s)%n", p.getName(), p.getDescription());
}

// Create experiment from a preset and run it
ChaosExperiment exp = new ChaosExperiment();
exp.setName("chaos-monkey-test");
exp.setNamespace("my-app");
exp.setPresetName("chaos-monkey");
exp.setDurationSec(300);

ChaosExperiment created = client.chaos().create(exp);
client.chaos().run(created.getId());

Safety & Guardrails

Namespace Protection

safety:
  denyNamespaces: [kube-system, monitoring, cert-manager, istio-system]
  allowNamespaces: [staging-app, test-env]  # Whitelist mode

Blast Radius Limits

safety:
  maxPodsAffected: 5
  minReplicasAlive: 2
  blastRadiusPercent: 30

Auto-Rollback

Automatically reverts all changes if steady-state checks fail:

safety:
  autoRollback: true       # Revert on failure
  haltOnSteadyFail: true   # Stop if steady-state check fails

CLI flag: --auto-rollback (enabled by default).

Manual Approval

For high-risk experiments, require explicit approval before execution:

safety:
  requireApproval: true

The experiment enters pending state and waits for approval via API or UI.

Steady-State Validation

Verify system behavior before, during, and after experiments:

steadyState:
  checks:
    # HTTP endpoint check
    - name: health-check
      type: http
      endpoint: "http://my-service.my-app.svc:8080/health"
      method: GET
      expected: 200
      intervalSec: 10
      timeoutSec: 5

    # Prometheus metric query
    - name: error-rate
      type: prometheus
      query: "rate(http_requests_total{job='frontend',code=~'5..'}[1m])"
      expected: 0.01
      tolerance: 0.005
      intervalSec: 30

    # Pod count check
    - name: min-pods
      type: pod_count
      query: "app=frontend"
      expected: 2
      tolerance: 0

Monitoring & Reports

Viewing Experiment Status

cURL

curl http://localhost:5770/api/v1/chaos/experiments/EXPERIMENT_ID \
  -H "Authorization: Bearer YOUR_TOKEN"

curl "http://localhost:5770/api/v1/chaos/experiments?status=active&limit=10" \
  -H "Authorization: Bearer YOUR_TOKEN"

CLI

mockarty-cli chaos get EXPERIMENT_ID
mockarty-cli chaos list --status active --limit 10
mockarty-cli chaos abort EXPERIMENT_ID

Go

// Get a single experiment
exp, err := client.Chaos().Get(ctx, "EXPERIMENT_ID")
fmt.Println("Status:", exp.Status)

// List active experiments
experiments, err := client.Chaos().List(ctx, &mockarty.ChaosListOptions{
    Status: "active",
    Limit:  10,
})

// Abort a running experiment
err = client.Chaos().Abort(ctx, "EXPERIMENT_ID")

Python

# Get a single experiment
exp = client.chaos.get("EXPERIMENT_ID")
print("Status:", exp["status"])

# List active experiments
experiments = client.chaos.list(status="active", limit=10)

# Abort a running experiment
client.chaos.abort("EXPERIMENT_ID")

Java

// Get a single experiment
ChaosExperiment exp = client.chaos().get("EXPERIMENT_ID");
System.out.println("Status: " + exp.getStatus());

// List active experiments
List<ChaosExperiment> experiments = client.chaos().list(null, "active", 10, 0);

// Abort a running experiment
client.chaos().abort("EXPERIMENT_ID");

Getting Reports

cURL

# JSON report
curl http://localhost:5770/api/v1/chaos/experiments/EXPERIMENT_ID/report \
  -H "Authorization: Bearer YOUR_TOKEN"

# Download in specific format: html, json
curl -o report.html \
  "http://localhost:5770/api/v1/chaos/experiments/EXPERIMENT_ID/report/download?format=html" \
  -H "Authorization: Bearer YOUR_TOKEN"

CLI

mockarty-cli chaos report EXPERIMENT_ID
mockarty-cli chaos report EXPERIMENT_ID --format html --output report.html
mockarty-cli chaos report EXPERIMENT_ID --format json --output report.json

Go

// Get JSON report
report, err := client.Chaos().GetReport(ctx, "EXPERIMENT_ID")

// Get metrics and events
metrics, err := client.Chaos().GetMetrics(ctx, "EXPERIMENT_ID")
events, err := client.Chaos().GetEvents(ctx, "EXPERIMENT_ID")

// Download report in a specific format: html, json
htmlBytes, err := client.Chaos().DownloadReport(ctx, "EXPERIMENT_ID", "html")
os.WriteFile("report.html", htmlBytes, 0644)

Python

# Get JSON report
report = client.chaos.get_report("EXPERIMENT_ID")

# Get metrics and events
metrics = client.chaos.get_metrics("EXPERIMENT_ID")
events = client.chaos.get_events("EXPERIMENT_ID")

# Download report in a specific format: html, json
html_bytes = client.chaos.download_report("EXPERIMENT_ID", format="html")
with open("report.html", "wb") as f:
    f.write(html_bytes)

Java

// Get JSON report
Map<String, Object> report = client.chaos().getReport("EXPERIMENT_ID");

// Get metrics and events
Map<String, Object> metrics = client.chaos().getMetrics("EXPERIMENT_ID");
Map<String, Object> events = client.chaos().getEvents("EXPERIMENT_ID");

// Download report in a specific format: html, json
byte[] htmlBytes = client.chaos().downloadReport("EXPERIMENT_ID", "html");
Files.write(Path.of("report.html"), htmlBytes);

Scheduling

Run experiments on a recurring schedule:

name: daily-resilience-check
namespace: staging
schedule:
  cronExpr: "0 3 * * *"
  repeatCount: 0        # 0 = infinite when cron is set
  jitter: 30            # random jitter in seconds added to start time
faults:
  - type: pod_kill_random
    intervalSec: 60
target:
  mode: random_one
  selector: { app: api }
  namespace: staging
durationSec: 300
safety:
  autoRollback: true
  denyNamespaces: [kube-system]

Cluster Inspection

Topology & Resources

# Cluster topology
curl http://localhost:5770/api/v1/chaos/clusters/CLUSTER_ID/topology \
  -H "Authorization: Bearer YOUR_TOKEN"

# Pod operations
curl -X DELETE http://localhost:5770/api/v1/chaos/pods/NAMESPACE/POD_NAME \
  -H "Authorization: Bearer YOUR_TOKEN"
curl http://localhost:5770/api/v1/chaos/pods/NAMESPACE/POD_NAME/logs \
  -H "Authorization: Bearer YOUR_TOKEN"

# Deployment operations
curl -X POST http://localhost:5770/api/v1/chaos/deployments/NAMESPACE/DEPLOY/scale \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -d '{"replicas": 3}'
curl -X POST http://localhost:5770/api/v1/chaos/deployments/NAMESPACE/DEPLOY/restart \
  -H "Authorization: Bearer YOUR_TOKEN"

Chaos Operator

The operator is a Kubernetes deployment that enables advanced experiment types. It creates RBAC resources and runs inside the cluster.

cURL

curl -X POST http://localhost:5770/api/v1/chaos/operator/install \
  -H "Authorization: Bearer YOUR_TOKEN"
curl http://localhost:5770/api/v1/chaos/operator/status \
  -H "Authorization: Bearer YOUR_TOKEN"

CLI

mockarty-cli chaos operator install
mockarty-cli chaos operator install --namespace my-chaos-ns
mockarty-cli chaos operator status
mockarty-cli chaos operator uninstall

Go

// Check operator status
status, err := client.Chaos().GetOperatorStatus(ctx, "my-chaos-ns")
fmt.Println("Operator installed:", status.Installed)

// Generate operator manifest
manifest, err := client.Chaos().GenerateOperatorManifest(ctx, "http://mockarty:5770", "")
fmt.Println(manifest)

Python

# Check operator status
status = client.chaos.get_operator_status(namespace="my-chaos-ns")
print("Operator installed:", status.get("installed"))

Java

// Check operator status
Map<String, Object> status = client.chaos().getOperatorStatus("my-chaos-ns");
System.out.println("Operator installed: " + status.get("installed"));

CI/CD Integration

Example GitHub Actions workflow:

name: Chaos Engineering Tests
on:
  schedule:
    - cron: '0 2 * * 1'
  workflow_dispatch:

jobs:
  chaos-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Install CLI
        run: curl -fsSL https://mockarty.ru/install.sh | sh
      - name: Configure cluster
        run: mockarty-cli chaos cluster add --name ci --kubeconfig ${{ secrets.KUBECONFIG_PATH }}
      - name: Run experiment
        run: |
          mockarty-cli chaos preset run --preset chaos-monkey \
            --k8s-namespace staging --selector app=api --duration 5m
      - name: Get report
        run: |
          sleep 330
          EXPERIMENT_ID=$(mockarty-cli chaos list --status completed --limit 1 -o json | jq -r '.[0].id')
          mockarty-cli chaos report $EXPERIMENT_ID --format html --output chaos-report.html
          mockarty-cli chaos report $EXPERIMENT_ID --format json --output chaos-results.json
      - uses: actions/upload-artifact@v4
        with:
          name: chaos-report
          path: |
            chaos-report.html
            chaos-results.xml

API Reference

Presets

Note: To run a preset, create an experiment with the presetName field set via POST /api/v1/chaos/experiments, then start it with POST /api/v1/chaos/experiments/:id/run.

Experiments

Profiles

Operator

Cluster Inspection

Contract Testing Integration

Chaos engineering and contract testing work together to provide comprehensive reliability assurance:

• Before chaos experiments: Run contract tests to ensure mocks and specs are in sync. This establishes a baseline of correctness before injecting faults.
• During chaos experiments: Use steady-state HTTP checks that point to contract-validated endpoints, ensuring you’re testing against known-good contracts.
• After chaos experiments: Re-run contract tests to verify that no drift was introduced during fault injection or recovery.

This combination catches both functional drift (contract testing) and resilience gaps (chaos engineering), providing end-to-end confidence in your service mesh.

Related Documentation

• API Reference – Full REST API documentation
• CLI User Guide – Command-line tool reference
• SDK Guide – SDK clients (Go, Python, Java)
• Performance Testing – Load testing capabilities
• Contract Testing – Contract testing with Pact