---
name: epok-aws-ecs-otel-traces
description: Send OpenTelemetry traces from an existing ECS service to Epok (AWS ECS on Fargate); use when a user asks to add tracing, APM or distributed traces for ECS, Fargate or AWS container workloads in Epok, or to verify or remove the Epok traces setup.
---

<!-- GENERATED from docs/integrations/manifests/aws-ecs-otel-traces.json (sha256 2b5c6407525bd0e56a2d1928ff3363e9eccc6a3110f96ed2e6f6ae579456f6bd) by scripts/gen_traces_setup_artifacts.py. Do not edit. -->

# AWS ECS on Fargate traces -> Epok with an OpenTelemetry Collector sidecar

Registers a new task definition revision of an existing ECS service that adds an OpenTelemetry Collector sidecar (OTLP in on localhost:4318, OTLP/HTTP out to Epok with x-api-key from an SSM SecureString) and points the app container's OTel SDK at it, then rolls the service. Uninstall rolls back to the previous revision.

- Manifest: `https://epok.cloud/setup/aws-ecs-otel-traces.json` (schema `epok.setup-manifest/1`, version 1.0.0)
- Human guide: `docs/integrations/traces/aws-ecs-fargate.md`
- Estimated time: 8 minutes
- **Not yet verified end to end by the Epok harness.** Treat any failure as possibly a manifest bug: report it, do not improvise.

If `scripts/epok_setup.py` is available, prefer it: it runs these exact steps, masks the keys and writes a transcript:

```bash
python3 scripts/epok_setup.py docs/integrations/manifests/aws-ecs-otel-traces.json --dry-run
python3 scripts/epok_setup.py docs/integrations/manifests/aws-ecs-otel-traces.json --json-transcript epok-setup.json
```

## Stop and ask a human

Stop and ask (do not work around it) when:

- `EPOK_API_KEY` or `EPOK_READ_API_KEY` is not in the environment. **A human creates keys in the Epok UI. Never create, guess, print, or write a key to a file or commit.**
- You are not sure which ECS service the user means, or it serves production traffic and the user has not said this change is fine. Every step changes only that one workload.
- The workload already has another tracer or APM agent (Datadog, New Relic, ADOT, X-Ray SDK). Two tracers on one workload conflict.
- The application code is not instrumented with OpenTelemetry and the runtime has no zero-code option here. Do not edit application code without asking.
- Any step fails with an access-denied error. Never grant IAM roles or policies to yourself.
- Calling the workload (the verify canary) has side effects.
- A failure mode below is marked STOP AND ASK.
- Never run the uninstall section unless the user asked for removal.

## Inputs

- `AWS_REGION` (required): Region of the cluster. Must match `^[a-z]{2}(-[a-z]+)+-[0-9]$`.
- `ECS_CLUSTER` (required): ECS cluster name. How to get it: aws ecs list-clusters Must match `^[A-Za-z0-9_-]{1,255}$`.
- `ECS_SERVICE` (required): ECS service (Fargate or awsvpc network mode) to instrument. How to get it: aws ecs list-services --cluster $ECS_CLUSTER (confirm with the human) Must match `^[A-Za-z0-9_-]{1,255}$`.
- `APP_CONTAINER` (required): Name of the application container in the task definition (it gets the OTEL_* variables). Must match `^[A-Za-z0-9_-]{1,255}$`.
- `OTEL_SERVICE_NAME` (required): service.name for the spans (how the service appears in Epok). How to get it: Ask the human, or use the workload name. Must match `^[A-Za-z0-9._-]{1,128}$`.
- `EPOK_KEY_PARAMETER` (default `/epok/ingest-api-key`): SSM Parameter Store name for the ingest key (SecureString, default aws/ssm KMS key; free standard tier). The sidecar reads it through the task execution role. An existing parameter is reused, not overwritten. Must match `^/[A-Za-z0-9_./-]{1,200}$`.
- `EPOK_KEY_STORE` (default `ssm`): Where the sidecar's key lives. ssm (default, recommended): SSM Parameter Store SecureString EPOK_KEY_PARAMETER, injected with secrets/valueFrom. s3: an ECS environment file (EPOK_API_KEY=<key>) at EPOK_KEY_S3_URI in a private bucket with SSE-S3, injected with environmentFiles; use it only where SSM and Secrets Manager are not available to you. Must match `^(ssm|s3)$`.
- `EPOK_KEY_S3_URI` (default ``): Only for EPOK_KEY_STORE=s3: s3://<existing private bucket>/<path>.env. The bucket must block all public access. How to get it: Ask the human which private bucket holds service configuration. Must match `^(s3://[a-z0-9][a-z0-9.-]{1,61}[a-z0-9]/[A-Za-z0-9_./-]{1,200}[.]env)?$`.
- `EPOK_COLLECTOR_IMAGE` (default `ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector:0.163.2`): Collector image for the sidecar (core distribution: otlp receiver + otlphttp exporter; multi-arch). Mirror it into ECR if tasks cannot reach ghcr.io. Must match `^[a-z0-9.-]+(/[A-Za-z0-9._-]+)+:[A-Za-z0-9._-]+$`.
- `EPOK_RESOURCE_TAGS` (default `managed-by=epok`): Comma-separated key=value tags for the SSM parameter. Must match `^[A-Za-z0-9_.:/=+@-]+=[A-Za-z0-9_.:/=+@ -]*(,[A-Za-z0-9_.:/=+@-]+=[A-Za-z0-9_.:/=+@ -]*){0,9}$`.
- `APP_URL` (default ``): Base URL that reaches the service (load balancer DNS name or a task address), no trailing slash. Needed only by verify. How to get it: aws elbv2 describe-load-balancers --query 'LoadBalancers[].DNSName' (match the service's target group), or ask the human. Must match `^(https?://[A-Za-z0-9.:-]+)?$`.
- `EPOK_VERIFY_PATH` (default `/`): Path verify requests. Pick one your instrumented HTTP server handles (the example app traces /work). Must match `^/[A-Za-z0-9_./?=&-]*$`.
- **`EPOK_API_KEY` — HUMAN-PROVIDED SECRET (ingest scope). Epok ingest-scope API key, dedicated to this workload.** A human creates it: Epok -> Settings -> API Keys -> New key (scope: ingest, name: traces-<workload>). Agents must ask for it, never create or guess it, and never write it to a file or a commit.
- **`EPOK_READ_API_KEY` — HUMAN-PROVIDED SECRET (read scope). Epok API key with read scope, used only by verify to look the canary trace up. May be the same key as EPOK_API_KEY if that key also has read scope.** A human creates it: Epok -> Settings -> API Keys (scope: read). Agents must ask for it.
- `EPOK_INGEST_HOST` (default `https://ingest.epok.cloud`): Epok OTLP/HTTP ingest base URL, no trailing slash. The exporter appends /v1/traces. Change only if Epok support tells you to. Must match `^(https://[A-Za-z0-9.-]+|http://(127\.0\.0\.1|localhost))(:[0-9]{1,5})?$`.
- `EPOK_API_HOST` (default `https://app.epok.cloud`): Epok API base URL used by verify, no trailing slash. Must match `^(https://[A-Za-z0-9.-]+|http://(127\.0\.0\.1|localhost))(:[0-9]{1,5})?$`.
- `EPOK_TENANT_ID` (required): Your Epok tenant (account) id, used in the verify API path. How to get it: Epok -> Settings -> Account (the numeric account id), or ask the human. Must match `^[0-9]{1,10}$`.
- `NONCE` (generated: nonce): Canary token: verify sends one request with W3C traceparent trace id 00000000<NONCE> and looks that trace up in Epok. Generated by the runner if not set (24 lowercase hex). Must match `^[0-9a-f]{24}$`.

Export every input as an environment variable before running the commands below.

## Prerequisites

- `aws` >= 2.15.0 (check: `aws --version`)
- `python3` >= 3.8.0 (check: `python3 --version`)
- `curl` >= 7.68.0 (check: `curl --version`)
- Caller permissions: `ecs:DescribeServices`, `ecs:DescribeTaskDefinition`, `ecs:RegisterTaskDefinition`, `ecs:UpdateService`, `ecs:DeregisterTaskDefinition`, `ecs:TagResource`, `ssm:PutParameter`, `ssm:GetParameter`, `ssm:AddTagsToResource`, `ssm:DeleteParameter`, `(s3 store) s3:PutObject, s3:GetObject, s3:DeleteObject, s3:GetBucketPublicAccessBlock`, `iam:PutRolePolicy`, `iam:DeleteRolePolicy`, `iam:PassRole`. iam:PassRole on the task's execution and task roles (needed to register a revision). The app must already use an OpenTelemetry SDK that reads the standard OTEL_* variables (examples/otel-traces/service shows the minimum); this setup does not change application code.

Run these checks; all must succeed:

```bash
aws sts get-caller-identity --query Account --output text >/dev/null
aws ecs describe-services --region "$AWS_REGION" --cluster "$ECS_CLUSTER" --services "$ECS_SERVICE" --query 'services[?status==`ACTIVE`].serviceName' --output text | grep -q .
```

## Cost

No new compute: the sidecar runs inside the existing tasks (memoryReservation 64 MiB, typically <0.05 vCPU; the task's cpu/memory must have that headroom). One SSM standard SecureString parameter (free). Epok bills the decoded OTLP bytes; the collector sends gzip and Epok meters after decompression. About 1-2 KB per 3-span trace.

- Fargate and other awsvpc tasks share localhost between containers, so the app exports to http://localhost:4318 and only the sidecar holds the key.
- The sidecar is essential=false: if it crashes the app keeps serving, spans are lost (and the collector logs say why).
- The new revision is tagged epok-previous=<old revision ARN>; uninstall uses that tag to roll back.
- AWS Distro for OpenTelemetry (public.ecr.aws/aws-observability/aws-otel-collector) works the same way if your platform team standardises on it; set EPOK_COLLECTOR_IMAGE and keep the config.

## Steps

Run in order. Each step runs as `bash -euo pipefail`. Where a check is given, run it first: exit 0 means the step is already done, so skip it. Stop at the first failing step and use the failure modes below.

### 1. `inspect-service` (actor: agent)

Check the service is ACTIVE, uses awsvpc, contains APP_CONTAINER and has an execution role. Prints EPOK_TASK_FAMILY, EPOK_EXECUTION_ROLE_NAME and EPOK_ACCOUNT_ID.

Idempotent: safe to re-run.

Run:

```bash
python3 -c '
import json, os, subprocess, sys
e = os.environ
def aws(*a):
    return json.loads(subprocess.check_output(["aws", *a, "--region", e["AWS_REGION"], "--output", "json"]))
s = aws("ecs", "describe-services", "--cluster", e["ECS_CLUSTER"], "--services", e["ECS_SERVICE"])["services"]
if not s or s[0]["status"] != "ACTIVE":
    sys.exit("service " + e["ECS_SERVICE"] + " not found or not ACTIVE in cluster " + e["ECS_CLUSTER"])
td = aws("ecs", "describe-task-definition", "--task-definition", s[0]["taskDefinition"])["taskDefinition"]
if td.get("networkMode") != "awsvpc":
    sys.exit("network mode " + str(td.get("networkMode")) + ": the sidecar needs awsvpc (Fargate) so localhost is shared")
names = [c["name"] for c in td["containerDefinitions"]]
if e["APP_CONTAINER"] not in names:
    sys.exit("container " + e["APP_CONTAINER"] + " not in task definition; containers: " + ", ".join(names))
role = td.get("executionRoleArn")
if not role:
    sys.exit("task definition has no executionRoleArn; ECS needs one to read the key parameter")
print("task definition " + td["taskDefinitionArn"] + ", containers " + ", ".join(names))
print("EPOK_TASK_FAMILY=" + td["family"])
print("EPOK_EXECUTION_ROLE_NAME=" + role.split("/")[-1])
print("EPOK_ACCOUNT_ID=" + role.split(":")[4])'
```

The output contains `NAME=value` lines. Set each one as an environment variable (`EPOK_TASK_FAMILY`, `EPOK_EXECUTION_ROLE_NAME`, `EPOK_ACCOUNT_ID`); read the lines, do not `eval` the output.

### 2. `store-key` (actor: agent)

Store the ingest key (ssm: SecureString via JSON on stdin, never on the command line; s3: an env file streamed from stdin with SSE-S3, refused unless the bucket blocks public access). Skipped if the parameter/object exists.

Check (skip the step if this exits 0):

```bash
B="${EPOK_KEY_S3_URI#s3://}"; BUCKET="${B%%/*}"; OBJ="${B#*/}"
case "$EPOK_KEY_STORE" in
  ssm) aws ssm get-parameter --name "$EPOK_KEY_PARAMETER" --region "$AWS_REGION" --query Parameter.Name --output text >/dev/null 2>&1 ;;
  s3) aws s3api head-object --bucket "$BUCKET" --key "$OBJ" --region "$AWS_REGION" >/dev/null 2>&1 ;;
esac
```

Run:

```bash
# AWS CLI v2 (2.17.0 to 2.40.3 tested) rejects --cli-input-json read from a pipe ("Invalid JSON received"). The key goes in through a plain parameter file on stdin instead; the other arguments are not secret.
B="${EPOK_KEY_S3_URI#s3://}"; BUCKET="${B%%/*}"; OBJ="${B#*/}"
case "$EPOK_KEY_STORE" in
  ssm)
    python3 -c '
import json, os, subprocess, sys
e = os.environ
tags = [dict(zip(("Key", "Value"), p.split("=", 1))) for p in e["EPOK_RESOURCE_TAGS"].split(",")]
sys.exit(subprocess.run(["aws", "ssm", "put-parameter", "--region", e["AWS_REGION"], "--name", e["EPOK_KEY_PARAMETER"],
                         "--type", "SecureString", "--tags", json.dumps(tags), "--value", "file:///dev/stdin",
                         "--query", "Version", "--output", "text"], input=e["EPOK_API_KEY"], text=True).returncode)
'
    echo "stored the key in SSM SecureString $EPOK_KEY_PARAMETER (default aws/ssm KMS key)" ;;
  s3)
    test -n "$OBJ" -a "$OBJ" != "$B" || { echo "set EPOK_KEY_S3_URI=s3://<bucket>/<path>.env" >&2; exit 1; }
    aws s3api get-public-access-block --bucket "$BUCKET" --query 'PublicAccessBlockConfiguration.[BlockPublicAcls,BlockPublicPolicy,IgnorePublicAcls,RestrictPublicBuckets]' --output text | grep -qv False \
      || { echo "bucket $BUCKET does not block public access; stop and ask" >&2; exit 1; }
    python3 -c 'import os; print("EPOK_API_KEY=" + os.environ["EPOK_API_KEY"])' | aws s3 cp - "$EPOK_KEY_S3_URI" --sse AES256 --only-show-errors
    echo "stored the key as an ECS environment file at $EPOK_KEY_S3_URI (SSE-S3)" ;;
  *) echo "EPOK_KEY_STORE must be ssm or s3" >&2; exit 1 ;;
esac
```

### 3. `grant-execution-role` (actor: agent)

Allow the task execution role to read exactly that parameter or object (inline policy epok-read-ingest-key).

Idempotent: safe to re-run.

Run:

```bash
B="${EPOK_KEY_S3_URI#s3://}"; BUCKET="${B%%/*}"; OBJ="${B#*/}"
case "$EPOK_KEY_STORE" in
  ssm) DOC="{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Action\":\"ssm:GetParameters\",\"Resource\":\"arn:aws:ssm:$AWS_REGION:$EPOK_ACCOUNT_ID:parameter$EPOK_KEY_PARAMETER\"}]}"
       WHAT="$EPOK_KEY_PARAMETER" ;;
  s3) DOC="{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Action\":\"s3:GetObject\",\"Resource\":\"arn:aws:s3:::$BUCKET/$OBJ\"},{\"Effect\":\"Allow\",\"Action\":\"s3:GetBucketLocation\",\"Resource\":\"arn:aws:s3:::$BUCKET\"}]}"
      WHAT="$EPOK_KEY_S3_URI" ;;
esac
aws iam put-role-policy --role-name "$EPOK_EXECUTION_ROLE_NAME" --policy-name epok-read-ingest-key --policy-document "$DOC"
echo "execution role $EPOK_EXECUTION_ROLE_NAME may read $WHAT"
```

### 4. `register-revision` (actor: agent)

Register a new revision: the current one plus the epok-otel-collector sidecar, plus OTEL_SERVICE_NAME / OTEL_TRACES_EXPORTER / OTEL_EXPORTER_OTLP_PROTOCOL / OTEL_EXPORTER_OTLP_ENDPOINT on APP_CONTAINER. Re-running replaces (does not duplicate) the sidecar. Prints EPOK_TASK_DEF_ARN.

Idempotent: safe to re-run.

Run:

```bash
python3 -c '
import json, os, subprocess, sys
e = os.environ
def aws(*a):
    out = subprocess.run(["aws", *a, "--region", e["AWS_REGION"], "--output", "json"],
                         capture_output=True, text=True)
    if out.returncode != 0:
        sys.exit(out.stderr)
    return json.loads(out.stdout)
svc = aws("ecs", "describe-services", "--cluster", e["ECS_CLUSTER"], "--services", e["ECS_SERVICE"])["services"][0]
cur = aws("ecs", "describe-task-definition", "--task-definition", svc["taskDefinition"], "--include", "TAGS")
td, tags = cur["taskDefinition"], cur.get("tags") or []
previous = next((t["value"] for t in tags if t["key"] == "epok-previous"), td["taskDefinitionArn"])
for k in ("taskDefinitionArn", "revision", "status", "requiresAttributes", "compatibilities",
          "registeredAt", "registeredBy", "deregisteredAt"):
    td.pop(k, None)
SIDECAR = "epok-otel-collector"
ours = {"OTEL_SERVICE_NAME", "OTEL_TRACES_EXPORTER", "OTEL_EXPORTER_OTLP_PROTOCOL", "OTEL_EXPORTER_OTLP_ENDPOINT"}
cfg = "receivers:\n  otlp:\n    protocols:\n      http:\n        endpoint: localhost:4318\nexporters:\n  otlphttp/epok:\n    endpoint: \x24{env:EPOK_INGEST_HOST}\n    headers:\n      x-api-key: \x24{env:EPOK_API_KEY}\nservice:\n  pipelines:\n    traces:\n      receivers: [otlp]\n      exporters: [otlphttp/epok]\n"
param = "arn:aws:ssm:" + e["AWS_REGION"] + ":" + e["EPOK_ACCOUNT_ID"] + ":parameter" + e["EPOK_KEY_PARAMETER"]
containers = [c for c in td["containerDefinitions"] if c["name"] != SIDECAR]
app_log = None
for c in containers:
    if c["name"] == e["APP_CONTAINER"]:
        c["environment"] = [v for v in c.get("environment", []) if v["name"] not in ours] + [
            {"name": "OTEL_SERVICE_NAME", "value": e["OTEL_SERVICE_NAME"]},
            {"name": "OTEL_TRACES_EXPORTER", "value": "otlp"},
            {"name": "OTEL_EXPORTER_OTLP_PROTOCOL", "value": "http/protobuf"},
            {"name": "OTEL_EXPORTER_OTLP_ENDPOINT", "value": "http://localhost:4318"}]
        deps = [d for d in c.get("dependsOn", []) if d["containerName"] != SIDECAR]
        c["dependsOn"] = deps + [{"containerName": SIDECAR, "condition": "START"}]
        app_log = c.get("logConfiguration")
sidecar = {"name": SIDECAR, "image": e["EPOK_COLLECTOR_IMAGE"], "essential": False,
           "command": ["--config=env:EPOK_OTEL_COLLECTOR_CONFIG"], "memoryReservation": 64,
           "environment": [{"name": "EPOK_OTEL_COLLECTOR_CONFIG", "value": cfg},
                           {"name": "EPOK_INGEST_HOST", "value": e["EPOK_INGEST_HOST"]}],
           }
if e["EPOK_KEY_STORE"] == "s3":
    sidecar["environmentFiles"] = [{"type": "s3", "value": "arn:aws:s3:::" + e["EPOK_KEY_S3_URI"][len("s3://"):]}]
else:
    sidecar["secrets"] = [{"name": "EPOK_API_KEY", "valueFrom": param}]
if app_log and app_log.get("logDriver") == "awslogs":
    opts = dict(app_log.get("options", {}))
    opts["awslogs-stream-prefix"] = SIDECAR
    sidecar["logConfiguration"] = {"logDriver": "awslogs", "options": opts}
td["containerDefinitions"] = containers + [sidecar]
tags = [t for t in tags if t["key"] != "epok-previous"] + [{"key": "epok-previous", "value": previous}]
if tags:
    td["tags"] = tags
new = aws("ecs", "register-task-definition", "--cli-input-json", json.dumps(td))  # no secret inside: only references
print("previous revision " + previous)
print("EPOK_TASK_DEF_ARN=" + new["taskDefinition"]["taskDefinitionArn"])'
```

The output contains `NAME=value` lines. Set each one as an environment variable (`EPOK_TASK_DEF_ARN`); read the lines, do not `eval` the output.

### 5. `deploy-revision` (actor: agent)

Roll the service to the new revision and wait until it is stable (normal ECS rolling deployment).

Idempotent: safe to re-run.

Run:

```bash
aws ecs update-service --region "$AWS_REGION" --cluster "$ECS_CLUSTER" --service "$ECS_SERVICE" \
  --task-definition "$EPOK_TASK_DEF_ARN" --query service.taskDefinition --output text
aws ecs wait services-stable --region "$AWS_REGION" --cluster "$ECS_CLUSTER" --services "$ECS_SERVICE"
echo "service $ECS_SERVICE runs $EPOK_TASK_DEF_ARN"
```

## Verify

Send one canary request to the ECS service carrying the W3C trace id `00000000$NONCE`:

```bash
if [ -z "${APP_URL:-}" ]; then echo "set APP_URL to the service URL (load balancer or task address)" >&2; exit 1; fi
curl -fsS -o /dev/null -w "HTTP %{http_code}\n" -H "traceparent: 00-00000000$NONCE-00f067aa0ba902b7-01" "${APP_URL}${EPOK_VERIFY_PATH}"
```

Then poll every 10 s for up to 180 s, re-sending the canary every 60 s (a collector may hold a batch until the next request):

```bash
printf 'header = "X-API-Key: %s"\n' "$EPOK_READ_API_KEY" | curl -K - -fsS \
  "$EPOK_API_HOST/api/v1/tenants/$EPOK_TENANT_ID/traces/00000000$NONCE" | python3 -c '
import json, sys
d = json.load(sys.stdin)
spans = [e for es in d.get("services", {}).values() for e in es
         if e.get("fields", {}).get("record_type") == "span"]
ids = {e.get("span_id") for e in spans}
services = sorted({e.get("service") for e in spans})
out = {
    "trace_id": d.get("trace_id"),
    "spans": len(spans),
    "services": services,
    "roots": sum(1 for e in spans if e.get("parent_span_id") not in ids),
    "children": sum(1 for e in spans if e.get("parent_span_id") in ids),
    "error_spans": sum(1 for e in spans if e.get("fields", {}).get("span_status") == "error"),
}
out["ok"] = True if spans and services == [sys.argv[1]] else None
print(json.dumps(out))' "$OTEL_SERVICE_NAME"
```

Success: the JSON field `ok` is not null (the trace is in Epok and every span has service.name = `$OTEL_SERVICE_NAME`). Report the full JSON line: spans, roots, children and error_spans show the trace shape.

## Failure modes

- **permission-denied** (step: inspect-service, store-key, grant-execution-role, register-revision, deploy-revision). **STOP AND ASK.** Symptom: AccessDenied / AccessDeniedException / not authorized to perform iam:PassRole. Check: `aws sts get-caller-identity --query Arn --output text`. Fix: The caller lacks a permission in prerequisites.permissions. Stop and ask a human. Never grant IAM to yourself.
- **not-awsvpc** (step: inspect-service). **STOP AND ASK.** Symptom: inspect-service exits with 'the sidecar needs awsvpc'. Check: `aws ecs describe-task-definition --task-definition "$EPOK_TASK_FAMILY" --region "$AWS_REGION" --query taskDefinition.networkMode`. Fix: bridge/host-mode EC2 tasks need a daemon collector per instance instead of a sidecar. Stop and ask.
- **deployment-not-stable** (step: deploy-revision). **STOP AND ASK.** Symptom: services-stable times out, or tasks stop with CannotPullContainerError / ResourceInitializationError. Check: `aws ecs describe-services --region "$AWS_REGION" --cluster "$ECS_CLUSTER" --services "$ECS_SERVICE" --query 'services[0].events[:5].message'`. Fix: ResourceInitializationError ... environment file / s3: the execution role cannot read the S3 env file (re-run grant-execution-role; an SSE-KMS bucket also needs kms:Decrypt). CannotPullContainerError: tasks have no route to ghcr.io (private subnet without NAT): mirror the collector image to ECR and set EPOK_COLLECTOR_IMAGE. ResourceInitializationError ... ssm: the execution role cannot read the parameter (re-run grant-execution-role; a customer-managed KMS key also needs kms:Decrypt). Out of memory: raise the task memory. If unsure, run the uninstall to roll back and stop and ask.
- **collector-not-exporting** (step: verify). Symptom: Requests succeed but the trace never arrives. Check: `G="$(aws ecs describe-task-definition --task-definition "$EPOK_TASK_DEF_ARN" --region "$AWS_REGION" --query 'taskDefinition.containerDefinitions[?name==`epok-otel-collector`].logConfiguration.options."awslogs-group" | [0]' --output text)"; aws logs tail "$G" --region "$AWS_REGION" --since 10m --log-stream-name-prefix epok-otel-collector | tail -30`. Fix: 401/403 = key (see epok-auth-rejected). No collector log stream = the app container has no awslogs config to copy; check the task's stopped reason. 'connection refused' in the app = the app's SDK is not reading OTEL_EXPORTER_OTLP_ENDPOINT (the app is not instrumented: stop and ask).
- **epok-auth-rejected** (step: verify). **STOP AND ASK.** Symptom: Spans never arrive; exporter or collector logs show HTTP 401 or 403 from the Epok ingest host. Check: `Ask the human which key they gave you and its scope (do not print it).`. Fix: The ingest key is wrong, revoked, or not ingest scope. Stop and ask the human for a new ingest-scope key. Do not retry in a loop.
- **verify-read-auth** (step: verify). **STOP AND ASK.** Symptom: The verify call fails with HTTP 401 or 403 on every poll. Check: `Does EPOK_READ_API_KEY have read scope, and is EPOK_TENANT_ID its tenant?`. Fix: 403 = the key lacks read scope or belongs to another tenant. Stop and ask the human.
- **trace-not-found** (step: verify). Symptom: The verify call keeps failing with HTTP 404 (Trace not found) until the timeout. Check: `Re-read the emit output: did the request/invocation succeed? Then check the failure modes for this runtime.`. Fix: The canary trace did not reach Epok, or your code does not continue an incoming W3C traceparent (then spans arrive under another trace id: look for OTEL_SERVICE_NAME in Epok -> Traces). Report; do not change sampling or keys to make it pass.
- **wrong-service-name** (step: verify). **STOP AND ASK.** Symptom: Verify finds the trace but ok is null and services is not [OTEL_SERVICE_NAME]. Check: `Look at services in the verify JSON.`. Fix: Something else sets service.name (OTEL_RESOURCE_ATTRIBUTES or code). Report it to the human; do not edit application code without asking.

## Uninstall (only when the user asks)

Each step tolerates resources that are already gone. Continue past a failing step and report it.

### U1. `restore-previous-revision` DESTRUCTIVE

If the running revision carries the epok-previous tag, roll the service back to that revision, wait until stable, and deregister the Epok revision. Otherwise do nothing.

```bash
python3 -c '
import json, os, subprocess, sys
e = os.environ
def aws(*a):
    out = subprocess.run(["aws", *a, "--region", e["AWS_REGION"], "--output", "json"], capture_output=True, text=True)
    if out.returncode != 0:
        if "NotFound" in out.stderr or "ServiceNotActive" in out.stderr:
            return None
        sys.exit(out.stderr)
    return json.loads(out.stdout) if out.stdout.strip() else {}
s = aws("ecs", "describe-services", "--cluster", e["ECS_CLUSTER"], "--services", e["ECS_SERVICE"])
s = [x for x in (s or {}).get("services", []) if x["status"] == "ACTIVE"]
if not s:
    print("service not found or inactive; nothing to restore")
    sys.exit(0)
cur = aws("ecs", "describe-task-definition", "--task-definition", s[0]["taskDefinition"], "--include", "TAGS")
prev = next((t["value"] for t in cur.get("tags") or [] if t["key"] == "epok-previous"), None)
if not prev:
    print("current revision " + s[0]["taskDefinition"] + " was not created by this setup; nothing to restore")
    sys.exit(0)
aws("ecs", "update-service", "--cluster", e["ECS_CLUSTER"], "--service", e["ECS_SERVICE"], "--task-definition", prev)
subprocess.run(["aws", "ecs", "wait", "services-stable", "--region", e["AWS_REGION"], "--cluster", e["ECS_CLUSTER"],
                "--services", e["ECS_SERVICE"]], check=True)
aws("ecs", "deregister-task-definition", "--task-definition", s[0]["taskDefinition"])
print("service back on " + prev + "; deregistered " + s[0]["taskDefinition"])'
```

### U2. `remove-role-policy` DESTRUCTIVE

Delete the inline policy epok-read-ingest-key from the execution role (already gone counts as done).

```bash
ROLE="$(python3 -c '
import json, os, subprocess
e = os.environ
r = subprocess.run(["aws", "ecs", "describe-services", "--cluster", e["ECS_CLUSTER"], "--services", e["ECS_SERVICE"],
                    "--region", e["AWS_REGION"], "--output", "json"], capture_output=True, text=True)
s = [x for x in json.loads(r.stdout or "{}").get("services", []) if x["status"] == "ACTIVE"] if r.returncode == 0 else []
if s:
    td = json.loads(subprocess.check_output(["aws", "ecs", "describe-task-definition", "--task-definition",
                    s[0]["taskDefinition"], "--region", e["AWS_REGION"], "--output", "json"]))["taskDefinition"]
    print((td.get("executionRoleArn") or "").split("/")[-1])
')"
ROLE="${ROLE:-${EPOK_EXECUTION_ROLE_NAME:-}}"
if [ -z "$ROLE" ]; then echo "execution role unknown (service gone); set EPOK_EXECUTION_ROLE_NAME to clean up"; exit 0; fi
if OUT="$(aws iam delete-role-policy --role-name "$ROLE" --policy-name epok-read-ingest-key 2>&1)"; then
  echo "removed policy epok-read-ingest-key from $ROLE"
elif grep -q NoSuchEntity <<< "$OUT"; then echo "policy already gone"; else echo "$OUT" >&2; exit 1; fi
```

### U3. `delete-key-parameter` DESTRUCTIVE

Delete the SSM parameter or S3 env file holding the key (already gone counts as done). Skip this step if another service still uses it.

```bash
B="${EPOK_KEY_S3_URI#s3://}"; BUCKET="${B%%/*}"; OBJ="${B#*/}"
case "$EPOK_KEY_STORE" in
  ssm)
    if OUT="$(aws ssm delete-parameter --region "$AWS_REGION" --name "$EPOK_KEY_PARAMETER" 2>&1)"; then
      echo "deleted $EPOK_KEY_PARAMETER"
    elif grep -q ParameterNotFound <<< "$OUT"; then echo "parameter already gone"; else echo "$OUT" >&2; exit 1; fi ;;
  s3)
    aws s3 rm "$EPOK_KEY_S3_URI" --only-show-errors && echo "deleted $EPOK_KEY_S3_URI (or it was already gone)" ;;
esac
```

## Final report

Finish with: manifest id and sha256, result, each step's status, the verify JSON line, and any failure-mode ids. Never include a key.
