---
name: epok-aws-lambda-otel-traces
description: Send OpenTelemetry traces from an existing Lambda function to Epok (AWS Lambda); use when a user asks to add tracing, APM or distributed traces for Lambda, serverless or AWS function workloads in Epok, or to verify or remove the Epok traces setup.
---

<!-- GENERATED from docs/integrations/manifests/aws-lambda-otel-traces.json (sha256 a36773c973f1138d09addfe950cf58f22121512bc99062942a2dd9c3f6b3c6d4) by scripts/gen_traces_setup_artifacts.py. Do not edit. -->

# AWS Lambda (Python) traces -> Epok with the OpenTelemetry Lambda layers

Adds the public OpenTelemetry Python layer (auto-instrumentation, AWS_LAMBDA_EXEC_WRAPPER) and the OpenTelemetry Collector layer to an existing Python Lambda function. The in-function collector forwards OTLP/HTTP to Epok with the x-api-key header. No code change, no redeploy of the function package.

- Manifest: `https://epok.cloud/setup/aws-lambda-otel-traces.json` (schema `epok.setup-manifest/1`, version 1.0.0)
- Human guide: `docs/integrations/traces/aws-lambda.md`
- Estimated time: 3 minutes
- **Not yet verified end to end by the Epok harness.** Treat any failure as possibly a manifest bug: report it, do not improvise.

If `scripts/epok_setup.py` is available, prefer it: it runs these exact steps, masks the keys and writes a transcript:

```bash
python3 scripts/epok_setup.py docs/integrations/manifests/aws-lambda-otel-traces.json --dry-run
python3 scripts/epok_setup.py docs/integrations/manifests/aws-lambda-otel-traces.json --json-transcript epok-setup.json
```

## Stop and ask a human

Stop and ask (do not work around it) when:

- `EPOK_API_KEY` or `EPOK_READ_API_KEY` is not in the environment. **A human creates keys in the Epok UI. Never create, guess, print, or write a key to a file or commit.**
- You are not sure which Lambda function the user means, or it serves production traffic and the user has not said this change is fine. Every step changes only that one workload.
- The workload already has another tracer or APM agent (Datadog, New Relic, ADOT, X-Ray SDK). Two tracers on one workload conflict.
- The application code is not instrumented with OpenTelemetry and the runtime has no zero-code option here. Do not edit application code without asking.
- Any step fails with an access-denied error. Never grant IAM roles or policies to yourself.
- Calling the workload (the verify canary) has side effects.
- A failure mode below is marked STOP AND ASK.
- Never run the uninstall section unless the user asked for removal.

## Inputs

- `AWS_REGION` (required): Region of the function. How to get it: aws configure get region, or the function ARN. Must match `^[a-z]{2}(-[a-z]+)+-[0-9]$`.
- `FUNCTION_NAME` (required): Name of the existing Python (python3.x, zip package) Lambda function to instrument. How to get it: aws lambda list-functions --query 'Functions[?starts_with(Runtime, `python3`)].FunctionName' (confirm with the human). Must match `^[A-Za-z0-9_-]{1,64}$`.
- `OTEL_SERVICE_NAME` (required): service.name for the spans (how the service appears in Epok). How to get it: Ask the human, or use the workload name. Must match `^[A-Za-z0-9._-]{1,128}$`.
- `EPOK_KEY_STORE` (default `secretsmanager`): Where the collector reads the key. secretsmanager (default, recommended): a Secrets Manager secret read by the collector's secretsmanager config provider with the function role. env (NOT RECOMMENDED): the key as the plain function environment variable EPOK_API_KEY, readable by anyone with lambda:GetFunctionConfiguration; use only when Secrets Manager is unavailable and the human agrees. Must match `^(secretsmanager|env)$`.
- `EPOK_KEY_SECRET_NAME` (default `epok/ingest-api-key`): Secrets Manager secret name for the key (EPOK_KEY_STORE=secretsmanager). An existing secret is reused, not overwritten. About $0.40/month per secret plus $0.05 per 10,000 API calls (one read per cold start). Must match `^[A-Za-z0-9/_+=.@-]{1,200}$`.
- `EPOK_RESOURCE_TAGS` (default `managed-by=epok`): Comma-separated key=value tags for the secret. Must match `^[A-Za-z0-9_.:/=+@-]+=[A-Za-z0-9_.:/=+@ -]*(,[A-Za-z0-9_.:/=+@-]+=[A-Za-z0-9_.:/=+@ -]*){0,9}$`.
- `EPOK_OTEL_PYTHON_LAYER_VERSION` (default `0_22_0:1`): Version suffix of the public layer arn:aws:lambda:<region>:184161586896:layer:opentelemetry-python-<v> (layer-python/0.22.0 = opentelemetry-python 1.45.1). Supports arm64 and x86_64. Must match `^[0-9]+_[0-9]+_[0-9]+:[0-9]+$`.
- `EPOK_OTEL_COLLECTOR_LAYER_VERSION` (default `0_24_0:1`): Version suffix of the public layer arn:aws:lambda:<region>:184161586896:layer:opentelemetry-collector-<amd64|arm64>-<v> (layer-collector/0.24.0 = collector v0.163.0). The architecture is read from the function. Must match `^[0-9]+_[0-9]+_[0-9]+:[0-9]+$`.
- **`EPOK_API_KEY` — HUMAN-PROVIDED SECRET (ingest scope). Epok ingest-scope API key, dedicated to this workload.** A human creates it: Epok -> Settings -> API Keys -> New key (scope: ingest, name: traces-<workload>). Agents must ask for it, never create or guess it, and never write it to a file or a commit.
- **`EPOK_READ_API_KEY` — HUMAN-PROVIDED SECRET (read scope). Epok API key with read scope, used only by verify to look the canary trace up. May be the same key as EPOK_API_KEY if that key also has read scope.** A human creates it: Epok -> Settings -> API Keys (scope: read). Agents must ask for it.
- `EPOK_INGEST_HOST` (default `https://ingest.epok.cloud`): Epok OTLP/HTTP ingest base URL, no trailing slash. The exporter appends /v1/traces. Change only if Epok support tells you to. Must match `^(https://[A-Za-z0-9.-]+|http://(127\.0\.0\.1|localhost))(:[0-9]{1,5})?$`.
- `EPOK_API_HOST` (default `https://app.epok.cloud`): Epok API base URL used by verify, no trailing slash. Must match `^(https://[A-Za-z0-9.-]+|http://(127\.0\.0\.1|localhost))(:[0-9]{1,5})?$`.
- `EPOK_TENANT_ID` (required): Your Epok tenant (account) id, used in the verify API path. How to get it: Epok -> Settings -> Account (the numeric account id), or ask the human. Must match `^[0-9]{1,10}$`.
- `NONCE` (generated: nonce): Canary token: verify sends one request with W3C traceparent trace id 00000000<NONCE> and looks that trace up in Epok. Generated by the runner if not set (24 lowercase hex). Must match `^[0-9a-f]{24}$`.

Export every input as an environment variable before running the commands below.

## Prerequisites

- `aws` >= 2.15.0 (check: `aws --version`)
- `python3` >= 3.8.0 (check: `python3 --version`)
- `curl` >= 7.68.0 (check: `curl --version`)
- Caller permissions: `lambda:GetFunctionConfiguration`, `lambda:UpdateFunctionConfiguration`, `lambda:GetLayerVersion`, `lambda:InvokeFunction`, `secretsmanager:CreateSecret`, `secretsmanager:DescribeSecret`, `secretsmanager:TagResource`, `secretsmanager:DeleteSecret`, `iam:PutRolePolicy`, `iam:DeleteRolePolicy`. lambda:InvokeFunction is only for the verify canary. The layers are public (account 184161586896). The secretsmanager:* and iam:*RolePolicy permissions are for the default key store (an inline policy on the function's own role, scoped to one secret ARN); EPOK_KEY_STORE=env needs none of them.

Run these checks; all must succeed:

```bash
aws sts get-caller-identity --query Account --output text >/dev/null
aws lambda get-function-configuration --function-name "$FUNCTION_NAME" --region "$AWS_REGION" --query Runtime --output text | grep -q '^python3\.'
aws lambda get-layer-version-by-arn --region "$AWS_REGION" --arn "arn:aws:lambda:$AWS_REGION:184161586896:layer:opentelemetry-python-$EPOK_OTEL_PYTHON_LAYER_VERSION" --query LayerVersionArn --output text >/dev/null
```

## Cost

One Secrets Manager secret (about $0.40/month, plus $0.05 per 10,000 reads; the collector reads it once per cold start). The collector extension shares the function's memory and adds roughly 100-300 ms to cold starts and a few ms per invocation (billed duration). Epok bills the decoded OTLP bytes: about 1-2 KB per 3-span trace from these layers, so 1 million invocations/day with 3 spans each is about 1.5 GB/day.

- Python functions only (zip package). For Node.js use the opentelemetry-nodejs layer with the same collector layer and collector config; the steps are otherwise identical.
- The collector config is passed in the EPOK_OTEL_COLLECTOR_CONFIG environment variable (OPENTELEMETRY_COLLECTOR_CONFIG_URI=env:...), so the function package does not change.
- Default key store: Secrets Manager. The collector layer's secretsmanager config provider resolves ${secretsmanager:<secret ARN>} at collector start (one GetSecretValue per cold start) with the function role, which gets GetSecretValue on that one ARN. The key never appears in the function configuration. A customer-managed KMS key on the secret also needs kms:Decrypt for the role.
- EPOK_KEY_STORE=env is NOT RECOMMENDED: it writes the key into the function environment (EPOK_API_KEY), encrypted at rest but readable by anyone with lambda:GetFunctionConfiguration and visible in the console.
- The root span is created by the layer (one SERVER span per invocation) and continues a W3C traceparent found in event.headers (API Gateway, Function URLs, ALB).
- Verify invokes the function once with the event {"headers":{"traceparent":...}}. If invoking the function has side effects, ask the human before verify.

## Steps

Run in order. Each step runs as `bash -euo pipefail`. Where a check is given, run it first: exit 0 means the step is already done, so skip it. Stop at the first failing step and use the failure modes below.

### 1. `inspect-function` (actor: agent)

Read the function configuration: it must be a python3.x zip function. Prints EPOK_COLLECTOR_ARCH (amd64 or arm64) for the collector layer.

Idempotent: safe to re-run.

Run:

```bash
aws lambda get-function-configuration --function-name "$FUNCTION_NAME" --region "$AWS_REGION" --output json | python3 -c '
import json, sys
c = json.load(sys.stdin)
rt = c.get("Runtime") or ""
if not rt.startswith("python3."):
    sys.exit("runtime " + (rt or str(c.get("PackageType"))) + " is not a Python zip function; this manifest instruments Python only")
arch = (c.get("Architectures") or ["x86_64"])[0]
print("runtime " + rt + ", architecture " + arch + ", layers " + str(len(c.get("Layers") or [])))
print("EPOK_COLLECTOR_ARCH=" + ("arm64" if arch == "arm64" else "amd64"))
print("EPOK_FUNCTION_ROLE_NAME=" + c["Role"].split("/")[-1])
print("EPOK_ACCOUNT_ID=" + c["FunctionArn"].split(":")[4])'
```

The output contains `NAME=value` lines. Set each one as an environment variable (`EPOK_COLLECTOR_ARCH`, `EPOK_FUNCTION_ROLE_NAME`, `EPOK_ACCOUNT_ID`); read the lines, do not `eval` the output.

### 2. `store-key` (actor: agent)

EPOK_KEY_STORE=secretsmanager (default): create the Secrets Manager secret EPOK_KEY_SECRET_NAME holding the key (value read from JSON on stdin, never on the command line). Skipped if the secret exists, and always skipped with EPOK_KEY_STORE=env.

Check (skip the step if this exits 0):

```bash
case "$EPOK_KEY_STORE" in
  env) exit 0 ;;   # nothing to store: the key goes into the function environment (not recommended)
  secretsmanager) aws secretsmanager describe-secret --secret-id "$EPOK_KEY_SECRET_NAME" --region "$AWS_REGION" --query ARN --output text >/dev/null 2>&1 ;;
esac
```

Run:

```bash
# AWS CLI v2 (2.17.0 to 2.40.3 tested) rejects --cli-input-json read from a pipe ("Invalid JSON received"). The key goes in through a plain parameter file on stdin instead; the other arguments are not secret.
python3 -c '
import json, os, subprocess, sys
e = os.environ
tags = [dict(zip(("Key", "Value"), p.split("=", 1))) for p in e["EPOK_RESOURCE_TAGS"].split(",")]
sys.exit(subprocess.run(["aws", "secretsmanager", "create-secret", "--region", e["AWS_REGION"],
                         "--name", e["EPOK_KEY_SECRET_NAME"], "--description", "Epok ingest API key for OpenTelemetry traces",
                         "--tags", json.dumps(tags), "--secret-string", "file:///dev/stdin",
                         "--query", "ARN", "--output", "text"], input=e["EPOK_API_KEY"], text=True).returncode)
'
```

### 3. `grant-function-role` (actor: agent)

Allow the function's execution role secretsmanager:GetSecretValue on exactly that secret ARN (inline policy epok-read-ingest-key). No-op with EPOK_KEY_STORE=env.

Idempotent: safe to re-run.

Run:

```bash
case "$EPOK_KEY_STORE" in
  env) echo "EPOK_KEY_STORE=env: no IAM change (not recommended: the key sits in the function configuration)"; exit 0 ;;
esac
ARN="$(aws secretsmanager describe-secret --secret-id "$EPOK_KEY_SECRET_NAME" --region "$AWS_REGION" --query ARN --output text)"
aws iam put-role-policy --role-name "$EPOK_FUNCTION_ROLE_NAME" --policy-name epok-read-ingest-key \
  --policy-document "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Action\":\"secretsmanager:GetSecretValue\",\"Resource\":\"$ARN\"}]}"
echo "function role $EPOK_FUNCTION_ROLE_NAME may read exactly $ARN"
```

### 4. `configure-function` (actor: agent)

Add the two OpenTelemetry layers (replacing older versions of them, keeping every other layer) and merge the OTEL_* / Epok variables into the existing environment (every other variable is kept). The collector reads the key with ${secretsmanager:<secret ARN>} (default) or, with EPOK_KEY_STORE=env, from the EPOK_API_KEY variable. Waits until the update is applied.

Idempotent: safe to re-run.

Run:

```bash
# AWS CLI v2 (2.17.0 to 2.40.3 tested) rejects --cli-input-json read from a pipe ("Invalid JSON received"). The key goes in through a plain parameter file on stdin instead; the other arguments are not secret.
python3 -c '
import json, os, subprocess, sys
e = os.environ
c = json.loads(subprocess.check_output(["aws", "lambda", "get-function-configuration", "--function-name",
                                        e["FUNCTION_NAME"], "--region", e["AWS_REGION"], "--output", "json"]))
otel = ":184161586896:layer:opentelemetry-"
py = "arn:aws:lambda:" + e["AWS_REGION"] + otel[:-1] + "-python-" + e["EPOK_OTEL_PYTHON_LAYER_VERSION"]
col = ("arn:aws:lambda:" + e["AWS_REGION"] + otel + "collector-" + e["EPOK_COLLECTOR_ARCH"] + "-"
       + e["EPOK_OTEL_COLLECTOR_LAYER_VERSION"])
layers = [l["Arn"] for l in c.get("Layers") or [] if otel not in l["Arn"]] + [py, col]
if len(layers) > 5:
    sys.exit("the function would have more than 5 layers; stop and ask")
env = (c.get("Environment") or {}).get("Variables") or {}
wrapper = env.get("AWS_LAMBDA_EXEC_WRAPPER")
if wrapper not in (None, "/opt/otel-instrument"):
    sys.exit("AWS_LAMBDA_EXEC_WRAPPER is already " + wrapper + "; stop and ask")
cfg = "receivers:\n  otlp:\n    protocols:\n      http:\n        endpoint: localhost:4318\nexporters:\n  otlphttp/epok:\n    endpoint: \x24{env:EPOK_INGEST_HOST}\n    headers:\n      x-api-key: \x24{env:EPOK_API_KEY}\nservice:\n  pipelines:\n    traces:\n      receivers: [otlp]\n      exporters: [otlphttp/epok]\n"
for k in ("EPOK_API_KEY", "EPOK_KEY_SECRET_ARN"):
    env.pop(k, None)
if e["EPOK_KEY_STORE"] == "secretsmanager":
    arn = subprocess.check_output(["aws", "secretsmanager", "describe-secret", "--secret-id", e["EPOK_KEY_SECRET_NAME"],
                                   "--region", e["AWS_REGION"], "--query", "ARN", "--output", "text"], text=True).strip()
    cfg = cfg.replace("\x24{env:EPOK_API_KEY}", "\x24{secretsmanager:" + arn + "}")   # \x24 = "$" (collector placeholder, not shell)
else:
    env["EPOK_API_KEY"] = e["EPOK_API_KEY"]   # not recommended: readable with lambda:GetFunctionConfiguration
env.update({
    "AWS_LAMBDA_EXEC_WRAPPER": "/opt/otel-instrument",
    "OTEL_SERVICE_NAME": e["OTEL_SERVICE_NAME"],
    "OTEL_TRACES_EXPORTER": "otlp",
    "OTEL_METRICS_EXPORTER": "none",
    "OTEL_LOGS_EXPORTER": "none",
    "OTEL_EXPORTER_OTLP_PROTOCOL": "http/protobuf",
    "OTEL_EXPORTER_OTLP_ENDPOINT": "http://localhost:4318",
    "OPENTELEMETRY_COLLECTOR_CONFIG_URI": "env:EPOK_OTEL_COLLECTOR_CONFIG",
    "EPOK_OTEL_COLLECTOR_CONFIG": cfg,
    "EPOK_INGEST_HOST": e["EPOK_INGEST_HOST"],
})
# The environment may hold the key (EPOK_KEY_STORE=env): stdin, never argv.
r = subprocess.run(["aws", "lambda", "update-function-configuration", "--region", e["AWS_REGION"],
                    "--function-name", e["FUNCTION_NAME"], "--layers", *layers, "--environment", "file:///dev/stdin",
                    "--query", "join(`,`, [FunctionName, LastUpdateStatus])", "--output", "text"],
                   input=json.dumps({"Variables": env}), text=True)
if r.returncode:
    sys.exit(r.returncode)
'
aws lambda wait function-updated-v2 --function-name "$FUNCTION_NAME" --region "$AWS_REGION"
echo "instrumented $FUNCTION_NAME"
```

## Verify

Send one canary request to the Lambda function carrying the W3C trace id `00000000$NONCE`:

```bash
OUT="$(mktemp)"
aws lambda invoke --function-name "$FUNCTION_NAME" --region "$AWS_REGION" --cli-binary-format raw-in-base64-out \
  --payload '{"headers":{"traceparent":"00-00000000'"$NONCE"'-00f067aa0ba902b7-01"}}' \
  --query StatusCode --output text "$OUT"
rm -f "$OUT"
```

Then poll every 10 s for up to 180 s, re-sending the canary every 60 s (a collector may hold a batch until the next request):

```bash
printf 'header = "X-API-Key: %s"\n' "$EPOK_READ_API_KEY" | curl -K - -fsS \
  "$EPOK_API_HOST/api/v1/tenants/$EPOK_TENANT_ID/traces/00000000$NONCE" | python3 -c '
import json, sys
d = json.load(sys.stdin)
spans = [e for es in d.get("services", {}).values() for e in es
         if e.get("fields", {}).get("record_type") == "span"]
ids = {e.get("span_id") for e in spans}
services = sorted({e.get("service") for e in spans})
out = {
    "trace_id": d.get("trace_id"),
    "spans": len(spans),
    "services": services,
    "roots": sum(1 for e in spans if e.get("parent_span_id") not in ids),
    "children": sum(1 for e in spans if e.get("parent_span_id") in ids),
    "error_spans": sum(1 for e in spans if e.get("fields", {}).get("span_status") == "error"),
}
out["ok"] = True if spans and services == [sys.argv[1]] else None
print(json.dumps(out))' "$OTEL_SERVICE_NAME"
```

Success: the JSON field `ok` is not null (the trace is in Epok and every span has service.name = `$OTEL_SERVICE_NAME`). Report the full JSON line: spans, roots, children and error_spans show the trace shape.

## Failure modes

- **permission-denied** (step: inspect-function, configure-function). **STOP AND ASK.** Symptom: AccessDeniedException on get-function-configuration or update-function-configuration. Check: `aws sts get-caller-identity --query Arn --output text`. Fix: The caller lacks a permission in prerequisites.permissions. Stop and ask a human to grant it. Never grant IAM to yourself.
- **secret-access-denied** (step: store-key, grant-function-role, verify). **STOP AND ASK.** Symptom: AccessDeniedException on create-secret/describe-secret, or (verify) collector logs show 'secretsmanager' / 'GetSecretValue' AccessDenied and no trace arrives. Check: `aws logs tail "/aws/lambda/$FUNCTION_NAME" --region "$AWS_REGION" --since 10m | grep -iE 'secretsmanager|GetSecretValue|AccessDenied' | tail -10`. Fix: Caller side: you lack a secretsmanager:* permission in prerequisites.permissions (stop and ask; never grant IAM to yourself). Function side: re-run grant-function-role; IAM can take up to a minute to propagate (verify re-invokes every 60 s); a customer-managed KMS key also needs kms:Decrypt. Do not fall back to EPOK_KEY_STORE=env without the human's agreement.
- **not-python** (step: inspect-function). **STOP AND ASK.** Symptom: inspect-function exits with 'is not a Python zip function'. Check: `aws lambda get-function-configuration --function-name "$FUNCTION_NAME" --region "$AWS_REGION" --query '[Runtime,PackageType]'`. Fix: Container-image or non-Python functions need the SDK in the image (see the Epok ECS/Cloud Run guides) or the matching language layer. Stop and ask which path the human wants.
- **layer-not-found** (step: configure-function). Symptom: update-function-configuration fails with 'Layer version ... does not exist' or AccessDenied on the layer. Check: `aws lambda get-layer-version-by-arn --region "$AWS_REGION" --arn "arn:aws:lambda:$AWS_REGION:184161586896:layer:opentelemetry-collector-$EPOK_COLLECTOR_ARCH-$EPOK_OTEL_COLLECTOR_LAYER_VERSION"`. Fix: The layer version is not published in this region. Pick a published version from https://github.com/open-telemetry/opentelemetry-lambda/releases and re-run with EPOK_OTEL_*_LAYER_VERSION set.
- **wrapper-conflict** (step: configure-function). **STOP AND ASK.** Symptom: configure-function exits with 'AWS_LAMBDA_EXEC_WRAPPER is already ...' or 'more than 5 layers'. Check: `aws lambda get-function-configuration --function-name "$FUNCTION_NAME" --region "$AWS_REGION" --query '[Layers[].Arn, Environment.Variables.AWS_LAMBDA_EXEC_WRAPPER]'`. Fix: Another wrapper or APM layer (Datadog, New Relic, ADOT) is installed. Two tracers on one function conflict. Stop and ask the human which one to keep.
- **collector-not-exporting** (step: verify). Symptom: The invocation succeeds but the trace never arrives. Check: `aws logs tail "/aws/lambda/$FUNCTION_NAME" --region "$AWS_REGION" --since 10m | grep -iE 'otel|collector|export|401|403' | tail -20`. Fix: Look for collector errors: 401/403 = key (see epok-auth-rejected); 'connection refused' on 4318 = the collector layer is missing or failed to start (check the layer ARN architecture matches the function); DNS/timeout = the function runs in a VPC without internet egress (add a NAT gateway or VPC endpoint route; stop and ask).
- **epok-auth-rejected** (step: verify). **STOP AND ASK.** Symptom: Spans never arrive; exporter or collector logs show HTTP 401 or 403 from the Epok ingest host. Check: `Ask the human which key they gave you and its scope (do not print it).`. Fix: The ingest key is wrong, revoked, or not ingest scope. Stop and ask the human for a new ingest-scope key. Do not retry in a loop.
- **verify-read-auth** (step: verify). **STOP AND ASK.** Symptom: The verify call fails with HTTP 401 or 403 on every poll. Check: `Does EPOK_READ_API_KEY have read scope, and is EPOK_TENANT_ID its tenant?`. Fix: 403 = the key lacks read scope or belongs to another tenant. Stop and ask the human.
- **trace-not-found** (step: verify). Symptom: The verify call keeps failing with HTTP 404 (Trace not found) until the timeout. Check: `Re-read the emit output: did the request/invocation succeed? Then check the failure modes for this runtime.`. Fix: The canary trace did not reach Epok, or your code does not continue an incoming W3C traceparent (then spans arrive under another trace id: look for OTEL_SERVICE_NAME in Epok -> Traces). Report; do not change sampling or keys to make it pass.
- **wrong-service-name** (step: verify). **STOP AND ASK.** Symptom: Verify finds the trace but ok is null and services is not [OTEL_SERVICE_NAME]. Check: `Look at services in the verify JSON.`. Fix: Something else sets service.name (OTEL_RESOURCE_ATTRIBUTES or code). Report it to the human; do not edit application code without asking.

## Uninstall (only when the user asks)

Each step tolerates resources that are already gone. Continue past a failing step and report it.

### U1. `remove-instrumentation` DESTRUCTIVE

Remove the two OpenTelemetry layers and exactly the variables configure-function added (other layers and variables are kept). A missing function counts as done.

```bash
CFG_JSON="$(python3 -c '
import json, os, subprocess
e = os.environ
r = subprocess.run(["aws", "lambda", "get-function-configuration", "--function-name", e["FUNCTION_NAME"],
                    "--region", e["AWS_REGION"], "--output", "json"], capture_output=True, text=True)
if r.returncode == 0:
    c = json.loads(r.stdout)
    otel = ":184161586896:layer:opentelemetry-"
    ours = {"AWS_LAMBDA_EXEC_WRAPPER", "OTEL_SERVICE_NAME", "OTEL_TRACES_EXPORTER", "OTEL_METRICS_EXPORTER",
            "OTEL_LOGS_EXPORTER", "OTEL_EXPORTER_OTLP_PROTOCOL", "OTEL_EXPORTER_OTLP_ENDPOINT",
            "OPENTELEMETRY_COLLECTOR_CONFIG_URI", "EPOK_OTEL_COLLECTOR_CONFIG", "EPOK_INGEST_HOST", "EPOK_API_KEY",
            "EPOK_KEY_SECRET_ARN"}
    env = {k: v for k, v in ((c.get("Environment") or {}).get("Variables") or {}).items() if k not in ours}
    layers = [l["Arn"] for l in c.get("Layers") or [] if otel not in l["Arn"]]
    print(json.dumps({"FunctionName": e["FUNCTION_NAME"], "Layers": layers, "Environment": {"Variables": env}}))
')"
if [ -z "$CFG_JSON" ]; then echo "function $FUNCTION_NAME not found; nothing to do"; exit 0; fi
aws lambda update-function-configuration --region "$AWS_REGION" \
  --cli-input-json "$CFG_JSON" --query 'join(`,`, [FunctionName, LastUpdateStatus])' --output text
aws lambda wait function-updated-v2 --function-name "$FUNCTION_NAME" --region "$AWS_REGION"
echo "removed Epok tracing layers and variables from $FUNCTION_NAME"
```

### U2. `remove-role-policy` DESTRUCTIVE

Delete the inline policy epok-read-ingest-key from the function role (already gone counts as done). Not run while the function may still read the secret.

```bash
ROLE="$(aws lambda get-function-configuration --function-name "$FUNCTION_NAME" --region "$AWS_REGION" --query Role --output text 2>/dev/null | sed 's#.*/##')" || true
ROLE="${ROLE:-${EPOK_FUNCTION_ROLE_NAME:-}}"
if [ -z "$ROLE" ]; then echo "function role unknown (function gone); set EPOK_FUNCTION_ROLE_NAME to clean up"; exit 0; fi
if OUT="$(aws iam delete-role-policy --role-name "$ROLE" --policy-name epok-read-ingest-key 2>&1)"; then
  echo "removed policy epok-read-ingest-key from $ROLE"
elif grep -q NoSuchEntity <<< "$OUT"; then echo "policy already gone"; else echo "$OUT" >&2; exit 1; fi
```

### U3. `delete-key-secret` DESTRUCTIVE

Delete the Secrets Manager secret (no recovery window: the key itself lives in Epok and can be re-issued). Skipped with EPOK_KEY_STORE=env; already gone counts as done. Skip this step if another function still uses the same secret.

```bash
if [ "$EPOK_KEY_STORE" = env ]; then echo "EPOK_KEY_STORE=env: no secret to delete"; exit 0; fi
if OUT="$(aws secretsmanager delete-secret --region "$AWS_REGION" --secret-id "$EPOK_KEY_SECRET_NAME" --force-delete-without-recovery 2>&1)"; then
  echo "deleted secret $EPOK_KEY_SECRET_NAME"
elif grep -q ResourceNotFoundException <<< "$OUT"; then echo "secret already gone"; else echo "$OUT" >&2; exit 1; fi
```

## Final report

Finish with: manifest id and sha256, result, each step's status, the verify JSON line, and any failure-mode ids. Never include a key.
