---
name: epok-gcp-cloudrun-otel-traces
description: Send OpenTelemetry traces from an existing Cloud Run service to Epok (Google Cloud Run); use when a user asks to add tracing, APM or distributed traces for Cloud Run or GCP serverless container workloads in Epok, or to verify or remove the Epok traces setup.
---

<!-- GENERATED from docs/integrations/manifests/gcp-cloudrun-otel-traces.json (sha256 e2fce4d461b75b6a9d780bb0da05f262ddcd4edd883540b6231657244a02fd23) by scripts/gen_traces_setup_artifacts.py. Do not edit. -->

# Google Cloud Run traces -> Epok with the OpenTelemetry SDK (OTLP/HTTP direct)

Points the OpenTelemetry SDK of an existing Cloud Run service at Epok with the standard OTEL_* variables (OTLP/HTTP protobuf, x-api-key header) and rolls a new revision. No collector, no new GCP resources.

- Manifest: `https://epok.cloud/setup/gcp-cloudrun-otel-traces.json` (schema `epok.setup-manifest/1`, version 1.0.0)
- Human guide: `docs/integrations/traces/gcp-cloud-run.md`
- Estimated time: 3 minutes
- **Not yet verified end to end by the Epok harness.** Treat any failure as possibly a manifest bug: report it, do not improvise.

If `scripts/epok_setup.py` is available, prefer it: it runs these exact steps, masks the keys and writes a transcript:

```bash
python3 scripts/epok_setup.py docs/integrations/manifests/gcp-cloudrun-otel-traces.json --dry-run
python3 scripts/epok_setup.py docs/integrations/manifests/gcp-cloudrun-otel-traces.json --json-transcript epok-setup.json
```

## Stop and ask a human

Stop and ask (do not work around it) when:

- `EPOK_API_KEY` or `EPOK_READ_API_KEY` is not in the environment. **A human creates keys in the Epok UI. Never create, guess, print, or write a key to a file or commit.**
- You are not sure which Cloud Run service the user means, or it serves production traffic and the user has not said this change is fine. Every step changes only that one workload.
- The workload already has another tracer or APM agent (Datadog, New Relic, ADOT, X-Ray SDK). Two tracers on one workload conflict.
- The application code is not instrumented with OpenTelemetry and the runtime has no zero-code option here. Do not edit application code without asking.
- Any step fails with an access-denied error. Never grant IAM roles or policies to yourself.
- Calling the workload (the verify canary) has side effects.
- A failure mode below is marked STOP AND ASK.
- Never run the uninstall section unless the user asked for removal.

## Inputs

- `PROJECT_ID` (required): GCP project of the service. How to get it: gcloud config get-value project Must match `^[a-z][a-z0-9-]{4,28}[a-z0-9]$`.
- `REGION` (required): Cloud Run region of the service. How to get it: gcloud run services list --format='value(metadata.name,region)' Must match `^[a-z]+-[a-z]+[0-9]+$`.
- `CLOUD_RUN_SERVICE` (required): Cloud Run service name (confirm with the human). Must match `^[a-z]([-a-z0-9]{0,61}[a-z0-9])?$`.
- `OTEL_SERVICE_NAME` (required): service.name for the spans (how the service appears in Epok). How to get it: Ask the human, or use the workload name. Must match `^[A-Za-z0-9._-]{1,128}$`.
- `EPOK_VERIFY_PATH` (default `/`): Path verify requests. Pick one your instrumented HTTP server handles (the example app traces /work). Must match `^/[A-Za-z0-9_./?=&-]*$`.
- `EPOK_KEY_STORE` (default `secretmanager`): Where the key lives. secretmanager (default, recommended): a Secret Manager secret mounted as OTEL_EXPORTER_OTLP_HEADERS with --update-secrets. env (NOT RECOMMENDED): the header as a plain env var, readable by anyone with run.services.get; use only when Secret Manager is unavailable and the human agrees. Must match `^(secretmanager|env)$`.
- `EPOK_KEY_SECRET` (default `epok-otlp-headers`): Secret Manager secret id (EPOK_KEY_STORE=secretmanager). An existing secret is reused, not overwritten; its value must be x-api-key=<key>. Must match `^[A-Za-z0-9_-]{1,255}$`.
- `EPOK_LABELS` (default `managed-by=epok`): Comma-separated key=value labels for the secret (GCP label rules). Must match `^[a-z][a-z0-9_-]{0,62}=[a-z0-9_-]{0,63}(,[a-z][a-z0-9_-]{0,62}=[a-z0-9_-]{0,63}){0,15}$`.
- **`EPOK_API_KEY` — HUMAN-PROVIDED SECRET (ingest scope). Epok ingest-scope API key, dedicated to this workload.** A human creates it: Epok -> Settings -> API Keys -> New key (scope: ingest, name: traces-<workload>). Agents must ask for it, never create or guess it, and never write it to a file or a commit.
- **`EPOK_READ_API_KEY` — HUMAN-PROVIDED SECRET (read scope). Epok API key with read scope, used only by verify to look the canary trace up. May be the same key as EPOK_API_KEY if that key also has read scope.** A human creates it: Epok -> Settings -> API Keys (scope: read). Agents must ask for it.
- `EPOK_INGEST_HOST` (default `https://ingest.epok.cloud`): Epok OTLP/HTTP ingest base URL, no trailing slash. The exporter appends /v1/traces. Change only if Epok support tells you to. Must match `^(https://[A-Za-z0-9.-]+|http://(127\.0\.0\.1|localhost))(:[0-9]{1,5})?$`.
- `EPOK_API_HOST` (default `https://app.epok.cloud`): Epok API base URL used by verify, no trailing slash. Must match `^(https://[A-Za-z0-9.-]+|http://(127\.0\.0\.1|localhost))(:[0-9]{1,5})?$`.
- `EPOK_TENANT_ID` (required): Your Epok tenant (account) id, used in the verify API path. How to get it: Epok -> Settings -> Account (the numeric account id), or ask the human. Must match `^[0-9]{1,10}$`.
- `NONCE` (generated: nonce): Canary token: verify sends one request with W3C traceparent trace id 00000000<NONCE> and looks that trace up in Epok. Generated by the runner if not set (24 lowercase hex). Must match `^[0-9a-f]{24}$`.

Export every input as an environment variable before running the commands below.

## Prerequisites

- `gcloud` >= 470.0.0 (check: `gcloud --version`)
- `python3` >= 3.8.0 (check: `python3 --version`)
- `curl` >= 7.68.0 (check: `curl --version`)
- Caller permissions: `roles/run.developer`, `roles/iam.serviceAccountUser`, `roles/run.invoker`, `roles/secretmanager.admin`. roles/secretmanager.admin (create the secret, bind the runtime service account, delete on uninstall) is for the default key store only; the Secret Manager API (secretmanager.googleapis.com) must be enabled. roles/iam.serviceAccountUser on the service's runtime service account (to roll a revision); roles/run.invoker only for the verify request when the service requires authentication. The app must already use an OpenTelemetry SDK that reads the standard OTEL_* variables and flushes spans before the response (Cloud Run throttles CPU between requests); examples/otel-traces/service shows the minimum.

Run these checks; all must succeed:

```bash
test -n "$(gcloud auth list --filter=status:ACTIVE --format='value(account)')"
gcloud run services describe "$CLOUD_RUN_SERVICE" --region="$REGION" --project="$PROJECT_ID" --format='value(status.url)' | grep -q '^https://'
[ "$EPOK_KEY_STORE" = env ] || gcloud services list --enabled --project="$PROJECT_ID" --filter=config.name=secretmanager.googleapis.com --format='value(config.name)' | grep -q secretmanager
```

## Cost

One Secret Manager secret (about $0.06/month per active version, plus $0.03 per 10,000 accesses; Cloud Run reads it when an instance starts). No other new GCP resources; one new revision. Exporting adds a few ms per request (the flush before the response). Epok bills the decoded OTLP bytes: about 1-2 KB per 3-span trace (the Python SDK sends uncompressed protobuf unless OTEL_EXPORTER_OTLP_COMPRESSION=gzip; Epok meters after decompression either way, so gzip saves egress, not Epok cost).

- Default key store: Secret Manager. The secret value is the header string x-api-key=<key>, mounted as OTEL_EXPORTER_OTLP_HEADERS with --update-secrets=OTEL_EXPORTER_OTLP_HEADERS=<secret>:latest; the runtime service account gets roles/secretmanager.secretAccessor on that secret only. The key never appears in the service configuration.
- EPOK_KEY_STORE=env is NOT RECOMMENDED: it sets OTEL_EXPORTER_OTLP_HEADERS=x-api-key=<key> as a plain env var, visible to anyone with run.services.get and in the console.
- Cloud Run throttles CPU outside requests: flush the span processor before the response (the example app does), or deploy with --no-cpu-throttling, or run a collector sidecar. Without one of these the last spans of an idle instance wait for the next request.
- Uninstall removes the OTEL_* variables and the secret mount this setup set, then the binding and the secret. If the service had its own values for them before, restore those by hand.

## Steps

Run in order. Each step runs as `bash -euo pipefail`. Where a check is given, run it first: exit 0 means the step is already done, so skip it. Stop at the first failing step and use the failure modes below.

### 1. `inspect-service` (actor: agent)

Read the service URL and its runtime service account. Prints EPOK_SERVICE_URL and EPOK_RUNTIME_SA.

Idempotent: safe to re-run.

Run:

```bash
URL="$(gcloud run services describe "$CLOUD_RUN_SERVICE" --region="$REGION" --project="$PROJECT_ID" --format='value(status.url)')"
test -n "$URL" || { echo "service $CLOUD_RUN_SERVICE has no URL" >&2; exit 1; }
echo "EPOK_SERVICE_URL=$URL"
SA="$(gcloud run services describe "$CLOUD_RUN_SERVICE" --region="$REGION" --project="$PROJECT_ID" --format='value(spec.template.spec.serviceAccountName)')"
if [ -z "$SA" ]; then
  NUM="$(gcloud projects describe "$PROJECT_ID" --format='value(projectNumber)')"
  SA="$NUM-compute@developer.gserviceaccount.com"   # Cloud Run default runtime identity
fi
echo "EPOK_RUNTIME_SA=$SA"
```

The output contains `NAME=value` lines. Set each one as an environment variable (`EPOK_SERVICE_URL`, `EPOK_RUNTIME_SA`); read the lines, do not `eval` the output.

### 2. `store-key` (actor: agent)

EPOK_KEY_STORE=secretmanager (default): create the Secret Manager secret EPOK_KEY_SECRET whose value is the header string x-api-key=<key> (from stdin, never on the command line). Skipped if the secret exists, and always skipped with EPOK_KEY_STORE=env.

Check (skip the step if this exits 0):

```bash
case "$EPOK_KEY_STORE" in
  env) exit 0 ;;   # nothing to store (not recommended)
  secretmanager) gcloud secrets describe "$EPOK_KEY_SECRET" --project="$PROJECT_ID" --format='value(name)' >/dev/null 2>&1 ;;
esac
```

Run:

```bash
# The secret value is the whole header string, mounted as OTEL_EXPORTER_OTLP_HEADERS. Read from stdin.
python3 -c 'import os, sys; sys.stdout.write("x-api-key=" + os.environ["EPOK_API_KEY"])' | gcloud secrets create "$EPOK_KEY_SECRET" --project="$PROJECT_ID" \
  --replication-policy=automatic --labels="$EPOK_LABELS" --data-file=- --format='value(name)'
```

### 3. `grant-runtime-sa` (actor: agent)

Give the service's runtime service account roles/secretmanager.secretAccessor on that one secret. No-op with EPOK_KEY_STORE=env.

Idempotent: safe to re-run.

Run:

```bash
if [ "$EPOK_KEY_STORE" = env ]; then echo "EPOK_KEY_STORE=env: no IAM change (not recommended)"; exit 0; fi
gcloud secrets add-iam-policy-binding "$EPOK_KEY_SECRET" --project="$PROJECT_ID" \
  --member="serviceAccount:$EPOK_RUNTIME_SA" --role=roles/secretmanager.secretAccessor --format='value(etag)' >/dev/null
echo "$EPOK_RUNTIME_SA may read secret $EPOK_KEY_SECRET (roles/secretmanager.secretAccessor on that secret only)"
```

### 4. `configure-export` (actor: agent)

Set OTEL_SERVICE_NAME, OTEL_TRACES_EXPORTER, OTEL_EXPORTER_OTLP_PROTOCOL and OTEL_EXPORTER_OTLP_ENDPOINT, and mount the secret as OTEL_EXPORTER_OTLP_HEADERS with --update-secrets (default; with EPOK_KEY_STORE=env the header is a plain env var instead). Other variables are kept. Waits for the new revision to be ready.

Idempotent: safe to re-run.

Run:

```bash
python3 -c '
import json, os, subprocess, sys
e = os.environ
variables = {"OTEL_SERVICE_NAME": e["OTEL_SERVICE_NAME"], "OTEL_TRACES_EXPORTER": "otlp",
             "OTEL_EXPORTER_OTLP_PROTOCOL": "http/protobuf", "OTEL_EXPORTER_OTLP_ENDPOINT": e["EPOK_INGEST_HOST"]}
flags = {"--update-env-vars": variables}
if e["EPOK_KEY_STORE"] == "secretmanager":
    flags["--update-secrets"] = {"OTEL_EXPORTER_OTLP_HEADERS": e["EPOK_KEY_SECRET"] + ":latest"}
elif e["EPOK_KEY_STORE"] == "env":
    variables["OTEL_EXPORTER_OTLP_HEADERS"] = "x-api-key=" + e["EPOK_API_KEY"]   # NOT RECOMMENDED
else:
    sys.exit("EPOK_KEY_STORE must be secretmanager or env")
subprocess.run(["gcloud", "run", "services", "update", e["CLOUD_RUN_SERVICE"], "--region=" + e["REGION"],
                "--project=" + e["PROJECT_ID"], "--quiet", "--flags-file=/dev/stdin",
                "--format=value(status.latestReadyRevisionName)"], input=json.dumps(flags), text=True, check=True)
'
```

## Verify

Send one canary request to the Cloud Run service carrying the W3C trace id `00000000$NONCE`:

```bash
URL="$(gcloud run services describe "$CLOUD_RUN_SERVICE" --region="$REGION" --project="$PROJECT_ID" --format='value(status.url)')"
TOKEN="$(gcloud auth print-identity-token --audiences="$URL" 2>/dev/null || gcloud auth print-identity-token)"
curl -fsS -o /dev/null -w "HTTP %{http_code}\n" -H "Authorization: Bearer $TOKEN" \
  -H "traceparent: 00-00000000$NONCE-00f067aa0ba902b7-01" "$URL$EPOK_VERIFY_PATH"
```

Then poll every 10 s for up to 180 s, re-sending the canary every 60 s (a collector may hold a batch until the next request):

```bash
printf 'header = "X-API-Key: %s"\n' "$EPOK_READ_API_KEY" | curl -K - -fsS \
  "$EPOK_API_HOST/api/v1/tenants/$EPOK_TENANT_ID/traces/00000000$NONCE" | python3 -c '
import json, sys
d = json.load(sys.stdin)
spans = [e for es in d.get("services", {}).values() for e in es
         if e.get("fields", {}).get("record_type") == "span"]
ids = {e.get("span_id") for e in spans}
services = sorted({e.get("service") for e in spans})
out = {
    "trace_id": d.get("trace_id"),
    "spans": len(spans),
    "services": services,
    "roots": sum(1 for e in spans if e.get("parent_span_id") not in ids),
    "children": sum(1 for e in spans if e.get("parent_span_id") in ids),
    "error_spans": sum(1 for e in spans if e.get("fields", {}).get("span_status") == "error"),
}
out["ok"] = True if spans and services == [sys.argv[1]] else None
print(json.dumps(out))' "$OTEL_SERVICE_NAME"
```

Success: the JSON field `ok` is not null (the trace is in Epok and every span has service.name = `$OTEL_SERVICE_NAME`). Report the full JSON line: spans, roots, children and error_spans show the trace shape.

## Failure modes

- **permission-denied** (step: inspect-service, configure-export). **STOP AND ASK.** Symptom: PERMISSION_DENIED on describe or update, or 'Permission iam.serviceaccounts.actAs denied'. Check: `gcloud auth list --filter=status:ACTIVE --format='value(account)'`. Fix: The caller lacks roles/run.developer or roles/iam.serviceAccountUser on the runtime service account. Stop and ask a human. Never grant IAM to yourself.
- **revision-not-ready** (step: configure-export). **STOP AND ASK.** Symptom: The update fails with 'Revision ... is not ready' or the container failed to start. Check: `gcloud run revisions list --service="$CLOUD_RUN_SERVICE" --region="$REGION" --project="$PROJECT_ID" --limit=3`. Fix: Setting env vars does not change code, so the app crashed on start for its own reason (read its logs) or an OTEL_* value is malformed. Cloud Run keeps serving the previous revision. Run uninstall to remove the variables and stop and ask.
- **secret-not-available** (step: store-key, grant-runtime-sa, configure-export). **STOP AND ASK.** Symptom: PERMISSION_DENIED / SERVICE_DISABLED on gcloud secrets, or the revision fails with 'Permission denied on secret' / 'secret ... not found'. Check: `gcloud services list --enabled --project="$PROJECT_ID" --filter=config.name=secretmanager.googleapis.com; gcloud secrets get-iam-policy "$EPOK_KEY_SECRET" --project="$PROJECT_ID"`. Fix: The Secret Manager API is disabled, you lack roles/secretmanager.admin, or the runtime service account lacks secretAccessor (re-run grant-runtime-sa). Stop and ask; do not fall back to EPOK_KEY_STORE=env without the human's agreement. 'already been set with a different type': the service already has OTEL_EXPORTER_OTLP_HEADERS as a plain env var; ask before removing it.
- **emit-unauthorized** (step: verify). **STOP AND ASK.** Symptom: The emit curl fails with HTTP 401 or 403 from Cloud Run (not from Epok). Check: `gcloud run services get-iam-policy "$CLOUD_RUN_SERVICE" --region="$REGION" --project="$PROJECT_ID"`. Fix: Your identity lacks roles/run.invoker, or the service sits behind IAP/ingress=internal. Ask the human for a way to send one request (or for them to send it with the traceparent header shown).
- **sdk-not-exporting** (step: verify). Symptom: Requests succeed but the trace never arrives. Check: `gcloud logging read 'resource.type="cloud_run_revision" AND resource.labels.service_name="'"$CLOUD_RUN_SERVICE"'" AND textPayload:("otlp" OR "Failed to export" OR "401")' --project="$PROJECT_ID" --limit=20 --freshness=15m`. Fix: 'Failed to export ... 401/403' = key (see epok-auth-rejected). No export log lines at all = the app has no OpenTelemetry SDK, or it does not read OTEL_* variables, or spans are buffered under CPU throttling (flush before the response). Stop and ask before changing application code.
- **epok-auth-rejected** (step: verify). **STOP AND ASK.** Symptom: Spans never arrive; exporter or collector logs show HTTP 401 or 403 from the Epok ingest host. Check: `Ask the human which key they gave you and its scope (do not print it).`. Fix: The ingest key is wrong, revoked, or not ingest scope. Stop and ask the human for a new ingest-scope key. Do not retry in a loop.
- **verify-read-auth** (step: verify). **STOP AND ASK.** Symptom: The verify call fails with HTTP 401 or 403 on every poll. Check: `Does EPOK_READ_API_KEY have read scope, and is EPOK_TENANT_ID its tenant?`. Fix: 403 = the key lacks read scope or belongs to another tenant. Stop and ask the human.
- **trace-not-found** (step: verify). Symptom: The verify call keeps failing with HTTP 404 (Trace not found) until the timeout. Check: `Re-read the emit output: did the request/invocation succeed? Then check the failure modes for this runtime.`. Fix: The canary trace did not reach Epok, or your code does not continue an incoming W3C traceparent (then spans arrive under another trace id: look for OTEL_SERVICE_NAME in Epok -> Traces). Report; do not change sampling or keys to make it pass.
- **wrong-service-name** (step: verify). **STOP AND ASK.** Symptom: Verify finds the trace but ok is null and services is not [OTEL_SERVICE_NAME]. Check: `Look at services in the verify JSON.`. Fix: Something else sets service.name (OTEL_RESOURCE_ATTRIBUTES or code). Report it to the human; do not edit application code without asking.

## Uninstall (only when the user asks)

Each step tolerates resources that are already gone. Continue past a failing step and report it.

### U1. `remove-export-config` DESTRUCTIVE

Remove the OTEL_* variables and the OTEL_EXPORTER_OTLP_HEADERS secret mount this setup set and roll a new revision. A missing service counts as done.

```bash
if ! gcloud run services describe "$CLOUD_RUN_SERVICE" --region="$REGION" --project="$PROJECT_ID" --format='value(metadata.name)' >/dev/null 2>&1; then
  echo "service $CLOUD_RUN_SERVICE not found; nothing to do"; exit 0
fi
gcloud run services update "$CLOUD_RUN_SERVICE" --region="$REGION" --project="$PROJECT_ID" --quiet \
  --remove-env-vars=OTEL_SERVICE_NAME,OTEL_TRACES_EXPORTER,OTEL_EXPORTER_OTLP_PROTOCOL,OTEL_EXPORTER_OTLP_ENDPOINT"$([ "$EPOK_KEY_STORE" = env ] && echo ,OTEL_EXPORTER_OTLP_HEADERS)" \
  $([ "$EPOK_KEY_STORE" = env ] || echo --remove-secrets=OTEL_EXPORTER_OTLP_HEADERS) \
  --format='value(status.latestReadyRevisionName)'
```

### U2. `remove-secret-access` DESTRUCTIVE

Remove the runtime service account's secretAccessor binding on the secret. Not run while the service may still mount it.

```bash
if [ "$EPOK_KEY_STORE" = env ]; then echo "EPOK_KEY_STORE=env: nothing to unbind"; exit 0; fi
# Uninstall may run in a fresh session: derive the runtime SA again (same rule as inspect-service).
SA="${EPOK_RUNTIME_SA:-}"
if [ -z "$SA" ] && gcloud run services describe "$CLOUD_RUN_SERVICE" --region="$REGION" --project="$PROJECT_ID" --format='value(metadata.name)' >/dev/null 2>&1; then
  SA="$(gcloud run services describe "$CLOUD_RUN_SERVICE" --region="$REGION" --project="$PROJECT_ID" --format='value(spec.template.spec.serviceAccountName)')"
  if [ -z "$SA" ]; then SA="$(gcloud projects describe "$PROJECT_ID" --format='value(projectNumber)')-compute@developer.gserviceaccount.com"; fi
fi
if [ -z "$SA" ]; then echo "service gone and EPOK_RUNTIME_SA unset: the binding is removed with the secret by delete-key-secret"; exit 0; fi
if OUT="$(gcloud secrets remove-iam-policy-binding "$EPOK_KEY_SECRET" --project="$PROJECT_ID" --member="serviceAccount:$SA" --role=roles/secretmanager.secretAccessor 2>&1)"; then echo "binding removed"
elif grep -qiE "not found|NOT_FOUND|does not have" <<< "$OUT"; then echo "binding or secret already gone"; else echo "$OUT" >&2; exit 1; fi
```

### U3. `delete-key-secret` DESTRUCTIVE

Delete the Secret Manager secret (already gone counts as done). Skip this step if another service still mounts it.

```bash
if [ "$EPOK_KEY_STORE" = env ]; then echo "EPOK_KEY_STORE=env: no secret to delete"; exit 0; fi
if OUT="$(gcloud secrets delete "$EPOK_KEY_SECRET" --project="$PROJECT_ID" --quiet 2>&1)"; then echo "deleted secret $EPOK_KEY_SECRET"
elif grep -qiE "not found|NOT_FOUND" <<< "$OUT"; then echo "secret already gone"; else echo "$OUT" >&2; exit 1; fi
```

## Final report

Finish with: manifest id and sha256, result, each step's status, the verify JSON line, and any failure-mode ids. Never include a key.
