epok
View as Markdown

Running on Google Cloud

Updated Nov 18, 2026 · today

This page says what to install for each Google Cloud compute service, and what we measured when we ran it in a real project. Two kinds of path exist:

  • An agent you run (the OpenTelemetry Collector). It sends OTLP, which the hosted service accepts today.
  • No agent: a Cloud Logging sink sends entries to a Pub/Sub topic, and a push subscription delivers them to Epok with a Google-signed token. Status: not yet available on Epok's hosted service. Use OpenTelemetry (OTLP) for now.

What is supported

ServiceLogsTracesMetricsChange eventsHow it arrives
Compute EngineYesYesYesNot listedOpenTelemetry Collector on the VM
Cloud LoggingYes (not yet on the hosted service)Not listedNot listedNot listedLogs Router sink and Pub/Sub push (OIDC)
Cloud RunYes (not yet on the hosted service)YesNot listedNot listedCloud Logging and Pub/Sub push; OpenTelemetry SDK

On this page: Choose a pathCompute EngineCloud Logging (no agent)Cloud RunGKEKnown limits

Choose a path

Where it runsLogsMetricsTraces
Compute EngineOTel Collector on the VM (tested), or Cloud Logging + Pub/Sub push (tested, not yet hosted)OTel Collector host metrics (tested)OTLP to the Collector on the VM (tested)
Cloud RunCloud Logging + Pub/Sub push (tested with a real service, not yet hosted)Not yet (Cloud Monitoring pull is built, off)OpenTelemetry SDK, setup manifest
GKEEpok Helm chart (tested on kind, not on GKE), or Cloud Logging + Pub/Sub pushEpok Helm chart (tested on kind, not on GKE)OTLP to the chart's collector

Compute Engine: the OpenTelemetry Collector on the VM

One Collector per VM reads your log files, collects host metrics (CPU, memory, disk, filesystem, network, load) and forwards OTLP from apps on the same VM. The resourcedetection processor reads the instance ID, machine type, zone and project from the metadata server, so the VM needs no service account for it. Today they reach Epok on host metrics only, not on logs or spans (see Known limits).

Install on Debian 12. The Collector project publishes no 0.123.0 Linux packages; use 0.123.1.

bash
V=0.123.1; ARCH=amd64   # arm64 on Arm (Tau T2A, C4A) instances
curl -fsSLo otelcol-contrib.deb \
  "https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v${V}/otelcol-contrib_${V}_linux_${ARCH}.deb"
sudo dpkg -i otelcol-contrib.deb

Put the key in the service's environment file, readable by root only:

bash
read -rsp "Epok ingest key: " EPOK_INGEST_KEY; echo
echo "EPOK_INGEST_KEY=$EPOK_INGEST_KEY" | sudo tee -a /etc/otelcol-contrib/otelcol-contrib.conf >/dev/null
sudo chmod 600 /etc/otelcol-contrib/otelcol-contrib.conf
unset EPOK_INGEST_KEY

Then write /etc/otelcol-contrib/config.yaml and run sudo systemctl restart otelcol-contrib:

yaml
# /etc/otelcol-contrib/config.yaml for a VM (EC2 or Compute Engine) sending to Epok:
# app log files + host metrics + OTLP from local apps.
# The key comes from the environment (EPOK_INGEST_KEY), never this file.
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 127.0.0.1:4317
      http:
        endpoint: 127.0.0.1:4318
  filelog:
    include: [/var/log/myapp/*.log]
    start_at: end
    # Plain-text lines carry no OTel severity; without this every line is
    # stored as level=info. Take the first level word in the line.
    operators:
      - type: regex_parser
        regex: '\b(?P<sev>TRACE|DEBUG|INFO|WARN|WARNING|ERROR|FATAL)\b'
        on_error: send_quiet
        severity:
          parse_from: attributes.sev
      - type: remove
        field: attributes.sev
        if: 'attributes.sev != nil'
  hostmetrics:
    collection_interval: 60s
    scrapers:
      cpu: {}
      memory: {}
      load: {}
      disk: {}
      filesystem: {}
      network: {}

processors:
  resourcedetection:
    detectors: [env, gcp, system]
    timeout: 5s
    override: false
  resource/service:
    attributes:
      - key: service.name
        value: myapp
        action: insert
  batch: {}

exporters:
  otlphttp/epok:
    endpoint: https://ingest.epok.cloud
    compression: gzip
    headers:
      x-api-key: "${env:EPOK_INGEST_KEY}"

service:
  pipelines:
    logs:
      receivers: [otlp, filelog]
      processors: [resourcedetection, resource/service, batch]
      exporters: [otlphttp/epok]
    metrics:
      receivers: [otlp, hostmetrics]
      processors: [resourcedetection, batch]
      exporters: [otlphttp/epok]
    traces:
      receivers: [otlp]
      processors: [resourcedetection, batch]
      exporters: [otlphttp/epok]

Measured on an e2-small, Debian 12

  • 60 of 60 lines written to the log file were stored once, with service=myapp.
  • One OTLP log and one OTLP span sent by an app to the local Collector were stored.
  • 22 host metric names arrived, each labeled with host_id, cloud_region, cloud_availability_zone, cloud_account_id (the project ID) and host_type.
  • Without the regex_parser step, every plain-text line was stored as level=info, including lines that say ERROR.

Cloud Logging without an agent (Pub/Sub push)

Status: not yet available on Epok's hosted service. Use OpenTelemetry (OTLP) for now.

Anything that writes to Cloud Logging (Cloud Run, GKE, Compute Engine platform logs, audit logs) can reach Epok through a project sink, a Pub/Sub topic with a dead-letter topic, and a push subscription that signs each request with a Google OIDC token. Setup with one command or Terraform, checks, troubleshooting and removal are on GCP: Cloud Logging.

Measured against real Pub/Sub push

  • Setup: 80 s. A new sink drops entries for its first minutes while Google propagates its permission; the first entry arrived 180 s after setup. Setup's data check re-sends its canary until it arrives.
  • Latency: 60 of 60 entries within 60 s; median 1.9 s, 95th percentile 6.0 s, slowest 17.5 s.
  • 1,000 of 1,000 entries stored. When we made Pub/Sub deliver all 1,000 again (a seek), Epok counted 1,000 duplicates and stored none of them.
  • Billing: the bytes Epok counted equalled the decoded payloads of the 1,063 messages, to the byte.
  • Removal: 94 s; no sink, topic, subscription or service account left.

Cloud Run

Status: Cloud Run logs through Cloud Logging and Pub/Sub push: not yet available on Epok's hosted service. Use OpenTelemetry (OTLP) for now. The traces path below uses OTLP.

Logs:Cloud Run writes request logs and your container's stdout and stderr to Cloud Logging. Route them with the sink above (for one service, a filter such as resource.type="cloud_run_revision" AND resource.labels.service_name="checkout"). Each entry is stored with service set to the Cloud Run service name. A Cloud Run request log carries no message of its own, only the request; Epok stores it with the message GET https://<service URL>/path 200 3.197 ms (method, URL without the query string, status, latency) and the http.* fields. Measured on a real Cloud Run service: 19 of 20 request logs stored, none rejected.

Traces: the OpenTelemetry SDK in the container, exporting OTLP to Epok with the key from Secret Manager. Manifest: gcp-cloudrun-otel-traces.json (skill: SKILL.md).

GKE

Use the Epok Helm chart: Kubernetes. We have run it on kind, not yet on GKE. Container logs that GKE already sends to Cloud Logging can also go through the sink; they arrive with service set to the container name.

Known limits (measured)

  • One project per sink; folders and organizations are not set up for you.
  • Pub/Sub push sends one message per log entry, and Pub/Sub bills at least 1 KB per message.
  • Collector: the instance, region and account fields are on host metrics only (as the labels host_id, cloud_region, cloud_account_id). OTLP log records and spans keep service, host and the Kubernetes fields, but not these, until the ingest change that adds them to logs and spans is switched on. Then they are stored as cloud.region, cloud.account_id and host_id, the same cloud.* names the Pub/Sub rows carry.