epok
View as Markdown

Running on AWS

Updated Nov 18, 2026 · today

This page says what to install for each AWS compute service, and what we measured when we ran it in a real AWS account. Three kinds of path exist:

  • An agent you run (the OpenTelemetry Collector). It sends OTLP, which the hosted service accepts today.
  • No agent: CloudWatch Logs and CloudWatch metrics reach Epok through Amazon Data Firehose. Status: not yet available on Epok's hosted service. Use OpenTelemetry (OTLP) for now.
  • Account changes from CloudTrail, read through a read-only IAM role: AWS CloudTrail change events. Status: not yet available on Epok's hosted service. Use OpenTelemetry (OTLP) for now.

What is supported

ServiceLogsTracesMetricsChange eventsHow it arrives
EC2YesYesYesNot listedOpenTelemetry Collector on the instance
CloudWatch LogsYes (not yet on the hosted service)Not listedNot listedNot listedSubscription filter and Amazon Data Firehose
CloudWatch metricsNot listedNot listedYes (not yet on the hosted service)Not listedCloudWatch Metric Streams and Firehose
LambdaYes (not yet on the hosted service)YesNot listedNot listedCloudWatch Logs and Firehose; OpenTelemetry Lambda layers
ECS on FargateNot listedYesNot listedNot listedOpenTelemetry Collector sidecar
CloudTrailNot listedNot listedNot listedYes (not yet on the hosted service)Read-only IAM role (CloudTrail lookup)

On this page: Choose a pathEC2CloudWatch Logs (no agent)ECSLambdaEKSKnown limits

Choose a path

Where it runsLogsMetricsTraces
EC2OTel Collector on the host (tested), or CloudWatch Logs + Firehose (tested, not yet hosted)OTel Collector host metrics (tested)OTLP to the Collector on the host (tested)
ECS on Fargateawslogs driver to CloudWatch Logs, then Firehose (not yet hosted; not yet tested from an ECS task)CloudWatch metric stream (tested 2026-11-09, not yet hosted)OTel Collector sidecar (setup manifest)
LambdaCloudWatch Logs + Firehose (tested with a real function, not yet hosted)CloudWatch metric stream (tested 2026-11-09, not yet hosted)OpenTelemetry Lambda layers (setup manifest)
EKSEpok Helm chart (tested on kind, not on EKS)Epok Helm chart (tested on kind, not on EKS)OTLP to the chart's collector

EC2: the OpenTelemetry Collector on the host

One Collector per instance reads your log files, collects host metrics (CPU, memory, disk, filesystem, network, load) and forwards OTLP from apps on the same host. The resourcedetection processor reads the instance ID, type, region, zone and account from the instance metadata service, so the instance needs no IAM role for it. Today they reach Epok on host metrics only, not on logs or spans (see Known limits).

Install on Amazon Linux 2023 (arm64 shown; use amd64 on x86). The Collector project publishes no 0.123.0 Linux packages; use 0.123.1.

bash
V=0.123.1; ARCH=arm64   # amd64 on x86 instances
curl -fsSLo otelcol-contrib.rpm \
  "https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v${V}/otelcol-contrib_${V}_linux_${ARCH}.rpm"
sudo rpm -i otelcol-contrib.rpm

Put the key in the service's environment file, readable by root only. Do not put it in the configuration file.

bash
read -rsp "Epok ingest key: " EPOK_INGEST_KEY; echo
echo "EPOK_INGEST_KEY=$EPOK_INGEST_KEY" | sudo tee -a /etc/otelcol-contrib/otelcol-contrib.conf >/dev/null
sudo chmod 600 /etc/otelcol-contrib/otelcol-contrib.conf
unset EPOK_INGEST_KEY

Then write /etc/otelcol-contrib/config.yaml and run sudo systemctl restart otelcol-contrib:

yaml
# /etc/otelcol-contrib/config.yaml for a VM (EC2 or Compute Engine) sending to Epok:
# app log files + host metrics + OTLP from local apps.
# The key comes from the environment (EPOK_INGEST_KEY), never this file.
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 127.0.0.1:4317
      http:
        endpoint: 127.0.0.1:4318
  filelog:
    include: [/var/log/myapp/*.log]
    start_at: end
    # Plain-text lines carry no OTel severity; without this every line is
    # stored as level=info. Take the first level word in the line.
    operators:
      - type: regex_parser
        regex: '\b(?P<sev>TRACE|DEBUG|INFO|WARN|WARNING|ERROR|FATAL)\b'
        on_error: send_quiet
        severity:
          parse_from: attributes.sev
      - type: remove
        field: attributes.sev
        if: 'attributes.sev != nil'
  hostmetrics:
    collection_interval: 60s
    scrapers:
      cpu: {}
      memory: {}
      load: {}
      disk: {}
      filesystem: {}
      network: {}

processors:
  resourcedetection:
    detectors: [env, ec2, system]
    timeout: 5s
    override: false
  resource/service:
    attributes:
      - key: service.name
        value: myapp
        action: insert
  batch: {}

exporters:
  otlphttp/epok:
    endpoint: https://ingest.epok.cloud
    compression: gzip
    headers:
      x-api-key: "${env:EPOK_INGEST_KEY}"

service:
  pipelines:
    logs:
      receivers: [otlp, filelog]
      processors: [resourcedetection, resource/service, batch]
      exporters: [otlphttp/epok]
    metrics:
      receivers: [otlp, hostmetrics]
      processors: [resourcedetection, batch]
      exporters: [otlphttp/epok]
    traces:
      receivers: [otlp]
      processors: [resourcedetection, batch]
      exporters: [otlphttp/epok]

Measured on a t4g.small, Amazon Linux 2023

  • 60 of 60 lines written to the log file were stored once, with service=myapp.
  • One OTLP log and one OTLP span sent by an app to the local Collector were stored.
  • 22 host metric names arrived, each labeled with host_id (the instance ID), cloud_region, cloud_availability_zone and cloud_account_id.
  • Without the regex_parser step, every plain-text line was stored as level=info, including lines that say ERROR. With it, the level comes from the line.

CloudWatch Logs without an agent (Firehose)

Status: not yet available on Epok's hosted service. Use OpenTelemetry (OTLP) for now.

Anything that already writes to CloudWatch Logs (Lambda, ECS with the awslogs driver, the CloudWatch agent on EC2, API Gateway, RDS) can reach Epok through one CloudFormation stack: a subscription filter per log group, a Firehose stream, and an S3 bucket for what Firehose cannot deliver. Setup, checks, troubleshooting and removal are on AWS: logs and metrics.

Measured against a real Firehose stream

  • Stack creation: 160 s. First canary event stored 17 s after it was written.
  • Latency, with the buffer interval at 0: 60 of 60 events within 60 s; median 5.2 s, 95th percentile 7.5 s, slowest 8.4 s. At the default 60 s interval, add up to a minute.
  • 1,000 of 1,000 events stored, once each.
  • When Epok answered 429 or 503, Firehose re-sent the same request about 2 s later and every event was stored once. When Epok stored a request but the answer was lost (a 500 after the write), Firehose re-sent it and Epok stored nothing twice and billed it once.
  • Billing: the bytes Epok counted equalled the decoded bytes Firehose sent and the copy in the stack's S3 bucket, to the byte.
  • A wrong key is refused with a 403 whose message Firehose writes to the stack's delivery-error log group. Another account's key never sees the data.

Each event gets service from the last part of its log group name, log_group, log_stream, cloud.provider, cloud.account_id and cloud.region. The level is the first of FATAL or CRITICAL, ERROR, WARN, DEBUG and TRACE found as a word in the message, else info. So a web access-log line with status 502 is stored as info.

ECS on Fargate

Logs: keep the awslogslog driver and add the task's log group (for example /ecs/api, stored as service=api) to the Firehose stack above. We tested this stack with Lambda and EC2 log groups, not yet with an ECS task.

Traces: add an OpenTelemetry Collector sidecar to the task definition. The steps, as a machine-readable manifest an agent or a human can run: aws-ecs-otel-traces.json (skill: SKILL.md). The key is stored as an SSM SecureString and injected into the sidecar only.

Lambda

Logs: add /aws/lambda/<function> to the Firehose stack. Measured with a real Python function: every line it printed or logged was stored once with service=<function name>; a line logged with logging.error was stored as level=error.

Traces: the OpenTelemetry Lambda layers (Python instrumentation plus the Collector extension). Manifest: aws-lambda-otel-traces.json (skill: SKILL.md). The key is stored in Secrets Manager and read by the Collector extension at start.

EKS

Use the Epok Helm chart: Kubernetes. We have run it on kind, not yet on EKS. Logs that EKS writes to CloudWatch (the control plane, or Container Insights) can also go through the Firehose stack, but they arrive with service set to the last part of the log group name (for example application), not the pod's app.

Known limits (measured)

  • Firehose stack: serviceis the last part of the log group name. For the CloudWatch agent's default group names (the file path, such as /var/log/nginx/access) that is access; set log_group_name in the agent to /ec2/<app> instead.
  • Firehose stack: to add a log group to an existing stack, run aws cloudformation update-stack with the template file again (--template-body file://epok-integration.yaml, from https://epok.cloud/setup/aws/epok-integration.yaml), the full LogGroupNames list, and UsePreviousValue=true for every other parameter. With --use-previous-templatenothing changes ("No updates are to be performed"): AWS stores the template with its Fn::ForEach already expanded. Re-running aws-logs.sh does not add groups either; it leaves an existing stack as it is.
  • Collector: the instance, region and account fields are on host metrics only (as the labels host_id, cloud_region, cloud_account_id). OTLP log records and spans keep service, host and the Kubernetes fields, but not these, until the ingest change that adds them to logs and spans is switched on. Then they are stored as cloud.region, cloud.account_id and host_id, the same cloud.* names the Firehose rows carry.