Install on Kubernetes (Helm-packaged OpenTelemetry Collector)

Prerequisites

• A Kubernetes cluster, Helm 3, and kubectl configured for it. Your installer account must be able to create the namespace, workloads, Secret, and read-only RBAC bindings.
• An ingest-scoped API key from Settings → API Keys. An ingest key is created automatically on signup.
• Logs and events work on any kernel supported by your Kubernetes setup; they have no eBPF kernel or BTF requirement. The eBPF service map requires Linux nodes with kernel >=5.8 and BTF. Check the nodes, not your laptop:

# Run on each Linux node that will run the eBPF collector.
uname -r                         # Must be >=5.8
test -r /sys/kernel/btf/vmlinux && echo "Kernel BTF is available"

Pod Security Standards restrictedand OpenShift SCC policies may reject the eBPF pod's hostNetwork, hostPID, or capabilities. A policy exception may be needed. The log collector also needs host-path mounts. Review what the Epok agent can and can't see with your platform team before installing.

Recommended: logs, events, and the eBPF service map

Install chart epok-collector version 0.2.0 from the published Epok Helm repository. Replace <YOUR_INGEST_KEY> with your key, keeping the quotes.

helm repo add epok https://charts.getepok.dev
helm repo update
helm install epok epok/epok-collector --version 0.2.0 \
  --namespace epok --create-namespace \
  --set apiKey='<YOUR_INGEST_KEY>' \
  --set ebpf.enabled=true

Logs-only variant (no eBPF)

Choose this alternative if your nodes lack the kernel/BTF support or your cluster policy does not allow eBPF. Omit ebpf.enabled; it defaults to false. Kubernetes events remain enabled. For strictly container logs only, also add --set events.enabled=false. Run one install variant, since both use the release name epok.

# Alternative install after adding/updating the Helm repo above.
helm install epok epok/epok-collector --version 0.2.0 \
  --namespace epok --create-namespace \
  --set apiKey='<YOUR_INGEST_KEY>'

What to expect

• Logs: about 30 seconds after the collectors are running and your applications emit logs. Open Epok → Live Tail to verify them. Kubernetes events arrive as log records as the cluster emits them.
• Service map: a periodic cycle, default ~6 hours. Even when network metrics are arriving, the map may take up to that long to first appear. It needs observed traffic between workloads. Log arrival does not mean the map has been built yet.

Verify and troubleshoot the bundle

These commands use the release and namespace from the install above. Running pods alone do not prove successful delivery; check each collector's own logs for authentication and export errors.

kubectl -n epok get pods

# Vector container logs
kubectl -n epok logs daemonset/epok-epok-collector -c vector --tail=100

# Kubernetes events
kubectl -n epok logs deployment/epok-epok-collector-events --tail=100

# eBPF network metrics (when enabled)
kubectl -n epok logs daemonset/epok-epok-collector-ebpf -c beyla --tail=100

• Nothing arrives: check the ingest key and outbound HTTPS access to ingest.getepok.dev. Container logs go to https://ingest.getepok.dev/api/v1/ingest; eBPF metrics go to https://ingest.getepok.dev/v1/metrics. The events collector sends OTLP log records to https://ingest.getepok.dev/v1/logs.
• No eBPF pod or permission errors: check the node kernel and BTF, then kubectl -n epok get events --sort-by=.lastTimestamp for scheduling or admission failures. If capabilities alone cannot load eBPF, --set ebpf.privileged=true is a fallback on the bundle install. It grants broad host access and may still require a policy exception; it does not supply a missing kernel feature. The default is privileged: false.
• Logs arrive but the map is empty: check the eBPF collector for export errors and confirm traffic between workloads. Allow for the default ~6-hour map-building cycle. If a full cycle has elapsed with traffic and successful exports, contact support with the collector errors and install details, excluding your key.
• Using existingSecret: the Secret must contain api-key with the raw ingest key and, with eBPF enabled, otlp-header with the complete value x-api-key=<same key>. The chart creates both automatically when you use apiKey instead.

# Create this in the epok namespace before installing with existingSecret.
kubectl create namespace epok
kubectl -n epok create secret generic epok-credentials \
  --from-literal=api-key='<YOUR_INGEST_KEY>' \
  --from-literal=otlp-header='x-api-key=<YOUR_INGEST_KEY>'

helm install epok epok/epok-collector --version 0.2.0 \
  --namespace epok \
  --set existingSecret=epok-credentials \
  --set ebpf.enabled=true

Skip namespace creation if epok already exists. Replace both placeholders with the same key. As the chart NOTES explain, rotate both Secret entries together and restart the collectors after changing them.

1. Store the API key as a secret

kubectl create namespace epok
kubectl -n epok create secret generic epok-credentials \
  --from-literal=api-key=epk_REPLACE_ME

2. Vector config (ConfigMap)

# epok-vector-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: epok-vector-config
  namespace: epok
data:
  vector.yaml: |
    sources:
      kube_logs:
        type: kubernetes_logs
        # Vector auto-discovers pods, attaches namespace/pod/container
        # labels, and parses the CRI/JSON log format. No extra config
        # needed for the common case.
        glob_minimum_cooldown_ms: 200

    transforms:
      enrich:
        type: remap
        inputs: [kube_logs]
        source: |
          # Map the kubernetes_logs fields to Epok's expected shape.
          .service = .kubernetes.container_name
          .namespace = .kubernetes.pod_namespace
          .pod = .kubernetes.pod_name
          .node = .kubernetes.pod_node_name
          ._msg = .message
          del(.message)

    sinks:
      epok:
        type: elasticsearch
        inputs: [enrich]
        endpoints: ["https://ingest.getepok.dev"]
        bulk:
          index: logs
        # Required. Epok is not a real Elasticsearch: the default
        # api_version "auto" sniffs the version against our root (404) and the
        # sink then never starts, so nothing ships and nothing says why.
        api_version: v8
        healthcheck:
          enabled: false
        auth:
          strategy: basic
          user: ${EPOK_API_KEY}
          password: x
        # Sensible defaults for k8s log volume.
        batch:
          max_events: 1000
          timeout_secs: 5
        buffer:
          type: memory
          max_events: 5000
          when_full: drop_newest

3. RBAC for Vector

Vector needs read access to pods + namespaces to enrich log entries with metadata.

# epok-vector-rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: epok-vector
  namespace: epok
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: epok-vector
rules:
- apiGroups: [""]
  resources: ["pods", "namespaces", "nodes"]
  verbs: ["list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: epok-vector
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: epok-vector
subjects:
- kind: ServiceAccount
  name: epok-vector
  namespace: epok

4. DaemonSet

# epok-vector-daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: epok-vector
  namespace: epok
spec:
  selector:
    matchLabels: { app: epok-vector }
  template:
    metadata:
      labels: { app: epok-vector }
    spec:
      serviceAccountName: epok-vector
      tolerations:
      - operator: Exists                  # Run on every node, incl. tainted ones
      containers:
      - name: vector
        # Pin to a specific version, not :latest, so a Vector release
        # never breaks your log pipeline. Bump deliberately after testing.
        image: timberio/vector:0.43.0-alpine
        env:
        - name: EPOK_API_KEY
          valueFrom:
            secretKeyRef:
              name: epok-credentials
              key: api-key
        resources:
          requests: { cpu: 50m,  memory: 128Mi }
          limits:   { cpu: 500m, memory: 512Mi }
        volumeMounts:
        - { name: config,   mountPath: /etc/vector }
        - { name: var-log,  mountPath: /var/log,           readOnly: true }
        - { name: var-lib,  mountPath: /var/lib/docker,    readOnly: true }
        - { name: data,     mountPath: /vector-data }
      volumes:
      - { name: config,   configMap: { name: epok-vector-config } }
      - { name: var-log,  hostPath:  { path: /var/log } }
      - { name: var-lib,  hostPath:  { path: /var/lib/docker } }
      - { name: data,     emptyDir: {} }

5. Apply everything

kubectl apply -f epok-vector-config.yaml
kubectl apply -f epok-vector-rbac.yaml
kubectl apply -f epok-vector-daemonset.yaml

# Watch the pods come up
kubectl -n epok rollout status daemonset/epok-vector

# Tail one of the agents to confirm
kubectl -n epok logs -l app=epok-vector --tail=20 -f

Verify

1. Open app.getepok.dev → Live Tail. Container logs should stream in within 30–60 s, tagged with namespace, pod, container.
2. Trigger a failure: kubectl delete pod <some-pod> and force a CrashLoopBackOff. Within ~2 minutes the K8s Intelligence detector fires an alert in New Errors.
3. Open Services. Each unique container_name appears as a service card with hit counts.

When the key is wrong, nothing says so

This is the failure mode to internalize before you go looking anywhere else. The DaemonSet carries the key in the sink config, and we told you to set healthcheck.enabled: false above (you have to — the sink otherwise never starts). The consequence is that Vector comes up healthy with a bad key. The pods are Running, the rollout succeeded, kubectl get podsis clean, and Epok shows nothing — because the rejection is happening one layer down, on every batch, and the only place it is written down is the agent's own log.

# The sink errors live in the agent's OWN log, not in Epok.
kubectl -n epok logs -l app=epok-vector --tail=200 | grep -iE 'error|401|403|unauthorized|forbidden'

# Check the secret for a trailing newline — the #1 cause of a 403 here.
# --from-literal is clean; --from-file keeps the file's final newline.
kubectl -n epok get secret epok-credentials \
  -o jsonpath='{.data.api-key}' | base64 -d | xxd | tail -2

The newline trap is specific to this setup. The Vector config authenticates with strategy: basic, which sends the key as the HTTP Basic username. Epok trims whitespace around a key sent via X-API-Key or Bearer, but the username parsed out of a Basic credential is used as-is — so any stray whitespace that survives into that username hashes to something that is not your key, and you get a 403 rather than a 401. --from-literal (step 1) stores the key clean; --from-file keeps whatever trailing newline the file had, which is why the xxd check above is worth ten minutes of staring at YAML. Use printf, not echo, if you must write that file.

To separate a key problem from a config problem, take the cluster out of the loop and hit the same endpoint the sink hits:

curl -i -X POST https://ingest.getepok.dev/_bulk \
  -H 'Content-Type: application/x-ndjson' \
  -H 'X-API-Key: YOUR_API_KEY' \
  --data-binary $'{"create":{}}\n{"_msg":"auth probe","service":"probe"}\n'

# 200 {"status":"ok","accepted":1}            key + header are both fine
# 401 {"error":"unauthorized","detail":...}   no key found in ANY accepted header
# 403 {"error":"forbidden","detail":...}      a key WAS found, and rejected
# 429 {"error":"rate_limited",...}            key is FINE — quota or rate limit

401 and 403 mean different things. 401 is “no credential found in any accepted header” — the key never made it out of the secret, or the sink is sending a header name Epok does not read. 403 is “a key was read, and rejected” — rotated, invalid (the newline case), missing the ingest scope, or the tenant disabled. The detail field names which one. A quota or plan gate is not a 403: an expired trial or a blown daily volume cap answers 429 with a retry_after, which means your key is good and your billing is not. Every accepted credential form is in Authentication; the plan gates are in Limits & error codes.

Common gotchas

• Containerd vs Docker. Modern clusters use containerd — logs live under /var/log/containers/ (already mounted above). The /var/lib/docker mount is a no-op on pure-containerd nodes and harmless to leave in.
• PodSecurity / Pod Security Admission. Host-path mounts are disallowed by both baseline and restricted policies. Arrange an appropriate policy exception with your platform team for this collector namespace.
• EKS + Fargate.Fargate nodes can't run DaemonSets. Use Fluent Bit's built-in EKS Fargate logging instead — config snippet in the AWS guide.
• Resource limits. The 500 m CPU / 512 Mi memory cap is right for ~5 k logs/sec/node. Bump if you see Vector OOMing under heavy log bursts.

Kubernetes metrics (CPU, memory, pods)

The Vector DaemonSet above ships logs. To get node and pod CPU/memory, restart counts, and pod phase — the signals the metric detectors and the per-service infrastructure panel read — run the OpenTelemetry Collector as a DaemonSet with the kubeletstats and k8s_cluster receivers. Add this to the collector ConfigMap:

receivers:
  kubeletstats:                       # per-node + per-pod resource usage
    collection_interval: 30s
    auth_type: serviceAccount
    endpoint: https://${env:K8S_NODE_NAME}:10250
    insecure_skip_verify: true
    metric_groups: [node, pod, container, volume]
  k8s_cluster:                        # cluster-level state (pod phase, restarts)
    collection_interval: 30s
processors:
  # The Vector log DaemonSet above keys logs by container name as `service`.
  # Stamp the same onto metrics so pod CPU/memory JOIN the right service —
  # kubeletstats leaves service unset, so without this the metrics land with no
  # service and never correlate with the logs on the service page.
  transform/service:
    metric_statements:
      - context: resource
        statements:
          - set(attributes["service.name"], attributes["k8s.container.name"]) where attributes["k8s.container.name"] != nil
exporters:
  otlphttp/epok:
    metrics_endpoint: https://ingest.getepok.dev/v1/metrics
    encoding: json                    # set this — correct against every build
    headers: { Authorization: "Bearer YOUR_API_KEY" }
service:
  pipelines:
    metrics:
      receivers: [kubeletstats, k8s_cluster]
      processors: [transform/service]
      exporters: [otlphttp/epok]

The collector's ServiceAccount needs read access to the kubelet (/stats/summary). Once metrics arrive, the metric detectors (saturation, anomaly, slow drift, reporting gap) switch on automatically for the cluster, and pod CPU/memory appear on each service page. For cluster-wide series like Pending pods or restart storms, add kube-state-metrics.

Your first query

Shipping is not the finish line — reading it back is. The enrich transform in step 2 puts a namespace on every line the DaemonSet sends, so namespace:* (the field exists) is a clean way to ask “show me everything that arrived from the cluster.” Your tenant id is in Settings → API Keys, and in the app URL.

curl -X POST https://app.getepok.dev/api/v1/tenants/YOUR_TENANT_ID/search \
  -H 'Authorization: Bearer YOUR_API_KEY' \
  -H 'Content-Type: application/json' \
  -d '{
  "query": "namespace:*",
  "start": "-1h",
  "end": "now",
  "limit": 100
}'

A good result is a logs array of up to 100 entries, each carrying _msg, _time, and the service / namespace / pod / node fields the remap set — plus a count and an elapsed_ms. If count is 0 but the agent logs are clean, widen start to -24h before concluding anything: an idle cluster genuinely may not have logged in the last hour. The request body takes exactly five fields — query, start, end, limit, and a before timestamp cursor for paging backward.

The more useful second query is a breakdown — which pods are actually talking, and how much:

{
  "query": "namespace:* | stats by (pod) count()",
  "start": "-1h",
  "end": "now",
  "limit": 100
}

In the product this is Explore — same syntax, same results, with the field list down the side. Note that container logs arrive without a level unless your application emits one as a JSON field, so level:error across a whole cluster is usually a thinner filter than people expect; the full grammar, and how to work around that, is in Search syntax.