Skip to content

security: limit Alloy's Docker access, mount scripts read-only, refuse Loki deletes - #526

Open
OisinKyne wants to merge 7 commits into
mainfrom
oisin/defense-in-depth
Open

OisinKyne wants to merge 7 commits into
mainfrom
oisin/defense-in-depth

Conversation

@OisinKyne

Copy link
Copy Markdown
Contributor

These are defense-in-depth follow-ups to the Grafana hardening, so that one compromised service can reach less. It is the CDVN version of lido-charon-distributed-validator-node#360, plus #365 (Kalo's token change). There is one commit per change, so each can be backported to main-v1.11 on its own.

  • Alloy gets only the Docker API calls it uses.
    • The tecnativa socket proxy with CONTAINERS=1 allowed every GET under /containers. Anything on the docker-api network could read any container's environment (/json), files (/archive) and full filesystem (/export), including validator key shares.
    • In logging.yml, the proxy is now the official haproxy image with a short config in this repo, docker-socket-proxy/haproxy.cfg.
    • It allows only GET and HEAD of _ping, version, containers/json, containers/<id>/json, /logs and networks, which are the calls Alloy makes. It checks the path after URL-decoding and logs refused requests.
    • It runs with no capabilities, a read-only filesystem and no-new-privileges. It restarts if the Docker socket goes away, for example after a Docker engine update.
    • Only Alloy and the proxy are on the internal docker-api network.
    • check_exposure.py now fails if another service joins that network, or if the allowlist admits archive, export, exec, create and similar calls. It also fails if the allowlist refuses a call Alloy needs.
    • An earlier commit in this PR used wollomatic/socket-proxy; a later commit replaces it, so the PR trusts only the official image.
  • Alloy is hardened.
    • It now discovers only containers labelled alloy-monitored=true. It used to list and tail every container on the host and drop the unlabelled ones afterwards.
    • It runs as the image's alloy user, with no capabilities, no-new-privileges and a read-only filesystem.
    • It writes its generated config to a tmpfs and its data to an alloy-data volume, so log positions survive a recreate. Usage reporting is off.
  • Startup scripts and configs are mounted read-only. Each VC's run.sh, and the prometheus, alloy, loki and tempo mounts, are now :ro. A compromised container can no longer rewrite the script it runs on its next start, or the host file behind it. Prometheus and Alloy render their configs to /tmp in the container instead of next to the template, so the remote-write token and Loki addresses no longer land in files on the host.
  • The Prometheus remote-write token is passed as a Compose secret (/run/secrets/prom_remote_write_token), not as an environment variable.
    • Why it matters: Alloy has to inspect containers to tail their logs, and inspect returns the container's environment. With the token in the environment, a compromised Alloy could read it.
    • run.sh reads the file, and still accepts the variable if an override file sets it.
  • Reth's JSON-RPC modules are listed explicitly (--http.api=eth,net,web3).
    • This is already Reth's default, so nothing changes today. It pins the set so that a future default can't widen it.
    • check_exposure.py now rejects admin, debug, personal, trace, parity and miner on any execution client. That is the kind of module abused in the incident, through Nethermind's personal_importRawKey.
  • Loki refuses delete requests (limits_config.deletion_mode: disabled). By default, anything that can reach Loki can queue deletion of stored logs, which would let an attacker erase their traces. Retention still applies.

Upgrade notes for the release:

  • PROM_REMOTE_WRITE_TOKEN must be set in .env, even if empty (PROM_REMOTE_WRITE_TOKEN=""). If the line is missing or commented out, as in older samples, docker compose up stops with required by secret ... is not set. The new samples set it.
  • The prometheus/prometheus.yml (which contains the token) and alloy/config.alloy that earlier releases generated on the host are no longer used, and can be deleted.
  • Keep the files in this directory readable by other users, which is the default for a git clone, because the containers read the read-only mounts as non-root users.

Testing:

  • On a Linux host (Docker 29.9.2, Compose v5.6.1, amd64):
    • The CDVN logging.yml proxy path, with the earlier proxy, collected logs and refused the other calls.
    • Loki with the new setting starts on an old data directory and answers 403 to deletes.
    • On LCDVN, the haproxy proxy and Alloy hardening collect logs through the proxy and refuse archive, export and other calls, including URL-encoded paths. They also honour the label filter and handle an upgrade from the previous commit.
    • The same changes, with the token secret, shipped in LCDVN v0.3.17 after a full run over every EL×CL pair.
  • Locally: check_exposure.py passes on this branch.

Not tested yet:

  • This exact CDVN tree on a host, including the token secret with Prometheus enabled by default.
  • A Docker engine restart with the proxy healthcheck.
  • The Nimbus VC, VCs with real keys, and arm64.

OisinKyne and others added 7 commits November 11, 2026 15:24
tecnativa/docker-socket-proxy with CONTAINERS=1 allows every GET under
/containers, including /archive and /export, so a compromised Alloy
could read any file from any container, such as validator key shares.

Switch to wollomatic/socket-proxy with an anchored allowlist: ping,
version, container and network listing, container inspect and logs.
It accepts connections from alloy only, runs with no capabilities on a
read-only filesystem, and restarts if the socket goes away.

check_exposure.py now fails if the proxy accepts other clients, allows
methods other than GET and HEAD, or allows a forbidden request.
Prometheus, Alloy and every validator client could write to the
script they start from on the host (prometheus/run.sh, alloy/run.sh,
<vc>/run.sh), so a compromised container could plant code that runs
again after an upgrade, the same persistence path closed for Grafana.

Mount those files and directories, loki.yml and tempo read-only.
Prometheus and Alloy now render their configs to /tmp inside the
container, so the remote-write token and Loki addresses no longer land
in prometheus/prometheus.yml or alloy/config.alloy on the host.
Reth was started with --http and no --http.api, so it served whatever
its defaults are (eth, net, web3 in v2.8.1). Pin that list, as done for
Nethermind, so a future default can't widen the API the stack exposes.

check_exposure.py now fails if an execution client doesn't list its
modules or enables admin, debug, personal, trace, parity or miner.
Loki's default deletion_mode lets any client that reaches it queue deletion of stored logs, which would let an attacker erase their traces. Retention is unaffected.
… image

Replaces wollomatic/socket-proxy with the official haproxy image and a short config in this repo that allows the same calls: GET and HEAD of _ping, version, containers/json, containers/<id>/json and /logs, and networks. Only Alloy and the proxy are on the internal docker-api network, which the exposure check now enforces in place of socket-proxy's -allowfrom.

Alloy itself now runs with no capabilities, no-new-privileges and a read-only filesystem (rendered config on tmpfs, data in a volume, so log positions survive a recreate), with usage reporting off. Its Docker discovery only lists containers labelled alloy-monitored=true, so it no longer tails every container on the host.
With every capability dropped, root can't write to /var/lib/alloy/data, which the image owns as uid 473 with mode 770, so Alloy crash-looped with 'mkdir /var/lib/alloy/data: permission denied'. Running as the image's alloy user fixes that without adding a capability. Also renames the haproxy backend so it no longer shares the frontend's name, which haproxy warns about.
Port of lido-charon-distributed-validator-node#365 by Kalo:
- Prometheus gets PROM_REMOTE_WRITE_TOKEN as a Compose secret file
  instead of an environment variable; run.sh reads the file, and still
  accepts the variable from an override file.
- The sample .env files set PROM_REMOTE_WRITE_TOKEN="" so the secret
  always has a source.
- The socket proxy restarts if the Docker socket goes away, logs refused
  requests, and check_exposure.py asserts the calls Alloy needs stay
  allowed.
- .gitignore and README cover the configs older releases rendered into
  the repo (prometheus/prometheus.yml, alloy/config.alloy).

Co-authored-by: kalo <[email protected]>
@sonarqubecloud

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant