Repository navigation
Conversation
tecnativa/docker-socket-proxy with CONTAINERS=1 allows every GET under /containers, including /archive and /export, so a compromised Alloy could read any file from any container, such as validator key shares. Switch to wollomatic/socket-proxy with an anchored allowlist: ping, version, container and network listing, container inspect and logs. It accepts connections from alloy only, runs with no capabilities on a read-only filesystem, and restarts if the socket goes away. check_exposure.py now fails if the proxy accepts other clients, allows methods other than GET and HEAD, or allows a forbidden request.
Prometheus, Alloy and every validator client could write to the script they start from on the host (prometheus/run.sh, alloy/run.sh, <vc>/run.sh), so a compromised container could plant code that runs again after an upgrade, the same persistence path closed for Grafana. Mount those files and directories, loki.yml and tempo read-only. Prometheus and Alloy now render their configs to /tmp inside the container, so the remote-write token and Loki addresses no longer land in prometheus/prometheus.yml or alloy/config.alloy on the host.
Reth was started with --http and no --http.api, so it served whatever its defaults are (eth, net, web3 in v2.8.1). Pin that list, as done for Nethermind, so a future default can't widen the API the stack exposes. check_exposure.py now fails if an execution client doesn't list its modules or enables admin, debug, personal, trace, parity or miner.
Loki's default deletion_mode lets any client that reaches it queue deletion of stored logs, which would let an attacker erase their traces. Retention is unaffected.
… image Replaces wollomatic/socket-proxy with the official haproxy image and a short config in this repo that allows the same calls: GET and HEAD of _ping, version, containers/json, containers/<id>/json and /logs, and networks. Only Alloy and the proxy are on the internal docker-api network, which the exposure check now enforces in place of socket-proxy's -allowfrom. Alloy itself now runs with no capabilities, no-new-privileges and a read-only filesystem (rendered config on tmpfs, data in a volume, so log positions survive a recreate), with usage reporting off. Its Docker discovery only lists containers labelled alloy-monitored=true, so it no longer tails every container on the host.
With every capability dropped, root can't write to /var/lib/alloy/data, which the image owns as uid 473 with mode 770, so Alloy crash-looped with 'mkdir /var/lib/alloy/data: permission denied'. Running as the image's alloy user fixes that without adding a capability. Also renames the haproxy backend so it no longer shares the frontend's name, which haproxy warns about.
Port of lido-charon-distributed-validator-node#365 by Kalo: - Prometheus gets PROM_REMOTE_WRITE_TOKEN as a Compose secret file instead of an environment variable; run.sh reads the file, and still accepts the variable from an override file. - The sample .env files set PROM_REMOTE_WRITE_TOKEN="" so the secret always has a source. - The socket proxy restarts if the Docker socket goes away, logs refused requests, and check_exposure.py asserts the calls Alloy needs stay allowed. - .gitignore and README cover the configs older releases rendered into the repo (prometheus/prometheus.yml, alloy/config.alloy). Co-authored-by: kalo <[email protected]>
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



These are defense-in-depth follow-ups to the Grafana hardening, so that one compromised service can reach less. It is the CDVN version of lido-charon-distributed-validator-node#360, plus #365 (Kalo's token change). There is one commit per change, so each can be backported to
main-v1.11on its own.CONTAINERS=1allowed everyGETunder/containers. Anything on thedocker-apinetwork could read any container's environment (/json), files (/archive) and full filesystem (/export), including validator key shares.logging.yml, the proxy is now the officialhaproxyimage with a short config in this repo,docker-socket-proxy/haproxy.cfg.GETandHEADof_ping,version,containers/json,containers/<id>/json,/logsandnetworks, which are the calls Alloy makes. It checks the path after URL-decoding and logs refused requests.no-new-privileges. It restarts if the Docker socket goes away, for example after a Docker engine update.docker-apinetwork.check_exposure.pynow fails if another service joins that network, or if the allowlist admits archive, export, exec, create and similar calls. It also fails if the allowlist refuses a call Alloy needs.wollomatic/socket-proxy; a later commit replaces it, so the PR trusts only the official image.alloy-monitored=true. It used to list and tail every container on the host and drop the unlabelled ones afterwards.alloyuser, with no capabilities,no-new-privilegesand a read-only filesystem.alloy-datavolume, so log positions survive a recreate. Usage reporting is off.run.sh, and theprometheus,alloy,lokiandtempomounts, are now:ro. A compromised container can no longer rewrite the script it runs on its next start, or the host file behind it. Prometheus and Alloy render their configs to/tmpin the container instead of next to the template, so the remote-write token and Loki addresses no longer land in files on the host./run/secrets/prom_remote_write_token), not as an environment variable.run.shreads the file, and still accepts the variable if an override file sets it.--http.api=eth,net,web3).check_exposure.pynow rejectsadmin,debug,personal,trace,parityandmineron any execution client. That is the kind of module abused in the incident, through Nethermind'spersonal_importRawKey.limits_config.deletion_mode: disabled). By default, anything that can reach Loki can queue deletion of stored logs, which would let an attacker erase their traces. Retention still applies.Upgrade notes for the release:
PROM_REMOTE_WRITE_TOKENmust be set in.env, even if empty (PROM_REMOTE_WRITE_TOKEN=""). If the line is missing or commented out, as in older samples,docker compose upstops withrequired by secret ... is not set. The new samples set it.prometheus/prometheus.yml(which contains the token) andalloy/config.alloythat earlier releases generated on the host are no longer used, and can be deleted.git clone, because the containers read the read-only mounts as non-root users.Testing:
logging.ymlproxy path, with the earlier proxy, collected logs and refused the other calls.archive,exportand other calls, including URL-encoded paths. They also honour the label filter and handle an upgrade from the previous commit.check_exposure.pypasses on this branch.Not tested yet: