Skip to content

Cloudflare deploy stops on a lagging Worker version list (inheritance-source-changed, triggers-failed) #891

Open

Description

@konaito

Summary

void deploy --platform cloudflare stops with inheritance-source-changed or triggers-failed ("Live now: unknown") when Cloudflare's Worker version list or deployment list lags behind a write. The lag was longer than the retry windows in the deploy code. In one case the activation had already succeeded and production served the new version.

This is not the same as #868 (platform upgrade) or #880 (Durable Object bindings). Both fixed a single stale read with a retry of a second or less. The version list lagged by much longer in the runs below.

Environment

  • void 1.7.0 and 1.11.0, void deploy --platform cloudflare from GitHub Actions
  • A small Worker app with D1, R2 and a custom domain behind Cloudflare Access
  • Observed on 2026-11-20. Runs that started between 12:26 and 12:49 UTC failed, and a re-run of the last one at 12:51 succeeded

Observed

Void's deploy log (~/.void/logs/*.jsonl) and a read-only diagnostic step that ran cf workers versions list and deployments list right after each failure. Times are UTC. Version numbers are Cloudflare's per-Worker version numbers.

Run A: void 1.7.0, triggers-failed

Time Event
12:45:45 versions create creates version #23 (created_on 12:45:45.7)
12:46:00 Void activates #23 (deployments create, deployment created_on 12:46:00.5)
12:46:06 to 12:46:30 Void runs triggers deploy four times, each followed by two deployments list reads
12:46:34 Deploy stops: "the Worker activated, but Cloudflare trigger synchronization failed. Live now: unknown."
12:46:37 versions list page 1 reports total_count: 22 and its newest entry is #22. #23 is missing, 52 s after creation
12:46:40 deployments list shows the activation of #23 at 100%

Run B: void 1.11.0, inheritance-source-changed

Time Event
12:49:58 Void uploads version #24
12:50:02 Deploy stops: "the active Worker Version is not the latest upload, and its secrets could not be pinned safely."
12:50:06 versions list page 1: total_count: 24, lists #23 and #24
12:50:09 versions list page 3: total_count: 23

#23 was the active version when run B started its pre-upload checks, about 4 minutes after it was created. Run B's jsonl log was not kept, so I cannot say which list read was stale. The message and the order of events fit this sequence: the baseline read before the upload did not show #23, and the read after the upload showed #23 and #24, so the lineage check saw an unexpected history.

The app's owner also measured a version that appeared in the list about 5 minutes after it was created, and an activated version that appeared about 1.5 minutes after activation. I do not have the raw output for those two numbers.

What the logs support: the version list stayed behind a write for at least 52 seconds, and two pages read about three seconds apart disagreed about total_count, so different reads can see different snapshots.

Where it comes from

At 92a01567:

  • activateCloudflareVersionWithSecretsFile reads the version list once before the upload and compares it with the active version from deployments status. A list that does not contain the active version fails the lineage check with inheritance-source-changed. There is no retry.
  • The read after the upload retries twice, after 250 ms and 500 ms, and only when the list is still identical to the baseline. 4c47cc84c and 6c276f13b added this. Neither commit is in a release tag (git tag --contains is empty for both), so 1.11.0 behaves like 1.7.0 here: readRecentCloudflareVersionIds retries nothing, a failed read or a stale list ends the deploy.
  • stabilizeCloudflareVersionTriggers reads the deployment before and after each triggers deploy, runs up to four rounds with no wait, and returns status-failed (reported as triggers-failed with activeVersionId: null) when the reads keep disagreeing. Run A shows exactly four rounds.

What I could not confirm

For run A (triggers-failed) the log records which commands ran but not what each deployments list returned. The four rounds show that the "before" and "after" reads kept disagreeing, which fits reads that alternate between a stale and a fresh snapshot, but I have not verified that. I am not proposing a change for that path until someone can show the reads.

Proposal

Wait for the version list instead of failing on the first stale read, in the two places where the expected state is known:

  1. Before the upload, while the list does not contain the version that deployments status reports as active.
  2. After the upload, while the list is still identical to the baseline (extend the existing 250 ms and 500 ms retries).

Keep the failure when the list never catches up, and keep failing at once on any other mismatch, so a real concurrent deploy is still rejected. Keep the two quick retries for a failed (not stale) read. I'll open a PR for this and link it here.

Question for maintainers: how long should a deploy wait? The Cloudflare lag I measured was 52 s to about 4.5 minutes. I'm proposing a schedule that doubles up to 30 s and adds up to about 5.5 minutes, with a progress message. If you want a shorter cap, a version list that lags more than that will still stop a deploy.

Activity

  1. davidgustys commented on Nov 21, 2026

    @davidgustys
    Contributor

    A second, independent report of the same thing, later the same day, on void 1.6.2 (void deploy --platform cloudflare from GitHub Actions, EU account, Worker with D1, R2 and Durable Objects).

    Three deploys in a row stopped at the read after the upload. Times are UTC, 2026-11-20:

    Time Result
    19:53 inheritance-source-changed. The upload was left unused on top
    20:02:52 upload, 20:02:59 stop inheritance-source-changed. The active version was the newest upload before this deploy started, so the pre-upload read had nothing unusual to see
    20:08 inheritance-source-unavailable ("could not verify which Worker Version would supply inherited secrets")
    20:19 Passed, with no change on our side. The ten deploys after it passed too

    What we could check afterwards: read later, the version list showed exactly what the check expects (the upload numbered one past the previous newest), the same versions list read run by hand twelve times over 30 s gave one consistent answer, and Cloudflare records no other writer on the Worker. Our logs name the failed check, not what Void read, so the stale read is an inference here too.

    One guess about the 20:08 unavailable case, not shown by our logs: versions.list reads two pages and fails with "The Cloudflare Worker version history changed during inspection" when they overlap. A first page that doesn't yet show the upload beside a second page that does would overlap that way, which matches the two pages disagreeing about total_count in run B above.

    Cloudflare's status page listed an EU Dashboard/API incident that day from 18:24 to 18:53, which doesn't cover these times or the ones above.

    The same evening a different check hit the same lag and left the Worker undeployable; that is a separate problem, filed as #895.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions