Skip to content

feat: add iter to the Workflow Stream, with Direct reads, appends and closes in Core - #2132

Draft
moetemp wants to merge 40 commits into
moe/streams-apifrom
moe/streams-direct
Draft

moetemp wants to merge 40 commits into
moe/streams-apifrom
moe/streams-direct

Conversation

@moetemp

@moetemp moetemp commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

What changed?

This PR adds a Direct read to the Workflow Stream, and makes its append and close Direct calls. Core calls the Store itself, and Workflow code gets each Record as it arrives.

It's a prototype, created for the purpose of getting your feedback on the API. Its base is #2129, and Core's side is temporalio/sdk-rust#1801.

events = workflow.get_stream(events_ref, body_type=ModelEvent)
status = workflow.get_stream(status_ref, body_type=str)

# Workflow code decides on each event while the model Activity still runs.
tools = []
async for record in events.iter():
    event = record.body
    if event.kind == "tool_call":
        tools.append(
            workflow.start_activity(
                run_tool, event.tool_call, start_to_close_timeout=timedelta(minutes=1)
            )
        )
        await status.append(f"Running {event.tool_call.name}")
results = await asyncio.gather(*tools)
  • iter. It returns a StreamReading with no await. The read starts at the first step of the loop. By default it has no bound, so it reads until the Stream is closed. A timeout, an idle_timeout, max_records or max_bytes ends it before that. Then stop and done say why it ended, and last_offset is the from_offset of the next read. aclose ends a read before a stop.
  • append. It keeps its signature. Core writes the Records with the Workflow as the Producer and the Workflow Task as the epoch, and no Local Activity runs.
  • close. It keeps its signature. The Workflow Task that asks for the close records only the intent. Core closes the Stream after the server took that task, and the next task gives the result. So an awaited close spans two Workflow Tasks, and no Activity runs. A Workflow Task that fails closes nothing, so its retry can still append.
  • History. One iteration costs one core_stream marker for each Workflow Task that it spans, whatever the number of Records. By default the marker has a range, a count and a digest. With workflow_read_replay_from=StreamReplaySource.HISTORY on the StreamConfig, it has the Records.
  • Replay. Workflow code gets the same Records at the same points. By default Core reads them from the Store again, so they must stay there for as long as the Workflow can replay. When Core can't read them, the Workflow Task fails with a StreamReplayError message.
  • Read-ahead. A read holds at most 4 MiB of its Records in the Worker while Workflow code is busy, then waits until the code takes them. StreamConfig.workflow_read_ahead_bytes changes that bound.
  • Metrics. Core records metrics for these calls, such as temporal_stream_read_records and temporal_stream_append_latency, each with a stream_store attribute. The README of the Streams helpers has the list.
  • Stores. A Direct call needs a Store that runs on the bridge, as the Redis Store and MemoryStreamStore do. For a Store written in Python, append keeps its Local Activity, close keeps its Activity, and iter fails the Workflow.

It depends on two things that have no release yet.

  • nexgen. iter is generated. Its WIT lines use sync=true on @nexus.helper and a @nexus.support-type, which only the fork moetemp/nexgen has yet. So the check-protos job builds the Stream package with a commit of that fork.
  • Core. The submodule points to a branch with the Direct calls and with the WIT of iter.

Why?

A library that iterates a Stream in Workflow code needs each Record while its Producer still runs. ADK is an example, since its own loop reads the model through an async generator. On #2129, Workflow code can't read a Stream at all.

It has these costs.

  • A read keeps its Workflow Task open. So a Signal, an Update, a cancel or an Activity result reaches the Workflow at the next Workflow Task, which is at the heartbeat at the latest (about 8 s by default).
  • A read that waits on a quiet Stream costs one Workflow Task about every 8 s. An idle_timeout ends it with Idle.
  • A replay on a Worker with no cache reads the Store again on every Workflow Task.
  • A close that code doesn't await can be lost, if the Worker dies when the Workflow ends in the task that asked for it. A graceful shutdown of the Worker waits for it.
  • A read of 200 Records that arrive over 2 s is 3 Workflow Tasks and 1 marker. The marker is 269 bytes, or about 35 kB with HISTORY.

These points are open.

  • StreamReading and StreamReadStop are written by hand in the support module, since nexgen generates no class with behavior. Is that the right place?
  • A read has no interceptor hook.

How did you test it?

The tests are in tests/nexus/test_stream_direct.py. They run on the local dev server with the memory Store. They cover a generator over iter that reads the Stream of a running Activity, each stop, a plain iter() across a quiet gap, a loop from last_offset, the marker with and without Records, a replay with and without a Stream config, a Worker with no cache, a Workflow Task that fails in the middle of a read, a cancel and a continue_as_new with a read open, two tasks that share one read, a payload codec, the metrics, the read-ahead bound, a tool Activity that starts from inside the loop, and a close in two Workflow Tasks (in a finally, after a failed task, and with a cancel). A Redis test reads with iter while an Activity appends, and replays its History from Redis.

Link to a test plan if any -

  • Unit Tests
  • Staging
  • End to End Tests

Core now reads and appends a Stream inside the Workflow Task. The bridge protos and the
payload visitor follow its new commands and jobs, so a payload codec covers their bodies.
The WIT declares `iter` as a helper method of the Workflow Stream. Only that file differs
from the pin before, so the bridge needs no rebuild.
The Stream WIT marks `iter` with `@nexus.helper sync=true` and returns a support class. Only
a fork of nexgen knows these tags until its `main` has them.
Core reads and appends a Stream of Workflow code through the runtime of its Store. The Worker
and the Replayer pass one for each Store of the client's Stream config that has one.
Core reads the Store and gives each page to the read in an activation, so a read costs one
marker for each Workflow Task. `StreamConfig.workflow_read_replay_from` decides whether
History holds the Records.
An append of a Workflow Producer on a Store with a Rust runtime goes to Core, which writes it as
the Workflow with the task as its epoch. Its tests now expect that Producer and that marker.
A marker with the Records needs no Store, so a Replayer with no Stream config can replay it.
The read takes its Store from the ref, as the first run had it.
The tests cover each stop, the marker, replays, a Worker with no cache, a failed task, a
Signal, a Store written in Python and a payload codec, and measure a read of 200 Records.
A read keeps its Workflow Task open, and a replay of a `STORE` read needs the Records in the
Store. The README, the changelog and the docs of the Store and the Replayer now say so.
Core now orders a Workflow's close after its own appends and evicts a replay whose marker
Records can't be read. Its protos didn't change, so the bridge protos and the visitor stay.
Core ends such a read with Done, and counts the idle time only while a read waits for the
Store. The stops, `iter_stream` and the README now say so.
A failed replay now reaches the caller with no cut-off, raised or in the result. New tests
cover a second read after a `break`, a completion, a cancel and a continue-as-new.
Core now gives a replayed call with no Store name the Store of its entry in History. So a
Replayer with no Stream config replays a Direct append and a `HISTORY` read of such a ref.
Each new test failed before its fix: a shared read, the codec context, replays of an unnamed
ref, a bad body, and an Activity-only Worker with a bad Redis option.
A second task that waited on one read replaced the future of the first, which then never
woke. A Record that failed to decode moved `last_offset` past it, so a new read lost it.
The Local Activity path, a client and an Activity run it with no serialization context. A
codec with a context then wrote bodies that no other reader of the Stream could decode.
Only Workflow code makes Direct calls, so an Activity-only Worker with a bad Redis option
starts again. The Store docstrings now say when a Worker builds the runtime.
The commit changes only a module that Core builds for its tests, so the bridge needs no
rebuild.
A Workflow's append on Redis now writes as the Workflow, with its Workflow Task as the epoch.
A new test reads a running Activity's Stream with `iter`, then replays it from Redis.
Two tasks can share one read. The task that didn't take the failure got a clean end, so it could miss that the read failed.
Such a Store has no Rust runtime, so Workflow code loses the Direct calls on it. The docs say what it loses.
Core now closes a Stream for Workflow code after History has the task that asked for it. The
bridge protos and the payload visitor follow its new command and job.
The crate refuses a Workflow close with no epoch, and a close in a failed task must close
nothing. Core records the intent and closes after the server took the task.
A Store's Activity runs as a regular Activity only for a close on a Store written in
Python, since Core closes a Stream on a Store with a Rust runtime.
Code must await its close, since a close that it doesn't await is lost if the Worker dies when
the Workflow ends. An append and a close of Workflow code carry the task as their epoch.
The tests cover the two tasks of an awaited close, a close in `finally`, a failed task, a
close not awaited, a cancel, replays with and without a config, and a Store in Python.
The edit for the Direct close left two lines over 100 characters.
@moetemp moetemp changed the title feat: add iter to the Workflow Stream, with Direct reads and appends in Core feat: add iter to the Workflow Stream, with Direct reads, appends and closes in Core Nov 15, 2026
Core changed only its own tests there, so the bridge is the same.
Core now reads an unset idle timeout as no bound and a zero as no wait. So `None` sends
nothing, and a zero goes to Core as a zero, with no 1 ms in its place.
A read with no `idle_timeout` holds its Workflow Task open on a quiet Stream, with one task
about every 8 s. The docs say so, and show one loop as the plain example.
Three Workflows dropped the outer loop that only the 1 s default needed. A plain `iter` now
reads across a quiet gap, and a zero ends with `Idle` on an empty Stream.
A read with a short timeout gave no Record when its Workflow Task started late, which failed
a test on the cloud run. The stubs of the Core protos have the new comments of the read bounds.
Core now records metrics for the Direct calls, and a graceful shutdown waits for a close in
flight. Its protos changed only in a comment that the generated files already had.
A close that the code doesn't await is lost only when the Worker dies or the Store doesn't
answer in 10 s. The README lists the metrics that Core records for the Direct calls.
One Workflow reads, appends and closes, and the buffer of its Runtime has each metric with
its count and Store. A shutdown returns after a close that the code didn't await.
The fork has `@nexus.helper`, its `sync` form and `@nexus.support-type` in one pull request.
The generated Stream package is the same with it.
The marker now holds each Stream ref once, so the generated stubs of `external_data` change.
No Python code reads a marker entry's Stream ref.
A read stops asking the Store while that many bytes of its Records wait for Workflow code.
The Worker and the Replayer give the bound to Core through the bridge.
A copy of each Record to another Stream is work that an Activity could do itself. The example
now starts each tool as its call arrives, and the docs tell of the read-ahead bound.
The first tool starts before the model Activity completes. A bound below 1 raises, and a
bound of 1 byte still gives every Record in order.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant