Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
ee438c5
Added a Workflow read of a Stream that waits for a predicate.
moetemp Nov 13, 2026
f40b9f7
Filled in the Records of a Workflow read before a replay.
moetemp Nov 13, 2026
7e886c1
Tested and measured the Workflow read.
moetemp Nov 13, 2026
6ec7c28
Gave the Workflow read a stop reason and added a fold over its Records.
moetemp Nov 14, 2026
32be5d9
Fitted the Workflow read to the Workflow append and close.
moetemp Nov 14, 2026
efdd6ec
Pointed the refused Workflow read at `workflow.read_stream`.
moetemp Nov 14, 2026
b3de42f
Failed the Workflow for a read with no Stream config.
moetemp Nov 14, 2026
dd59dcc
Replaced the Workflow read with take, filter and reduce calls.
moetemp Nov 14, 2026
77ebb5c
Let BufferedAppender write inside the function of a Stream call.
moetemp Nov 14, 2026
c2c9768
Tested the Stream calls in Workflow, client and Activity code.
moetemp Nov 14, 2026
43a8212
Fixed the type and docstring findings of the Stream calls.
moetemp Nov 14, 2026
061d2ef
Fitted the Stream calls to the move of BufferedAppender.
moetemp Nov 14, 2026
cb86e8a
Moved Core to the Stream WIT with take, filter and reduce.
moetemp Nov 14, 2026
1920f7b
Pinned the Stream generator to the nexgen build with helper methods.
moetemp Nov 14, 2026
eef58e3
Regenerated the Streams API with take, filter and reduce.
moetemp Nov 14, 2026
ad739ea
Wired the generated take, filter and reduce to the support helpers.
moetemp Nov 14, 2026
77d1818
Added a pipeline that runs a chain of Stream steps as one call.
moetemp Nov 14, 2026
5b1e15b
Kept the Records of take and filter out of History when a Worker asks.
moetemp Nov 14, 2026
0e19195
Tested where a replay of a take or a filter gets its Records.
moetemp Nov 14, 2026
764259d
Fixed the API docs build for the replay setting.
moetemp Nov 14, 2026
866dab6
Wrapped four docstring lines that were over 100 characters.
moetemp Nov 14, 2026
969d7cc
Moved Core to the Stream WIT with the method guard.
moetemp Nov 14, 2026
21326c0
Pinned the Stream generator to the nexgen build with `@nexus.guard`.
moetemp Nov 14, 2026
5da9911
Added a check of where code calls a Stream method.
moetemp Nov 14, 2026
18896e4
Regenerated the Stream classes with the method guard.
moetemp Nov 14, 2026
bbdd5f7
Tested that a Stream refuses a call from the wrong place.
moetemp Nov 14, 2026
b96fd69
Added a pipeline end that appends its items to another Stream.
moetemp Nov 15, 2026
4c89352
Tested the pipeline end that appends to another Stream.
moetemp Nov 15, 2026
4d30201
Tested a generator over `take` that reads a running Activity's Stream.
moetemp Nov 15, 2026
8eee982
Made the Store the default replay source of a Workflow read.
moetemp Nov 15, 2026
a7ecce1
Tested the Stream reads under the new default replay source.
moetemp Nov 15, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -228,8 +228,8 @@ jobs:
- run: uv tool install poethepoet
- run: uv sync --all-extras
- run: cargo install --locked nexgen --version 0.2.8 --features advanced --force
# The Stream package needs the `main` of nexgen, which no release has yet.
- run: cargo install --locked --git https://github.com/temporalio/nexgen --rev 9653a0f5a845df0c5059554f1b5813db6a74cd1a --features advanced --root "$RUNNER_TEMP/nexgen-stream" nexgen
# The Stream package needs `@nexus.helper`, which only a fork of nexgen has yet.
- run: cargo install --locked --git https://github.com/moetemp/nexgen --rev 9055618618e580b68d61c7d7e52d8f25aea3824c --features advanced --root "$RUNNER_TEMP/nexgen-stream" nexgen
- run: echo "NEX_GEN_STREAM_BIN=$RUNNER_TEMP/nexgen-stream/bin/nexgen" >> "$GITHUB_ENV"
- run: poe build-develop
- run: poe gen-protos
Expand Down
2 changes: 2 additions & 0 deletions changelog/added/chained-pipeline-wren.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
Add `temporalio.contrib.streams.pipeline`, which chains `filter`, `map`, `take_until` and `limit` steps over a Stream and runs the whole chain as one `take`, `filter` or `reduce` call. In Workflow code a chain is one Local Activity and one marker in History.
Add the `append_to` end of a pipeline. It appends each item of the chain to another Stream while the scan runs, and in Workflow code only the number of the items goes into History.
3 changes: 3 additions & 0 deletions changelog/added/patient-reader-heron.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
Add the `take`, `filter` and `reduce` methods to the client `Stream` and the Workflow `Stream`. Client, Activity and Workflow code call them to scan a Stream until a predicate matches, the Stream is done, or a limit is reached.
Add `StreamConfig.workflow_read_replay_from`. By default (`StreamReplaySource.STORE`), a `take` or a `filter` of Workflow code keeps only the offsets and a digest of its Records in History, and a replay reads the Records from the Store again. With `StreamReplaySource.HISTORY`, History holds the Records, and a replay needs no Store.
A method of a Workflow `Stream` raises `TypeError` outside Workflow code, and a method of a client `Stream` raises it in Workflow code. Both raise it in a function of a `take`, `filter` or `reduce` call, which can't await, so a write of that function is never lost with no error.
8 changes: 5 additions & 3 deletions scripts/gen_nexus_system_api.py
Original file line number Diff line number Diff line change
Expand Up @@ -28,8 +28,9 @@
workflow_init_path = base_dir / "temporalio" / "workflow" / "__init__.py"
NEX_GEN_VERSION = "0.2.8"
# The Stream package needs a newer nexgen, which rejects the other WITs. One pin is enough
# again once the api WITs and a nexgen release agree. No release has nexgen commit 9653a0f5,
# the `main` of nexgen, yet, so NEX_GEN_STREAM_BIN points at a build of it.
# again once the api WITs and a nexgen release agree. The Stream WIT uses `@nexus.helper`
# and `@nexus.guard`, which only the fork https://github.com/moetemp/nexgen has, at commit
# 90556186. So NEX_GEN_STREAM_BIN points at a build of that commit.
NEX_GEN_STREAM_VERSION = "0.2.12"
proto_files = [
gen_protos.api_proto_dir
Expand Down Expand Up @@ -87,7 +88,8 @@ def nex_gen_stream_command() -> list[str]:
bin_path = os.environ.get("NEX_GEN_STREAM_BIN")
if not bin_path:
raise RuntimeError(
"NEX_GEN_STREAM_BIN must point at a nexgen build of commit 9653a0f5"
"NEX_GEN_STREAM_BIN must point at a nexgen build of commit 90556186 "
"of https://github.com/moetemp/nexgen"
)
version = subprocess.check_output([bin_path, "--version"], text=True).strip()
if version != f"nexgen {NEX_GEN_STREAM_VERSION}":
Expand Down
2 changes: 2 additions & 0 deletions temporalio/client/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -187,6 +187,7 @@
)
from ._stream_store import (
StreamConfig,
StreamReplaySource,
StreamSelectContext,
StreamStore,
StreamStoreCallOptions,
Expand Down Expand Up @@ -357,6 +358,7 @@
"CloudOperationsClient",
"Plugin",
"StreamConfig",
"StreamReplaySource",
"StreamSelectContext",
"StreamStore",
"StreamStoreCallOptions",
Expand Down
4 changes: 4 additions & 0 deletions temporalio/client/_impl.py
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@
import temporalio.api.errordetails.v1
import temporalio.api.failure.v1
import temporalio.api.schedule.v1
import temporalio.api.streamservice.v1
import temporalio.api.taskqueue.v1
import temporalio.api.update.v1
import temporalio.api.workflowservice.v1
Expand Down Expand Up @@ -126,6 +127,7 @@
ScheduleUpdate,
ScheduleUpdateInput,
)
from ._stream_read import _note_page_sizes
from ._stream_store import (
_STREAM_SERVICE,
StreamConfig,
Expand Down Expand Up @@ -1686,6 +1688,8 @@ async def _start_stream_store_operation(
response = await _call_stream_store(
store, data_converter, input.operation, request
)
if isinstance(response, temporalio.api.streamservice.v1.ReadRecordsResponse):
_note_page_sizes(response)
handle: NexusOperationHandle[Any] = NexusOperationHandle(
self._client,
input.id,
Expand Down
Loading
Loading