Skip to content

Support generator expressions, evaluated lazily (#564) - #917

Merged
craigbarratt merged 2 commits into
custom-components:masterfrom
HoneyBearTech:fix-generator-expressions-564
Nov 21, 2026
Merged

craigbarratt merged 2 commits into
custom-components:masterfrom
HoneyBearTech:fix-generator-expressions-564

Conversation

@HoneyBearTech

@HoneyBearTech HoneyBearTech commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

Closes #564

What was broken
Generator expressions weren't implemented, so common idioms failed with not implemented ast ast_generatorexp (also reported in #834):

any(x > 3 for x in values)
total = sum(v * 2 for v in readings)
names = ", ".join(s.name for s in sensors)
first = next((e for e in entities if e.startswith("light.")), None)

What I changed
ast_generatorexp returns a real generator that evaluates each element only when it's asked for:

  • The first iterable is evaluated right away, as in Python.
  • Each element is evaluated in the scope where the generator was created, which is saved and restored around each step. Loop variables don't leak.
  • Evaluating an element in pyscript doesn't normally wait for anything, so each step is driven synchronously. That means native consumers like sum, any, "".join, sorted, zip and next work, and any() stops at the first match. An endless source such as itertools.count() is fine with next().
  • If an element really has to wait (for example task.sleep()) or makes a service call with blocking=True, its coroutine is closed and a RuntimeError says to use a list comprehension instead.
  • A generator can only be iterated in the event loop that created it. Using it from task.executor() gives a clear error.

yield and generator functions are still not supported.

Testing

  • test_generator_expressions_match_python (tests/test_unit_eval.py): 30 sources, each compared with CPython. They cover:
    • consumers: sum/any/all, min/max/sorted, zip/enumerate, dict/set/tuple, join, next and itertools.count();
    • several for/if clauses, closures and late binding, consumption by loops and comprehensions;
    • nested generators, exhaustion, exceptions and StopIteration;
    • methods, a class body, and the walrus operator.
  • test_generator_expression_errors: elements that wait give the clear error, scope is restored afterwards, and the task.executor guard works.
  • test_generator_expressions_in_triggers_and_services (tests/test_function.py): genexps in a @state_trigger expression, in trigger and service functions, and an element that makes a blocking service call.
  • The full test suite, ruff, pylint, mypy, codespell and sphinx-build -W pass on Python 3.14. The new tests run cleanly with -W error::RuntimeWarning.

Docs
Language Limitations now says generator expressions work, and lists their limits:

  • elements can't wait or make a service call with blocking=True;
  • they can only be iterated in the event loop;
  • they have the same loop-variable scope differences as pyscript's list comprehensions.

"generators" in the unsupported list now says "generator functions and the yield statement".

@craigbarratt craigbarratt left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @HoneyBearTech, this is a nice, self-contained way to get lazy generator expressions. Two small things:

  • It needs a rebase. tests/test_unit_eval.py conflicts with master only because both append at the end.
  • In the docs, "a service with blocking=True whose handler has to wait" undersells it. Any blocking=True call to a pyscript service fails, even when its handler doesn't wait, since pyscript runs the handler in its own task (and most HA services do I/O too). I'd just say an element can't make a service call with blocking=True.

Generator expressions such as any(x > 3 for x in values) failed with "not
implemented ast ast_generatorexp" (custom-components#564, also custom-components#834). They now return a real
generator that evaluates each element only when it's asked for, in the scope
where it was created; an element that has to wait (eg, for task.sleep()) raises
a RuntimeError that suggests a list comprehension instead.

Signed-off-by: Sean Stoube <[email protected]>
A service call with blocking=True fails in an element even when the handler
doesn't wait for anything: a pyscript service's handler runs in its own task,
and most other services do I/O.

Signed-off-by: Sean Stoube <[email protected]>
@HoneyBearTech

Copy link
Copy Markdown
Contributor Author

Thanks @craigbarratt! Rebased onto master. Only tests/test_unit_eval.py needed it: our tests now come after the new ones, and nothing else changed. The docs now just say an element can't make a service call with blocking=True. That's in its own commit so it's easy to see. I checked it: a blocking call to a pyscript service whose handler never waits does raise the error. (A native service with a plain @callback handler still works, but I agree the simple rule is better for the docs.)

@craigbarratt

Copy link
Copy Markdown
Member

Thanks @HoneyBearTech, looks great. Merging.

@craigbarratt
craigbarratt merged commit c4555c1 into custom-components:master Nov 21, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

not implemented ast ast_generatorexp

2 participants