Skip to content

fix: disconnect() tears down locally after sending the Leave instead of waiting for the server echo - #1208

Open
hiroshihorie wants to merge 6 commits into
mainfrom
hiroshi/complete-disconnect-on-socket-close
Open

hiroshihorie wants to merge 6 commits into
mainfrom
hiroshi/complete-disconnect-on-socket-close

Conversation

@hiroshihorie

@hiroshihorie hiroshihorie commented Sep 15, 2026 •

Copy link
Copy Markdown
Member

Problem

engine.disconnect() sent the Leave and relied on the server's echo to run cleanup and emit EngineDisconnectedEvent, and Room.disconnect() waited up to 10 s for that event. Media nodes drop queued leave messages when they close the signal sink (pkg/routing/signal.go, signalMessageSink.Close, comment in place), so the echo can be lost. Every lost echo was a 10 s stall, a TimeoutException out of disconnect(), and a room left half torn down because Room.disconnect() never reached its own cleanup.

Found by the e2e-flutter suite (livekit/e2e-flutter#7): one agent per run against a local livekit-server 1.13.6, SignalClient did disconnect with no Leave received before it while the server logged sendLeave: true.

This design dates from the Dec 2023 engine rewrite (v1.5.5). #559 (v2.2.2, issue #553) made Room.disconnect() wait for the event, which turned a silently missing event into a visible stall. No other SDK waits on the server here: JS, Swift, Android and Rust all send the Leave and tear down locally.

Fix

Flutter now does the same.

  • engine.disconnect() sends the Leave, runs cleanUp() and emits EngineDisconnectedEvent in one go. No dependency on the echo.
  • A Leave that arrives after disconnect() started is the echo, or a server Leave racing ours, and is ignored.
  • Room.disconnect() awaits the engine teardown and then its own cleanup, no wait on the server and no timeout. The engine event handler skips its cleanup while Room.disconnect() is running, so teardown has one owner and a connect() right after disconnect() returns cannot race a straggling handler. That is the case behind [bug] Actions stop working after second connect attempt to same room flutter livekit client #553, now covered by a test.

The first commit on this branch was a narrower fallback that completed the disconnect on socket close while keeping the echo dependency. The second commit replaces it with the alignment above.

Tests

test/core/disconnect_event_test.dart, four added:

  • socket close without echo completes disconnect() with one clientInitiated event
  • echo followed by socket close emits exactly one event
  • disconnect() returns with engine and room torn down when the server sends nothing at all
  • connect again to the same room right after disconnect() returns

The last two fail on main.

e2e-flutter

"Disconnect while the server stops answering", a new scenario for livekit/e2e-flutter#39. Only the target's signal socket goes through a proxy, which stops forwarding anything from the server once disconnect() starts, so neither the Leave echo nor the answer to the WebSocket close frame arrives. That is a lost echo, and also a user who leaves right after the network died. disconnect() must return within 3 s with one clientInitiated event and the room torn down, the other participant must see the leave, and the same Room must join again.

2.14.0 this PR
macOS, local livekit-server 1.13.11 TimeoutException after 10.0 s returns in 39 ms
macOS, LiveKit Cloud not run returns in 38 ms
iPhone 16 Pro (iOS 27.2.1), LiveKit Cloud TimeoutException after 10.0 s returns in 16 ms

The existing scenarios that exercise disconnect() pass in the same setups, with #1211 on top: "Connect and disconnect in a loop on one room" (ten connects and disconnects on one Room), "Disconnect reasons", "Dispose during stream writes", "RPC recipient leaves mid-call", "Disconnect during a full reconnect stays disconnected" and "Dispose during a full reconnect stays gone" (#1264, #1268).

The Cloud runs used a regional URL. Against a region pinned project, every connect on the project URL is refused and goes through region failover, and the connect loop then fails on its third connect on every SDK version, the issue #1204 addresses.

…echoing the leave

engine.disconnect() sends the Leave and relies on the server's echo to
run cleanup and emit EngineDisconnectedEvent. Media nodes drop queued
leave messages when they close the signal sink, so the echo can be
lost. The socket close was then ignored because _isClosed was set, and
Room.disconnect() waited out its 10 s timeout for an event that never
came, leaving the room half torn down.

The engine now remembers a disconnect that is waiting for its echo and,
on a socket close in that window, runs cleanup and emits the event
itself. A close that follows the echo is still ignored, so there is no
second event.
…of waiting for the echo

The engine used to send the Leave and rely on the server's echo to run
cleanup and emit EngineDisconnectedEvent, and Room.disconnect() waited
up to 10 s for that event. Media nodes drop queued leave messages when
they close the signal sink, so the echo can be lost, and every lost
echo was a 10 s stall, a TimeoutException, and a room left half torn
down. No other SDK waits on the server here.

disconnect() now sends the Leave, runs cleanUp() and emits the event in
one go, like JS, Swift, Android and Rust. A Leave that arrives after
disconnect() started is the echo and is ignored. Room.disconnect() owns
the teardown for the disconnect it started, so the engine event handler
skips its own cleanup in that case and a connect() right after
disconnect() returns cannot race it.

Replaces the earlier socket close fallback on this branch.
@hiroshihorie hiroshihorie changed the title fix: complete disconnect() when the server closes the socket without echoing the leave fix: disconnect() tears down locally after sending the Leave instead of waiting for the server echo Sep 16, 2026
@hiroshihorie

Copy link
Copy Markdown
Member Author

e2e-flutter suite (livekit/e2e-flutter#7) against this branch, local livekit-server 1.13.6, macOS, three full runs on 2026-09-16:

  • run 1: 8 pass, 10 skip, 0 warn, 0 fail
  • run 2: 8 pass, 10 skip, 0 warn, 1 fail in ParticipantMetadataTest, a race in the test itself (un-awaited waitUntil), fixed in e2e-flutter 920b3b2
  • run 3: 8 pass, 10 skip, 0 warn, 0 fail

ParticipantAttributesTest, the scenario whose agent hit the lost Leave echo on every run before, completed its teardown in about 0.5 s on all three runs with no warning. The 10 skips are camera scenarios, not runnable on this machine.

@bdlukaa

bdlukaa commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

@hiroshihorie any ETA on when this can be reviewed and merged?

Some of our users are having this fatal error and this PR fixes it:

TimeoutException: LiveKit Exception: [TimeoutException] Timeout
#0      EventsListenable.waitFor.<fn> (package:livekit_client/src/managers/event.dart:224)
#1      _RootZone.run (dart:async/zone.dart:1028)
#2      Future.timeout.<fn> (dart:async/future_impl.dart:1054)
  File "<asynchronous suspension>"
#4      EventsListenable.waitFor (package:livekit_client/src/managers/event.dart:222)
  File "<asynchronous suspension>"
#6      Room.disconnect (package:livekit_client/src/core/room.dart:725)
  File "<asynchronous suspension>"
#8      SessionController._disconnect (package:totem_core/features/sessions/controllers/core/session_controller.dart:732)
  File "<asynchronous suspension>"
#10      SessionController.leave (package:totem_core/features/sessions/controllers/core/session_controller.dart:607)

@hiroshihorie
hiroshihorie marked this pull request as ready for review October 16, 2026 21:29

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 2 potential issues.

Devin Review

Comment thread lib/src/core/engine.dart
await _signalListener.cancelAll();
_clearPendingReconnect();
}
await cleanUp();

@devin-ai-integration devin-ai-integration Bot Oct 16, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Old socket close disrupts immediate reconnect

When connect() follows disconnect(), the old socket's delayed close can mark the new connection disconnected. SignalClient._onSocketDispose does not distinguish the old socket from the new one, triggering an unnecessary reconnect.

Learn more

The signaling client stores connection state for the current socket, but each socket retains the same _onSocketDispose callback. On web, LiveKitWebSocketWeb calls WebSocket.close() without waiting for the later onClose event. SignalClient.cleanUp therefore finishes before the old callback runs. A subsequent connect can install a new socket before SignalClient._onSocketDispose receives the old close and marks that new connection disconnected.

Example: Disconnect session A on web, then immediately connect session B on the same Room. If A's onClose fires after B connects, B is marked disconnected and the engine begins reconnecting despite B's live socket.

Recommended fix: Bind close callbacks to the socket or connection generation that created them. Ignore closes from sockets that are no longer current; add a web-style delayed-close regression test after reconnect.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid, this is the case #1211 covers. It is stacked on this PR and they merge together. SignalClient now tags each socket's callbacks with a generation that connect() and cleanUp() advance, so a close reported by a socket that was already replaced is ignored. Covered by a close reported by a replaced socket does not disturb the next connection in test/core/stale_socket_close_test.dart.

Comment thread lib/src/core/room.dart Resolved
disconnect() let the engine handler emit RoomDisconnectedEvent as soon as the
engine had closed, while the room's own teardown was still running. An app that
connects again from that event started its new session in the middle of the
old teardown, which then disposed the new engine state and left the room
disconnected.

The handler now leaves the disconnect it did not start alone, and disconnect()
emits the event after _cleanUp() has finished.
@hiroshihorie
hiroshihorie added this pull request to stack #1242 October 16, 2026 21:59
@hiroshihorie

Copy link
Copy Markdown
Member Author

Merged main in (clean). Full suite 612 passing, analyze and format clean.

The e2e-flutter run promised in the description, on macOS against a local livekit-server 1.13.11, with #1211 on top: 13 of the 14 existing scenarios tagged reconnect or signaling pass, including "Disconnect during a full reconnect stays disconnected" and "Dispose during a full reconnect stays gone" (#1264, #1268). The 14th, "stale publisher answer after ice restart", is an expected failure unrelated to this. Of the new scenarios from livekit/e2e-flutter#39, 3 pass unexpectedly (fixed by #1211), 1 is the expected failure left for the #1219 follow-up, and the publisher-down one passes.

@bdlukaa this is moving again. It fixes the TimeoutException from EventsListenable.waitFor on disconnect you reported.

…eport it when a step fails

Two disconnect() calls that overlapped both passed the closed check, so each sent a Leave, ran the teardown and emitted RoomDisconnectedEvent. They now share the teardown in flight, released before the event so a rejoin from that event gets a teardown of its own on its next disconnect().

A teardown step that threw skipped the room's cleanup or the only RoomDisconnectedEvent, because the engine event handler stays quiet while disconnect() runs. The room's cleanup now runs after a failed engine teardown, the event is emitted either way, and the error still reaches the caller.

The tests now check that the Leave is sent, count engine events in the echo case, and check that the room is torn down when the disconnected event arrives. Found in review (Codex).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants