Skip to content

Fix pipe runtime meta push blocking user operations#17909

Open
Caideyipi wants to merge 12 commits into
apache:masterfrom
Caideyipi:fix-pipe-runtime-meta-best-effort
Open

Fix pipe runtime meta push blocking user operations#17909
Caideyipi wants to merge 12 commits into
apache:masterfrom
Caideyipi:fix-pipe-runtime-meta-best-effort

Conversation

@Caideyipi

@Caideyipi Caideyipi commented Jun 11, 2026

Copy link
Copy Markdown
Collaborator

Summary:

  • Add a shared best-effort short-timeout runtime metadata request path for background procedures.
  • Use it for pipe leader/meta change pushes and subscription leader-change commit progress/topic/consumer meta pushes so optional DataNode RPCs do not hold coordinator locks for the full sync interval.
  • Split subscription runtime state push during leader change: readable DataNodes remain strict, while unreadable DataNodes are updated best-effort with a short timeout.
  • Keep user-facing operations and periodic sync procedures on existing strict paths.

Fixes/targets:

  • IoTDBPipeClusterIT.testPipeAfterDataRegionLeaderStop flaky timeout on the second createPipe after full cluster restart: the client saw SocketTimeoutException: Read timed out while createPipe(p2) was waiting behind background pipe runtime metadata work.
  • In the 2026-06-11 failure logs, PipeHandleLeaderChangeProcedure was pushing PIPE_PUSH_ALL_META to restarting DataNodes, repeatedly hit Connection refused, held PipeTaskCoordinatorLock, and caused Pipe-Runtime-Heartbeat to skip rounds. This PR bounds those background runtime metadata pushes so they do not block user-facing pipe operations for the full sync interval.

Tests:

  • git diff --check
  • mvn -DskipTests spotless:apply -pl iotdb-core/confignode
  • mvn '-Ddevelocity.off=true' -DskipTests compile -pl iotdb-core/confignode -am
  • Attempted targeted IoTDBPipeClusterIT#testPipeAfterRegisterNewDataNode; local JVM native OOM during ConfigNode startup before pipe stage

@Caideyipi Caideyipi force-pushed the fix-pipe-runtime-meta-best-effort branch from 3d73343 to 8433b3d Compare June 11, 2026 06:31
@Caideyipi Caideyipi force-pushed the fix-pipe-runtime-meta-best-effort branch from 8433b3d to b334290 Compare June 11, 2026 06:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant