Skip to content

[FLUSS-3483][coordinator] Validate lake snapshot offsets before commit#3494

Open
Jackeyzhe wants to merge 2 commits into
apache:mainfrom
Jackeyzhe:fluss-3483
Open

[FLUSS-3483][coordinator] Validate lake snapshot offsets before commit#3494
Jackeyzhe wants to merge 2 commits into
apache:mainfrom
Jackeyzhe:fluss-3483

Conversation

@Jackeyzhe

Copy link
Copy Markdown
Contributor

Purpose

Linked issue: close #3483

This PR fixes a race where a stale V2 lake snapshot commit can be accepted after a table is dropped and recreated. In that case, the recreated table may get a /laketable znode pointing to an offsets metadata file generated for the old table id, causing lake tiering to repeatedly fail when reading the stale offsets path.

Brief change log

  • Reject V2 lake snapshot commits when the target table id no longer exists or is queued for deletion.
  • Validate the V2 lake snapshot offsets metadata before writing it to ZK:
    • the offsets file must be readable;
    • the file-level tableId must match the commit table id;
    • every TableBucket in the offsets file must also belong to the commit table id.
  • Validate both tiered offsets and readable offsets metadata paths when they differ.
  • Add regression coverage for:
    • non-existent table id commits;
    • table queued for deletion;
    • recreated table id with stale offsets metadata from the old table id;
    • RPC-level V2 commit after table drop.

Tests

mvn -pl fluss-server -DskipITs -Dcheckstyle.skip -Drat.skip -Dspotless.check.skip=true -Dtest=CoordinatorEventProcessorTest#testCommitLakeTableSnapshotV2RejectsMismatchedOffsetsFileTableId+testCommitLakeTableSnapshotV2RejectsNonExistentTable+testCommitLakeTableSnapshotV2RejectsQueuedForDeletionTable test
mvn -pl fluss-server -DskipITs -Dcheckstyle.skip -Drat.skip -Dspotless.check.skip=true -Dtest=CommitLakeTableSnapshotITCase#testCommitLakeTableSnapshotV2RejectedAfterDropTable test

API and Format

no

Documentation

no

jackeyzhe added 2 commits June 18, 2026 12:11
Reject V2 lake snapshot commits when the target table no longer exists,
is queued for deletion, or the offsets metadata file belongs to a
different table id.

This prevents stale lake snapshot paths from being written into the
recreated table's laketable znode after drop/recreate races.

Also add coordinator and RPC tests for dropped tables and mismatched
offsets metadata.
Reject V2 lake snapshot commits when the target table no longer exists,
is queued for deletion, or the offsets metadata file belongs to a
different table id.

This prevents stale lake snapshot paths from being written into the
recreated table's laketable znode after drop/recreate races.

Also add coordinator and RPC tests for dropped tables and mismatched
offsets metadata.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

commitLakeTableSnapshot writes stale snapshot path after dropTable, leaving the tiering pipeline permanently stuck

1 participant