Skip to content

[core] Reuse manifest merge results on commit retry#8229

Draft
leaves12138 wants to merge 1 commit into
apache:masterfrom
leaves12138:codex/reuse-manifest-merge-on-retry
Draft

[core] Reuse manifest merge results on commit retry#8229
leaves12138 wants to merge 1 commit into
apache:masterfrom
leaves12138:codex/reuse-manifest-merge-on-retry

Conversation

@leaves12138

@leaves12138 leaves12138 commented Jun 13, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Reuse the previous manifest merge result on commit retry when all previously merged input manifests still exist as a continuous sub-list in the latest snapshot.
  • If a retry cannot reuse that merge result, skip ManifestFileMerger.merge and write the latest snapshot data manifests directly as the new base manifest list.
  • Add ListUtils.tryReplace for continuous sub-list replacement and use it for manifest replacement.
  • Avoid caching the skipped no-op path as a reusable merge result for later retries.

Tests

  • mvn -pl paimon-common -Pfast-build -Dtest=ListUtilsTest test
  • mvn -pl paimon-core -am -Pfast-build -DfailIfNoTests=false -Dtest=FileStoreCommitTest#testCommitRetryReusePreviousManifestMergeResultWhenBeforeStillExists+testCommitRetrySkipsManifestMergeWhenBeforeExistsNonContiguously+testCommitRetrySkipsManifestMergeWhenPreviousMergeCannotBeReused+testCommitRetrySkipsManifestMergeAcrossMultipleRetries+testCommitRetryFromEmptyTableWithConcurrentFirstSnapshot+testCommitRetrySkipsManifestMergePreservesDeleteOrder test
  • mvn -pl paimon-core -am -Pfast-build -DfailIfNoTests=false -Dtest=FileStoreCommitTest test
  • mvn -pl paimon-common,paimon-core -am -DskipTests compile
  • git diff --check

@leaves12138 leaves12138 force-pushed the codex/reuse-manifest-merge-on-retry branch 2 times, most recently from 9785f48 to 5c1550b Compare June 15, 2026 03:33
@leaves12138 leaves12138 marked this pull request as ready for review June 17, 2026 04:28
@JingsongLi

JingsongLi commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

Sounds no issue to me.

+ mergedManifests.size()));
boolean insertedMergeAfter = false;
for (ManifestFileMeta currentManifest : currentManifests) {
int mergeBeforeIndex = remainingMergeBefore.indexOf(currentManifest);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can use ListUtils something like list.replaceAll.

@leaves12138 leaves12138 marked this pull request as draft June 18, 2026 07:49
@leaves12138 leaves12138 force-pushed the codex/reuse-manifest-merge-on-retry branch 2 times, most recently from e6100f9 to 5f16a12 Compare June 18, 2026 09:10
@leaves12138 leaves12138 force-pushed the codex/reuse-manifest-merge-on-retry branch from 5f16a12 to 9c0d773 Compare June 18, 2026 09:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants