Skip to content

[core][python] Add global index fast search option#8255

Draft
JingsongLi wants to merge 1 commit into
apache:masterfrom
JingsongLi:codex/vector-raw-fallback
Draft

[core][python] Add global index fast search option#8255
JingsongLi wants to merge 1 commit into
apache:masterfrom
JingsongLi:codex/vector-raw-fallback

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Jun 16, 2026

Copy link
Copy Markdown
Contributor

Summary

Add global-index.fast-search as the freshness/performance switch for global-index queries. It defaults to true for index-only fast search; when set to false, supported global-index reads scan unindexed row ranges and merge fresh raw-data matches.

Changes

  • Add the Java and Python global-index.fast-search option with generated docs.
  • Use the option in vector search, Spark vector reads, and generic data-evolution global-index scans.
  • Remove the unpublished vector raw-fallback option surface and derive vector slow-search metric from the vector indexer interface.
  • Add exact unindexed-range scanning for scalar global-index reads so slow search returns only rows matching the predicate.
  • Add Java and Python coverage for fast-search defaults, slow-search freshness, filtered unindexed scans, and Python extra-field scalar index matching.

Testing

  • git diff --check
  • rg -n "raw-fallback|RAW_FALLBACK|raw fallback|raw_fallback|rawFallback|RawFallback" . || true
  • python -m compileall -q paimon-python/pypaimon/globalindex/global_index_scanner.py paimon-python/pypaimon/read/scanner/file_scanner.py paimon-python/pypaimon/table/source/vector_search_read.py paimon-python/pypaimon/table/source/vector_search_scan.py paimon-python/pypaimon/common/options/core_options.py paimon-python/pypaimon/tests/vector_search_filter_test.py
  • python -m pytest paimon-python/pypaimon/tests/read_builder_explain_test.py paimon-python/pypaimon/tests/vector_search_filter_test.py paimon-python/pypaimon/tests/global_index_test.py -q
  • mvn -pl paimon-core -am -Pfast-build -DfailIfNoTests=false -Dtest=VectorSearchBuilderTest#testVectorSearchScansUnindexedDataWhenFastSearchDisabled+testVectorSearchFastSearchSkipsUnindexedDataByDefault+testVectorSearchSlowSearchScansFilteredUnindexedData,BtreeGlobalIndexTableTest#testBTreeGlobalIndexFastSearchControlsUnindexedData test
  • mvn -pl paimon-spark/paimon-spark-common -am -Pfast-build -DskipTests compile

Notes

vector.raw-fallback.enabled is intentionally not kept as a compatibility alias because it was not released.

@JingsongLi JingsongLi changed the title [core] Support raw fallback for vector search [WIP][core] Support raw fallback for vector search Jun 16, 2026
@JingsongLi JingsongLi changed the title [WIP][core] Support raw fallback for vector search [core] Support raw fallback for vector search Jun 16, 2026
@JingsongLi JingsongLi marked this pull request as draft June 16, 2026 13:46
@JingsongLi JingsongLi changed the title [core] Support raw fallback for vector search [core][python] Support raw fallback for vector search Jun 16, 2026
@JingsongLi JingsongLi force-pushed the codex/vector-raw-fallback branch from 658ab46 to baefa9d Compare June 18, 2026 05:43
@JingsongLi JingsongLi changed the title [core][python] Support raw fallback for vector search [core][python] Add global index fast search option Jun 18, 2026
@JingsongLi JingsongLi force-pushed the codex/vector-raw-fallback branch from baefa9d to 5fab662 Compare June 18, 2026 05:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant