[paimon] Add bitmap global index

[JingsongLi]

Summary

Add a new bitmap global index type for enum/tag-style scalar predicates. The index stores exact Roaring 64-bit row-id bitmaps per value, keeps per-file null/non-null row sets for correct complement semantics across multiple index files, and reuses the same sorted-index pruning/parallel-reading framework as BTree.

Changes

• Add bitmap global index SPI implementation and register it with GlobalIndexerFactory.
• Store value -> RoaringNavigableMap64(relativeRowId) with separate nullRows and nonNullRows bitmaps.
• Add bitmap index file format version 1 with Roaring bitmap blocks, compressible dictionary blocks, a compressible dictionary block index, and a fixed footer containing block handles, value count, version, and magic.
• Encode dictionary block and dictionary block-index counts, key lengths, block offsets, and block lengths with var-length integers to reduce metadata overhead.
• Open bitmap index files by reading only the footer; load null/non-null bitmaps, dictionary block index, dictionary blocks, and value bitmap blocks lazily as predicates need them.
• Add optional dictionary block compression via bitmap-index.compression and bitmap-index.compression-level; bitmap value blocks keep Roaring's native serialization.
• Add manifest-level sorted metadata containing logical first non-null key, last non-null key, null flags, and metadata version 1.
• Extract shared sorted-index components from BTree, including KeySerializer, SortedIndexFileMeta, SortedFileMetaSelector, ParallelFileGlobalIndexReader, and SortedFileGlobalIndexReader.
• Prune bitmap and BTree index files before opening them for equality, IN, null checks, prefix predicates, range predicates, BETWEEN, NOT BETWEEN, and compound AND / OR predicates.
• Add fallback scan size limits based on the total size of selected candidate index files, with bitmap-index.fallback-scan-max-size and btree-index.fallback-scan-max-size both defaulting to 256 mb.
• Add bitmap-index.dictionary-block-size to control dictionary block sizing; default is 16 kb.
• Support direct lookup for =, IN, string startsWith / LIKE 'prefix%', !=, NOT IN, IS NULL, and IS NOT NULL.
• Support budget-guarded dictionary-scan fallback for endsWith, contains, general LIKE, range predicates, BETWEEN, and NOT BETWEEN.
• Align null-literal semantics with Paimon predicates for = NULL, != NULL, IN (..., NULL), and NOT IN (..., NULL).
• Add common reader/writer tests for equality-family predicates, null semantics, fallback scan predicates, fallback budget disabling, selected-file fallback budget behavior, multi-file complement semantics, manifest-level pruning, prefix lookup, lazy IO behavior, and sorted metadata comparator pruning.
• Add table-level tests covering bitmap global index scan, core read pushdown, multi-range coverage, complement predicates, startsWith predicates, fallback scan predicates, and null handling.
• Add Bitmap Index documentation next to BTree docs, including use cases, options, query behavior, fallback behavior, manifest metadata, and on-disk file layout.

Testing

• mvn -pl paimon-common spotless:apply
• mvn -pl paimon-common -Pfast-build -DfailIfNoTests=false -Dtest=SortedFileMetaSelectorTest test
• mvn -pl paimon-common -Pfast-build -DfailIfNoTests=false -Dtest=LazyFilteredBitmapIndexReaderTest test
• mvn -pl paimon-common -Pfast-build -DfailIfNoTests=false -Dtest=LazyFilteredBTreeIndexReaderTest test
• mvn -pl paimon-core -am -Pfast-build -DfailIfNoTests=false -Dtest=BitmapGlobalIndexTableTest test
• git diff --check
• ~/.nvm/versions/node/v22.22.3/bin/node ./node_modules/.bin/docusaurus build (build generated successfully; existing site-wide /docs/master/concepts/overview broken-link warnings remain)

[leaves12138]

Thanks for the update. I reviewed the bitmap global index storage format and query paths in detail.

The file layout looks sound to me: the fixed footer carries magic/version and block offsets, null/non-null row sets are separate Roaring64 bitmaps, value bitmaps are addressed through dictionary entries, and the dictionary block index keeps point lookup lazy. The newly added manifest-level min/max/null metadata is also a good optimization because it prunes non-candidate index files before opening the bitmap file, and the fallback scan budget is now applied only to selected files.

I also checked the correctness-sensitive paths:

• equality / IN use serialized-key ordering consistently for dictionary block lookup;
• range predicates deserialize keys and use the logical comparator instead of serialized byte order;
• NOT / NOT IN complement only against each file's non-null bitmap, which keeps SQL null semantics and multi-file unions correct;
• relative row ids are preserved in the index file and offset by the outer range reader;
• string prefix lookup is safe with the current string key serializer, which stores raw UTF-8 bytes.

I ran the focused tests locally on the latest head, plus a temporary numeric negative/multi-dictionary-block equality/range check:

• mvn -pl paimon-common -Pfast-build -DfailIfNoTests=false -Dtest=BitmapGlobalIndexReaderTest test
• mvn -pl paimon-core -am -Pfast-build -DfailIfNoTests=false -Dtest=BitmapGlobalIndexTableTest test

Both passed. Non-blocking thought: for high-cardinality columns each distinct value still becomes one bitmap block plus one dictionary entry, so the docs' guidance to prefer BTree for high-cardinality/range-heavy workloads is important. With the default shard size this is acceptable for the intended enum/tag style use case.

[leaves12138]

Thanks for the update. I re-reviewed the latest head (78f1b46527263b280fa63338522deb8746832cf7) with focus on the bitmap storage format and the shared sorted-index abstractions.

The bitmap file layout still looks sound to me: the footer keeps magic/version and block offsets, null/non-null row bitmaps are stored separately, value bitmaps are referenced through dictionary entries, and the dictionary block index keeps point lookup lazy. The refactor to SortedIndexFileMeta, SortedFileMetaSelector, and ParallelFileGlobalIndexReader also looks reasonable and preserves the previous BTree behavior/compatibility from what I checked.

I also ran the focused common tests locally (BitmapGlobalIndexReaderTest, SortedIndexFileMetaTest, BTreeFileMetaSelectorTest, BTreeThreadSafetyTest, and LazyFilteredBTreeIndexReaderTest), and they passed.

However, the current CI is failing on Checkstyle:

src/test/java/org/apache/paimon/globalindex/btree/BTreeGlobalIndexBuilderTest.java:[30] (imports) ImportOrder: Import org.apache.paimon.globalindex.KeySerializer appears after other imports that it should precede

Please fix the import ordering in BTreeGlobalIndexBuilderTest so the PR can go green.

[leaves12138]

Thanks for the update. I pushed a small follow-up commit to fix the Checkstyle import ordering issue in BTreeGlobalIndexBuilderTest.

I re-checked the latest head and the only change after my previous review is the import reorder, so the previous storage-format review still stands: the bitmap file layout and the shared sorted-index abstractions look good to me.

[leaves12138]

Thanks for the update. I re-reviewed the latest head (2bce2202fdba0abc40a8d7f9add2b5a9c4fe6dd5).

The new split between SortedFileGlobalIndexReader, SortedFileMetaSelector, BitmapIndexReader, and LazyFilteredBitmapReader looks good to me. The bitmap storage layout is still sound, and the new compression only applies to dictionary/index blocks with per-block trailer/CRC, while the row bitmaps remain directly addressable. Manifest pruning, null semantics, fallback-scan budget, and multi-file complement behavior also look covered.

I ran focused tests locally:

mvn -pl paimon-common -Pfast-build -DfailIfNoTests=false -Dtest=LazyFilteredBitmapIndexReaderTest,SortedFileMetaSelectorTest,SortedIndexFileMetaTest,LazyFilteredBTreeIndexReaderTest,BTreeThreadSafetyTest test
# Tests run: 122, Failures: 0, Errors: 0, Skipped: 0

mvn -pl paimon-core -am -Pfast-build -DfailIfNoTests=false -Dtest=BitmapGlobalIndexTableTest test
# Tests run: 4, Failures: 0, Errors: 0, Skipped: 0

No further blockers from my side.