Add support for Deletion Vectors to MultiFileParquetPartitionReader #13744

razajafri · 2025-11-11T00:31:33Z

Description

Currently, we don't have support for the small file optimization in MultiFileParquetPartitionReader when Deletion Vectors are enabled on a Delta Table. If a query is run on such a table, the plugin will fall back to the MultiFileCloudParquetPartitionReader
With this optimization, if small files are present on a local system, the files will be processed much faster by coalescing them
This PR adds Deletion Vector processing for every batch by locating the original file used to create the buffer and read it's deletion vector.
A new integration test has been added, in addition to that the feature was tested against the baseline and results matched.
Due to creation of many deletion vectors, the tests were taking a long time. To speed up the tests, the test data files have been added to the test resource

Performance

	Baseline - GPU with FileScourceScan fallback to CPU	GPU COALESCING with DV
Percentage Deleted	Single run	Single run	Speedup
5% (500 small files)	8720.0	8111.0	1.08
10% (500 small files)	8556.0	8001.0	1.07
20% (500 small files)	8290.0	8501.0	0.98
40% (500 small files)	8278.0	8580.0	0.96

Baseline: commit id - 21afb61
Target: This PR
Dataset: TPC-DS (sf100_parquet)
Environment: Local
Spark Configs

export SPARK_CONF=("--master" "local[16]"
                   "--conf" "spark.driver.maxResultSize=2GB"
                   "--conf" "spark.driver.memory=50G"
                   "--conf" "spark.executor.cores=16"
                   "--conf" "spark.executor.instances=1"
                   "--conf" "spark.executor.memory=16G"
                   "--conf" "spark.driver.maxResultSize=4gb"
                   "--conf" "spark.sql.files.maxPartitionBytes=2gb"
                   "--conf" "spark.sql.adaptive.enabled=true"
                   "--conf" "spark.plugins=com.nvidia.spark.SQLPlugin"
                   "--conf" "spark.rapids.memory.host.spillStorageSize=16G"
                   "--conf" "spark.rapids.memory.pinnedPool.size=8g"
                   "--conf" "spark.rapids.sql.concurrentGpuTasks=3"
                   "--conf" "spark.rapids.sql.explain=all"
                   "--conf" "spark.sql.warehouse.dir=/home/rjafri/spark-warehouse"
                   "--conf" "spark.sql.legacy.createHiveTableByDefault=false"
                   "--conf" "spark.databricks.delta.deletionVectors.useMetadataRowIndex=false"
                   "--conf" "spark.rapids.sql.format.parquet.reader.type=COALESCING"
                   "--packages" "io.delta:delta-spark_2.12:3.3.0"
                   "--conf" "spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension"
                   "--conf" "spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog"
                   "--conf" "spark.driver.extraClassPath=$SPARK_RAPIDS_PLUGIN_JAR:$NDS_LISTENER_JAR"
                   "--conf" "spark.executor.extraClassPath=$SPARK_RAPIDS_PLUGIN_JAR:$NDS_LISTENER_JAR")

Query: select sum(ss_list_price) from store_sales

Checklists

This PR has added documentation for new or modified features or behaviors.
This PR has added new tests or modified existing tests to cover new code paths.
(Please explain in the PR description how the new code paths are tested, such as names of the new/existing tests that cover them.)
Performance testing has been performed and its results are added in the PR description. Or, an issue has been filed with a link in the PR description.

Signed-off-by: Raza Jafri <raza.jafri@gmail.com>

gerashegalov · 2025-11-12T19:38:58Z

The PR branch contains 4.7K files, need to drop unrelated files

razajafri · 2025-11-12T19:55:43Z

The PR branch contains 4.7K files, need to drop unrelated files

I added the deletion vectors as part of the PR because the tests were taking too long to run. The files are being added via Git LFS.

gerashegalov · 2025-11-13T01:22:55Z

IMO We need to find a way to have fast tests without adding 4.7K files.

razajafri · 2025-11-13T01:37:00Z

IMO We need to find a way to have fast tests without adding 4.7K files.

I have reduced the number of test files.

gerashegalov · 2025-11-13T04:15:17Z

IMO We need to find a way to have fast tests without adding 4.7K files.

I have reduced the number of test files.

you can reduce further by getting rid of the crc checksum files

greptile-apps · 2025-11-13T17:39:23Z

Greptile Overview

Greptile Summary

This PR successfully adds deletion vector support to the MultiFileParquetPartitionReader (coalescing reader), enabling performance optimization for small files on local systems when deletion vectors are present on Delta tables.

Key Changes:

Extended MultiFileCoalescingPartitionReaderBase with a finalizeOutputBatch callback that allows subclasses to process batches with extra context (like file boundaries and row indices)
Implemented DeltaCoalescingFileParquetPartitionReader that tracks which parquet files contribute to each batch using boundaries calculated from row counts
Added CoalescedRapidsDropMarkedRowsFilter that handles deletion vectors across multiple coalesced files by adjusting offsets based on file boundaries
Modified getCoalescingIterator to track row indices across batches for proper deletion vector application
Added comprehensive integration test with pre-generated test data to validate correctness

Architecture:
The implementation cleverly uses boundaries (cumulative row counts) to determine which files contribute to a given batch, then creates a coalesced filter that applies the appropriate deletion vector with offset adjustments for each contributing file. This ensures deleted rows are correctly identified even when multiple small files are coalesced into a single batch.

Performance:
The PR description shows performance is comparable to baseline (0.94-1.01x), which is expected since the optimization primarily benefits from coalescing small files rather than improving DV processing itself.

Confidence Score: 5/5

This PR is safe to merge with high confidence
The implementation is well-designed with proper boundary tracking, comprehensive testing, and follows established patterns in the codebase. The logic for mapping row indices to source files using boundaries is sound, and the offset adjustments in CoalescedRapidsDropMarkedRowsFilter correctly handle deletion vectors across coalesced files. Performance testing shows expected results, and the new integration test validates correctness with both sequential and random deletion patterns.
No files require special attention

Important Files Changed

File Analysis

Filename	Score	Overview
delta-lake/common/src/main/delta-33x-40x/scala/com/nvidia/spark/rapids/delta/common/GpuDeltaParquetFileFormatBase.scala	5/5	Added deletion vector support for coalescing reader in `DeltaCoalescingFileParquetPartitionReader` with proper boundary tracking and offset calculation
delta-lake/common/src/main/delta-33x-40x/scala/com/nvidia/spark/rapids/delta/common/RapidsRowIndexFilters.scala	5/5	Implemented `CoalescedRapidsDropMarkedRowsFilter` to handle deletion vectors across multiple coalesced files with proper offset adjustments
sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuMultiFileReader.scala	5/5	Added `finalizeOutputBatch` callback mechanism to `MultiFileCoalescingPartitionReaderBase` for per-batch processing with extra info
integration_tests/src/main/python/delta_lake_delete_test.py	5/5	Added comprehensive test `test_deletion_vectors_coalescing_multiple_files` for coalescing reader with deletion vectors using pre-generated test data

Sequence Diagram

sequenceDiagram
    participant User
    participant DeltaMultiFileReaderFactory
    participant DeltaCoalescingFileParquetPartitionReader
    participant MultiFileParquetPartitionReader
    participant RapidsDeletionVectorUtils
    participant CoalescedRapidsDropMarkedRowsFilter
    
    User->>DeltaMultiFileReaderFactory: createColumnarReader(partition)
    DeltaMultiFileReaderFactory->>DeltaMultiFileReaderFactory: Check if coalescing or multi-threaded
    
    alt Coalescing (Local Files)
        DeltaMultiFileReaderFactory->>DeltaCoalescingFileParquetPartitionReader: Create reader
        DeltaCoalescingFileParquetPartitionReader->>MultiFileParquetPartitionReader: readBatch()
        MultiFileParquetPartitionReader->>MultiFileParquetPartitionReader: Coalesce multiple small files
        MultiFileParquetPartitionReader->>DeltaCoalescingFileParquetPartitionReader: Return batch
        DeltaCoalescingFileParquetPartitionReader->>RapidsDeletionVectorUtils: getCoalescedRowIndexFilter()
        RapidsDeletionVectorUtils->>RapidsDeletionVectorUtils: Find relevant files based on boundaries
        RapidsDeletionVectorUtils->>CoalescedRapidsDropMarkedRowsFilter: Create filter with offsets
        CoalescedRapidsDropMarkedRowsFilter->>DeltaCoalescingFileParquetPartitionReader: Return filter
        DeltaCoalescingFileParquetPartitionReader->>RapidsDeletionVectorUtils: processBatchWithDeletionVector()
        RapidsDeletionVectorUtils->>RapidsDeletionVectorUtils: Apply deletion vector to batch
        RapidsDeletionVectorUtils->>DeltaCoalescingFileParquetPartitionReader: Return filtered batch
        DeltaCoalescingFileParquetPartitionReader->>User: Return batch with deleted rows marked
    else Multi-threaded (Cloud Files)
        DeltaMultiFileReaderFactory->>DeltaMultiFileParquetPartitionReader: Create reader
        DeltaMultiFileParquetPartitionReader->>DeltaMultiFileParquetPartitionReader: get()
        DeltaMultiFileParquetPartitionReader->>RapidsDeletionVectorUtils: getRowIndexFilter()
        RapidsDeletionVectorUtils->>DeltaMultiFileParquetPartitionReader: Return filter
        DeltaMultiFileParquetPartitionReader->>RapidsDeletionVectorUtils: processBatchWithDeletionVector()
        RapidsDeletionVectorUtils->>DeltaMultiFileParquetPartitionReader: Return filtered batch
        DeltaMultiFileParquetPartitionReader->>User: Return batch with deleted rows marked
    end

greptile-apps

_{87 files reviewed, no comments}

_{Edit Code Review Agent Settings | Greptile}

razajafri · 2025-11-14T00:19:22Z

The performance numbers aren't good for this PR. I am working on improving performance.

nvauto · 2025-11-17T01:34:20Z

NOTE: release/25.12 has been created from main. Please retarget your PR to release/25.12 if it should be included in the release.

greptile-apps

_{88 files reviewed, no comments}

_{Edit Code Review Agent Settings | Greptile}

razajafri · 2025-11-25T00:09:31Z

Here is the breakdown of performance numbers when benchmarking the Time to materialize

% deleted	MULTITHREADED	COALESCING
5	158 ms	932 ms
10	214 ms	1.8 s
20	238 ms	2.4 s
40	242 ms	4.1 s

Breaking it further reveals that as the delete percentage increases, the time taken to add offsets to the bitmap becomes the dominant factor in determining the Time to materialize

% deleted	Materialize
	MULTITHREADED	COALESCING
	Time to create array	Time to create array	Time to add offsets
5	15 ms	72 ms	254 ms
10	50 ms	125 ms	612 ms
20	80 ms	155 ms	971 ms
40	93 ms	271 ms	2200 ms

A table representing the performance numbers above as a percentage of the total materialization time reveals the adding offsets as a major contributor to the slowness

Array creation as percentage of total materialization
% deleted	MULTITHREADED	COALESCING
	Time to create array	Time to create array	Time to add offsets
5	9.5	7.7	27.3
10	23.4	7	34
20	33.6	6.5	40.5
40	38.4	6.6	53.7

razajafri added 7 commits November 12, 2025 08:47

coalescing reader

c3b94d5

save deletion vectors in test resources to save time

c4cc87c

Signing off

bb94534

Signed-off-by: Raza Jafri <raza.jafri@gmail.com>

updated test method to not collect

0c1fc79

Added schema to the dataframe before returning

57603e1

Moved the conf local to the file

144a8f7

Fix for no deletion vectors

a9ef584

razajafri force-pushed the SP-13617-coalescing-reader branch from d24af74 to a9ef584 Compare November 12, 2025 18:14

reduced the test file size

532ec5d

sameerz added the feature request New feature or request label Nov 13, 2025

removed crc files

6108a84

greptile-apps bot reviewed Nov 13, 2025

View reviewed changes

Performance improvment

39a9518

razajafri changed the base branch from main to release/25.12 November 20, 2025 18:35

greptile-apps bot reviewed Nov 20, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add support for Deletion Vectors to MultiFileParquetPartitionReader #13744

Add support for Deletion Vectors to MultiFileParquetPartitionReader #13744

Uh oh!

razajafri commented Nov 11, 2025 •

edited

Loading

Uh oh!

gerashegalov commented Nov 12, 2025

Uh oh!

razajafri commented Nov 12, 2025

Uh oh!

gerashegalov commented Nov 13, 2025

Uh oh!

razajafri commented Nov 13, 2025

Uh oh!

gerashegalov commented Nov 13, 2025

Uh oh!

greptile-apps bot commented Nov 13, 2025 •

edited

Loading

Uh oh!

greptile-apps bot left a comment

Uh oh!

razajafri commented Nov 14, 2025

Uh oh!

nvauto commented Nov 17, 2025

Uh oh!

greptile-apps bot left a comment

Uh oh!

razajafri commented Nov 25, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Add support for Deletion Vectors to MultiFileParquetPartitionReader #13744

Are you sure you want to change the base?

Add support for Deletion Vectors to MultiFileParquetPartitionReader #13744

Uh oh!

Conversation

razajafri commented Nov 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Performance

Checklists

Uh oh!

gerashegalov commented Nov 12, 2025

Uh oh!

razajafri commented Nov 12, 2025

Uh oh!

gerashegalov commented Nov 13, 2025

Uh oh!

razajafri commented Nov 13, 2025

Uh oh!

gerashegalov commented Nov 13, 2025

Uh oh!

greptile-apps bot commented Nov 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Overview

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Sequence Diagram

Uh oh!

greptile-apps bot left a comment

Choose a reason for hiding this comment

Uh oh!

razajafri commented Nov 14, 2025

Uh oh!

nvauto commented Nov 17, 2025

Uh oh!

greptile-apps bot left a comment

Choose a reason for hiding this comment

Uh oh!

razajafri commented Nov 25, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

razajafri commented Nov 11, 2025 •

edited

Loading

greptile-apps bot commented Nov 13, 2025 •

edited

Loading