Skip to content

fix(worker): avoid shared err race in multipart upload#443

Open
mesutoezdil wants to merge 1 commit into
NVIDIA:mainfrom
mesutoezdil:mesutoezdil/fix/worker-multipart-upload-race
Open

fix(worker): avoid shared err race in multipart upload#443
mesutoezdil wants to merge 1 commit into
NVIDIA:mainfrom
mesutoezdil:mesutoezdil/fix/worker-multipart-upload-race

Conversation

@mesutoezdil

@mesutoezdil mesutoezdil commented Jul 25, 2026

Copy link
Copy Markdown
Contributor

TL;DR

In multipartUpload, every part-upload goroutine wrote the shared function-scope
err instead of a goroutine-local one. Concurrent parts raced on err
(go test -race flags it), and a failing part whose err was overwritten to nil
by a peer could skip its error return and dereference a nil upload result,
panicking the worker on the large-response upload path. Extract the per-part
upload into uploadPart, keeping the error and result local to each call.

Issues

Closes #442

Testing

go test ./worker/ -race passes. Added TestUploadPart covering success,
failure (returns the error with no nil dereference), and a concurrent run where
one part fails and must not clobber the others. Verified the pre-fix shared-err
pattern reports a data race under -race. No QA needed.

Checklist

  • I am familiar with the Contributing Guidelines.
  • I have signed off my commits for Developer Certificate of Origin (DCO) compliance.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

Summary by CodeRabbit

  • Bug Fixes
    • Improved multipart upload reliability by centralizing per-part retry behavior and standardizing returned completion data.
    • Ensured that failures in one part during concurrent multipart uploads don’t contaminate results from other parts.
  • Tests
    • Added unit tests covering successful part uploads, correct error propagation with no partial results on failure, and concurrent multipart scenarios to verify error isolation per part.

@mesutoezdil
mesutoezdil requested a review from a team as a code owner July 25, 2026 10:20
@mesutoezdil
mesutoezdil requested a review from vrv3814 July 25, 2026 10:20
@coderabbitai

coderabbitai Bot commented Jul 25, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

Multipart part uploads now use a dedicated retrying helper that returns completed-part metadata. Concurrent upload handling delegates to this helper, and tests cover successful uploads, propagated failures, and per-part error isolation.

Changes

Multipart upload isolation

Layer / File(s) Summary
Part upload helper and concurrency tests
src/compute-plane-services/worker-utils/worker/large.go, src/compute-plane-services/worker-utils/worker/large_helpers_test.go
Adds an S3 part uploader abstraction and retrying uploadPart helper, with tests for success, failure propagation, and concurrent per-part error isolation.
Multipart upload integration
src/compute-plane-services/worker-utils/worker/large.go
Updates concurrent multipart uploads to call uploadPart and append returned completed parts under the existing mutex.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Suggested reviewers: vrv3814

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title uses a valid Conventional Commits fix(scope) format and accurately describes the multipart upload race fix.
Linked Issues check ✅ Passed The changes address issue #442 by moving per-part upload error/result state into goroutine-local logic and adding targeted tests.
Out of Scope Changes check ✅ Passed All changes stay focused on multipart upload race handling and test coverage, with no unrelated scope introduced.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 golangci-lint (2.12.2)

level=error msg="[linters_context] typechecking error: pattern ./...: directory prefix . does not contain main module or its selected dependencies"


Comment @coderabbitai help to get the list of available commands.

@mesutoezdil
mesutoezdil force-pushed the mesutoezdil/fix/worker-multipart-upload-race branch from 8f291dc to c2d83bc Compare July 25, 2026 12:24
Each part-upload goroutine wrote the shared multipart function-scope err
instead of a goroutine-local one. Concurrent parts raced on err, and a
failing part whose err was overwritten to nil by a peer could skip its
error return and dereference a nil upload result, panicking the worker.

Extract the per-part upload into uploadPart, which keeps the error and
result local to each call, and add tests covering the success, failure,
and concurrent paths under the race detector.

Closes NVIDIA#442

Signed-off-by: mesutoezdil <mesudozdil@gmail.com>
@mesutoezdil
mesutoezdil force-pushed the mesutoezdil/fix/worker-multipart-upload-race branch from c2d83bc to 126504f Compare July 27, 2026 07:38

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
src/compute-plane-services/worker-utils/worker/large.go (1)

251-253: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Preserve the failing part number in the returned error.

This new error boundary returns the raw SDK error, so errgroup.Wait() cannot identify which part failed. Wrap it with %w and include partInput.PartNumber while preserving errors.Is behavior.

As per path instructions, Go changes under src/** should follow subtree error-handling conventions and preserve error context with %w.

Proposed fix
-		return types.CompletedPart{}, err
+		return types.CompletedPart{}, fmt.Errorf("upload part %d: %w", aws.ToInt32(partInput.PartNumber), err)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/compute-plane-services/worker-utils/worker/large.go` around lines 251 -
253, Update the error return in the part-processing flow to wrap the SDK error
with `%w` and include `partInput.PartNumber` in the message, preserving
`errors.Is` compatibility while allowing `errgroup.Wait()` callers to identify
the failed part.

Source: Path instructions

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@src/compute-plane-services/worker-utils/worker/large.go`:
- Around line 251-253: Update the error return in the part-processing flow to
wrap the SDK error with `%w` and include `partInput.PartNumber` in the message,
preserving `errors.Is` compatibility while allowing `errgroup.Wait()` callers to
identify the failed part.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: b35d84c1-0bf7-4299-ad81-a64761204e3d

📥 Commits

Reviewing files that changed from the base of the PR and between c2d83bc and 126504f.

📒 Files selected for processing (2)
  • src/compute-plane-services/worker-utils/worker/large.go
  • src/compute-plane-services/worker-utils/worker/large_helpers_test.go
🚧 Files skipped from review as they are similar to previous changes (1)
  • src/compute-plane-services/worker-utils/worker/large_helpers_test.go

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

worker multipart upload shares one err across goroutines, racing and risking a nil dereference

1 participant