fix(data_utils): make GPQA answer permutation reproducible

[AUTHENSOR]

Summary

preprocess_gpqa in verifiers/utils/data_utils.py permutes the answer letters with random.shuffle(letters) using the unseeded process-global RNG. Because:

1. The permutation runs inside dataset.map(preprocess_fn, num_proc=10) — each of the 10 worker processes has an independent unseeded RNG state, and
2. Each python eval.py invocation re-seeds the global RNG fresh,

…the same GPQA question gets a different A/B/C/D answer mapping on every run. This breaks eval reproducibility: a model that is sensitive to answer-letter position receives a different score each time the eval is run.

The seed parameter in preprocess_dataset (line ~299) seeds the HuggingFace dataset's row-order shuffle (dataset.shuffle(seed=seed)), not this answer-letter permutation — so passing --seed does not fix this.

Reproduction

import random

# Two consecutive runs (simulating two interpreter invocations)
def shuffle_once():
    letters = ["A", "B", "C", "D"]
    random.shuffle(letters)
    return letters

run1 = [shuffle_once() for _ in range(5)]
run2 = [shuffle_once() for _ in range(5)]
# 5/5 permutations differ between run1 and run2

In the real eval, this means the correct answer to a given GPQA question is at a different letter position each run.

The fix

Seed a per-question RNG from a stable hash of the question text:

q_hash = int.from_bytes(
    hashlib.sha256(q.encode("utf-8")).digest()[:8], "little"
)
rng = random.Random(q_hash)
rng.shuffle(letters)

Properties:

• Same question → same mapping, reproducible across interpreter invocations and num_proc workers.
• Different questions → different mappings, preserving the position-bias mitigation the shuffle exists to provide.
• hashlib.sha256 is used, not built-in hash(), because Python string hashing is randomized per-process via PYTHONHASHSEED. Verified stable across PYTHONHASHSEED=0/1/12345/random.

Verification

PYTHONHASHSEED=0      -> ['D', 'C', 'B', 'A']
PYTHONHASHSEED=1      -> ['D', 'C', 'B', 'A']
PYTHONHASHSEED=12345  -> ['D', 'C', 'B', 'A']
PYTHONHASHSEED=random -> ['D', 'C', 'B', 'A']

Same question, same permutation regardless of interpreter hash seed. ruff check and ruff format --check both pass.

Scope

This PR is scoped to the GPQA preprocessing path only. I noticed a few other things while reviewing (the RubricGroup.score_group advantage-dropping that PR #2009 partially addresses, and a cluster of sandbox-cleanup race issues tracked in #1992/#1994) — those are out of scope here and already have tracking PRs/issues.

---

Note

Low Risk
Single-path dataset preprocessing change with no security or persistence impact; only affects how GPQA multiple-choice options are ordered.

Overview
GPQA preprocessing no longer uses unseeded random.shuffle for A/B/C/D letter assignment. preprocess_gpqa now derives a per-question seed from SHA256 of the question text and shuffles with a dedicated random.Random instance, so the same question always gets the same answer-letter mapping across runs, workers (num_proc=10), and PYTHONHASHSEED values.

Per-question permutations are unchanged in intent—only reproducibility is fixed. hashlib is added for stable hashing instead of built-in hash().

^{Reviewed by Cursor Bugbot for commit 8b547f0. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Make GPQA answer permutation deterministic per question in preprocess_gpqa

Previously, A/B/C/D option order in GPQA preprocessing used the process-global RNG (random.shuffle), making results vary across runs and processes. Now, a per-question seed is derived from the SHA-256 hash of the question text, and a local random.Random instance is used to shuffle options, making the order stable and reproducible. Behavioral Change: existing datasets or cached results may have a different answer ordering after this change.

^{Macroscope summarized 8b547f0.}