acceptance: schema-driven invariant fuzzing for bundle configs#5686
Draft
radakam wants to merge 45 commits into
Draft
acceptance: schema-driven invariant fuzzing for bundle configs#5686radakam wants to merge 45 commits into
radakam wants to merge 45 commits into
Conversation
Collaborator
Integration test reportCommit: 36576dc
8 interesting tests: 4 RECOVERED, 4 SKIP
Top 10 slowest tests (at least 2 minutes):
|
Contributor
Approval status: pending
|
denik
reviewed
Jun 26, 2026
- Correct misleading comments: the nightly test-fuzz job runs the same local harness against the fake server (wider seed window + drift on), not a real workspace. - Run config generation inside the per-seed subshell so a generator crash also prints the "reproduce with" hint. - Document the schema-driven fuzz subdir in the invariant README, including that a failure is a real CLI bug and how to reproduce it. - Drop the unused name hint in gen_config (objects ignore it).
Make the comments across the schema fuzz harness more concise while keeping the non-obvious "why" context.
Mirror the integration-test flow: comment on the PR that introduced the failing commit rather than opening/deduping a tracking issue.
…ating it Extract the no_drift deploy/drift/destroy body into a shared no_drift.sh sourced by both the no_drift test and the fuzzer, so the invariant lives in one place and other invariant tests can be fuzzed the same way.
A rejected config never deploys, so emitting the marker made the fuzzer read the re-plan's "needs create" as drift.
The fuzzer runs the shared no_drift.sh body with errexit off and classifies each seed from the captured exit code. The drift block ended with a no-panic check that reset $? to 0, so a config that deployed cleanly but drifted was silently treated as a pass. Accumulate the drift assertions into drift_rc and return it instead. The curated no_drift test (errexit on) is unaffected. Also make verify_no_drift.py fail cleanly on empty/unparseable plan output (when bundle plan itself failed) instead of crashing with a traceback, and tighten the fuzz harness comments.
Extract the migrate invariant body into a shared migrate.sh (mirroring no_drift.sh) and have the fuzzer source ../$FUZZ_INVARIANT.sh so it can exercise any invariant. Wire up FUZZ_INVARIANT=[no_drift, migrate] so the schema fuzzer now also stress-tests the Terraform->direct migration on random configs. The fuzzer's panic scan now globs LOG.* rather than naming LOG.validate/LOG.deploy, since different bodies write different logs.
CatalogsCreate only echoed a subset of the create request, so a re-read returned null for connection_name, managed_encryption_settings, and custom_max_retention_hours. Because connection_name is recreate_on_changes (immutable), the schema fuzzer's no_drift invariant saw a perpetual recreate; the others showed as update drift. Persist these fields on create so the re-read matches the deployed config. Also clamp the fuzzer's custom_max_retention_hours to UC-valid values (0 or 168-720 hours) so generated catalog configs deploy.
…variants Broaden the fuzz invariant matrix beyond no_drift/migrate with four more schema-driven invariant bodies, each selectable via FUZZ_INVARIANT and following the existing INPUT_CONFIG_OK / SKIP_DRIFT_CHECK contract: - redeploy.sh: deploy twice; the second deploy must be a clean no-op, which exercises the write path twice and catches create handlers that don't round-trip their inputs. - canonical.sh: `bundle validate -o json` must be byte-identical across two runs; guards against nondeterministic serialization. Cloud-independent, so it always runs (not gated behind SKIP_DRIFT_CHECK). - update.sh: edit a comment/description and assert the redeploy is an in-place update (not a recreate) that converges with no drift. Configs without an editable field are skipped before the marker (treated as a rejection). - destroy_recreate.sh: deploy then destroy; a re-plan must want to create everything again, proving destroy left no orphaned state. Add two stdlib-only helpers: edit_fuzz_config.py (flips one comment/description scalar via a line match, no YAML dependency) and verify_plan_action.py (asserts a plan shows the expected action, mirroring bundle/deployplan/action.go).
Add common.sh with invariant_deploy and _invariant_cleanup so the deploy-based bodies (no_drift, redeploy, update, destroy_recreate) and migrate no longer duplicate the validate/deploy/cleanup prologue. Unify the panic check on '!panic:' across all invariant bodies and fuzz/script so a random generated token containing "panic" can't be a false positive. Also fold in the supporting helpers: check_schema_types.py (fail loud on a schema type the generator can't produce), gen_fuzz_config_check.py plus its selftest (to_yaml contract), util.load_plan shared by the verify_* scripts, and a pass to shorten comments across the harness.
The comment describes connection_name, managed_encryption_settings and custom_max_retention_hours, which are the fields the fix added. It sat above StorageRoot/ProviderName/ShareName, which were already round-tripped. Move it directly above the three added fields so it annotates them.
Replace the shared `<name>.sh` invariant bodies (sourced via FUZZ_INVARIANT and common.sh) with self-contained invariant test directories that double as fuzz targets, selected by FUZZ_TARGET. Each target runs over the curated INPUT_CONFIG matrix and, when FUZZ_SEED is set, against a schema-generated random config. - Inline the deploy/drift logic into no_drift/script and migrate/script - Re-add redeploy, canonical, update, and destroy_recreate as invariant dirs - Drop common.sh and the standalone *.sh bodies - Point fuzz/script at ../$FUZZ_TARGET/script and refresh the README
The origin/main merge added the volume_path_job_ref.yml.tmpl fuzz template to the INPUT_CONFIG matrix, but the canonical/destroy_recreate/redeploy/update out.test.toml goldens were not regenerated, failing the "changed or new files" CI guard.
The generator produced random catalog_name/schema_name values and random or
empty grants. The fake test server accepts them, but real UC rejects them
(CATALOG_DOES_NOT_EXIST, invalid principal/privilege), so such configs deploy
locally yet drift or fail on cloud, masking real invariant coverage.
Pin catalog_name to "main" and schema_name to "default" (the seeded objects
used by the curated invariant configs), and emit one known-good grant per
grant-bearing securable type ("account users" plus a privilege valid for that
type). This removes the spurious grants drift and the reference-rejection class.
Extend the invariant fuzzer to emit more than one resource per config
(--resource-count, matrixed as FUZZ_RESOURCE_COUNT) and link two of them
with a ${resources.*} reference so the interpolation and deploy-ordering
paths are exercised. The reference targets an input identity field
(name/display_name) so it resolves for every resource type and converges
without drift.
Also improve generated-config validity: skip output-only/computed fields
(x-databricks-field-behaviors OUTPUT_ONLY, readOnly, and a name list) to
avoid false drift after migrate, emit valid permissions per resource type,
force prevent_destroy=false so destroy_recreate can run, and add required
fields the schema omits for registered_models.
Gate the resource key/name suffix on the resource index, not the total count, so a given seed produces the same first resource whether --resource-count is 1 or greater. Later resources stay index-suffixed to remain unique.
inject_cross_ref stopped after a single ${resources.*} edge, so a config
with N resources still exercised only one reference. Link each resource to
an earlier one instead, keeping the graph acyclic so deploy can order it,
and add resource-count 3 to the matrix so the multi-link path runs in CI.
test.toml already lists resource count 3; regenerate the recorded matrix to match so the acceptance framework does not flag a mismatch.
Point file_path/source_code_path fields at the staged data/. fixtures and pin typed-string fields so generated configs deploy instead of getting rejected. Also ignore the local .fuzztmp/ driver scratch directory.
Add a mutate engine (mutate_fuzz_config.py) that applies seeded delete/set/dangerous-value mutations to a curated base config, and a fuzz_gen_config.py dispatcher selected by FUZZ_MODE (generate|mutate). Wire the dispatcher into the six invariant target scripts, matrix FUZZ_MODE in fuzz/test.toml (excluding redundant mutate runs at resource-count 2/3), and add a selftest covering the YAML loader round-trip and mutation determinism.
The schema-walk generator pinned every scalar to a known-good value, so dangerous / near-range-end input was only exercised by mutate mode. Inject DANGEROUS_STRINGS/DANGEROUS_INTS into free-form scalars (~15% of the time) so generate mode probes empty/whitespace/over-long/control-char strings and int32/int64 boundaries while pinned fields keep values that still deploy. Centralize the two lists in gen_fuzz_config.py so mutate_fuzz_config.py reuses them unchanged.
The update invariant edits a comment/description and asserts an in-place update, but some resources classify that field as recreate_on_changes (e.g. model_serving_endpoints.description). A fuzz-generated config that set such a field made the correct recreate look like an invariant bug. edit_fuzz_config.py now reads recreate_on_changes from resources.yml, tracks the enclosing resource type while scanning, and skips immutable comment/description fields (picking a mutable one or reporting none). Adds a contract self-check (edit_fuzz_config_check.py) and a selftest.
The schema fuzzer generates resource types the testserver may not model. Hitting an unregistered route made the testserver call t.Errorf, failing the whole fuzz run for a coverage gap rather than a CLI bug. Add a testserver IgnoreUnhandledRequests option (exposed as a test.toml field, plumbed through startLocalServer) that returns 501 and logs the gap instead of failing. The invariant fuzz script already treats a non-zero result before INPUT_CONFIG_OK as a rejection, so such configs are now skipped cleanly. Curated tests leave the flag false so genuine missing-handler bugs stay loud. Covered by a testserver unit test.
Fuzz variants had two time failure modes that both read as bugs: a single seed could hang indefinitely, and a slow-but-progressing variant (mutate + drift at ~30s/seed) could exceed the per-test timeout across a large seed count. Both surfaced only as an opaque "test timed out". Add a per-seed cap (FUZZ_SEED_TIMEOUT, default 180s) enforced via GNU timeout: a seed past the cap is SIGQUIT'd (Go dumps goroutines, so a real deadlock is diagnosable) then SIGKILL'd, and reported as a hang with a reproduce hint, distinct from a drift bug. Where timeout is unavailable (macOS/Windows) seeds run uncapped as before, and FUZZ_SEED_TIMEOUT=0 disables the cap for live inspection. Add an optional overall budget (FUZZ_TIME_BUDGET): the loop stops launching new seeds past it and exits cleanly, so a slow variant tests as many seeds as fit instead of being force-killed. The nightly task sets it under a raised per-variant Timeout; the committed run keeps its defaults and empty output.
Triaging a fuzz run meant hand-parsing every seed's LOG.* across preserved temp dirs to sort deployed / rejected / testserver-gap / hang / invariant bug. Reuse the classification the loop already computes and append one machine-readable line per seed to LOG.summary, plus a per-variant totals block on a clean run. A 501 "No stub found" is tracked as a coverage gap distinct from a genuine config rejection. LOG.summary is a file, not stdout, so the committed run's empty-output assertion is unaffected.
Trim the multi-line explanations added across the fuzz harness changes to short, why-focused notes; no behavior change.
…n config The parent test.toml sets GOOSOnPR.windows/darwin=false and adds job_run.yml.tmpl, but the per-directory out.test.toml snapshots were stale, failing the CI "detected changed or new files" check.
Four improvements to the bundle schema fuzzer: - Oracle: in fuzz mode (SKIP_DRIFT_CHECK) no_drift/redeploy now assert plan determinism (two consecutive `plan` reads must be byte-identical) instead of only no-panic. This is independent of fake-server fidelity, unlike no-drift or no-destructive-drift (an unchanged config's recreate is just a local-vs-remote representation mismatch on an immutable field). redeploy also panic-scans its second deploy while tolerating fidelity-driven redeploy failures. - Coverage: add `task test-fuzz-cover` to run the corpus under -cover and report per-package coverage (0.0% = never exercised), plus opt-in FUZZ_CORPUS_DIR to persist deployable configs as a generator-independent regression corpus. - Rejection waste: emit non-ASCII scalars as literal UTF-8 (ensure_ascii=False) so hostile values reach bundle logic instead of dying at the YAML parser as an invalid surrogate-pair escape (was the single largest rejection bucket). - Truncation: FUZZ_TIME_BUDGET now defaults on (900s, under the 20m Timeout) so a slow variant stops cleanly and passes rather than being force-killed into a false failure; FUZZ_TIME_BUDGET=0 restores an uncapped run.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
A schema-driven config fuzzer for
acceptance/bundle/invariant/. Each invariant is a self-contained test directory whosescriptdoubles as a curated test over theINPUT_CONFIGmatrix and a fuzz target selected byFUZZ_TARGET:no_drift— deploy, then no driftmigrate— Terraform deploy, migrate to direct, then no driftredeploy— deploy twice; the second must be a no-opcanonical—validate -o jsonbyte-identical across two runsupdate— edit a description; the redeploy must update in place, not recreatedestroy_recreate— deploy then destroy; a re-plan must recreate everythingfuzz_gen_config.pyemits thedatabricks.ymlfor each seed and dispatches onFUZZ_MODE:generate(gen_fuzz_config.py) walks the livedatabricks bundle schema(resolving$ref, picking concreteoneOf/anyOfbranches) and builds a random, schema-valid config from scratch.FUZZ_RESOURCE_COUNTsets how many resources it emits; with more than one, resources are linked via${resources.*}references (each pointing at an earlier one, so the graph stays acyclic) to exercise interpolation and deploy ordering. To stay deployable and drift-free, it pins UC references to seeded objects, emits valid grants and permissions per resource type, skips output-only fields, and forcesprevent_destroy=false.mutate(mutate_fuzz_config.py) starts from a curated single-resource invariant config that already deploys and applies a few seeded mutations (delete a field, replace a scalar with a fuzz token, a boundary/dangerous value, or an empty container). Because the base resolves, it reaches a much higher deploy rate than the schema walk, so it exercises the CLI's handling of perturbed-but-realistic input rather than mostly the reject path.In both modes, free-form scalars are occasionally replaced with dangerous / near-range-end values (empty, whitespace, over-long, control characters, non-ASCII, quotes, a dangling
${...}reference, path traversal, int32/int64 boundaries) to probe the CLI's input handling. Non-ASCII scalars are emitted as literal UTF-8 so they survive YAML parsing and reach bundle logic instead of dying at the parser as an invalid surrogate-pair escape. Seeds are deterministic, so any failure reproduces from the printed seed.The corpus carries a real, fake-server-fidelity-independent oracle. Because the fake server doesn't round-trip every field, the exact no-drift check would false-positive on fuzzed configs (and "no destructive re-plan" is no better — for an unchanged config a recreate is just a local-vs-remote representation mismatch on an immutable field). So in fuzz mode
no_drift/redeployinstead assert plan determinism: two consecutiveplan -o jsonreads of the deployed state must be byte-identical. A diff means nondeterministic planning/serialization (unstable map order, per-run randomness), which is a real bug; curated configs still run the full drift check.Two supporting tools:
task test-fuzz-coverruns the corpus under a coverage-instrumented CLI (CLI_GOCOVERDIR) and reports per-package coverage (ascending, so0.0%= never exercised) — a cheap way to see which code the corpus does and doesn't reach.FUZZ_CORPUS_DIR(opt-in) persists configs that actually deploy, accumulating a generator-independent regression corpus that stays valid even after the generator changes.Why
Random schema-valid and perturbed configs exercise the full deploy path across every resource type and surface panics and invariant violations real-world configs rarely hit. A rejected config isn't a bug — scripts print
INPUT_CONFIG_OKonce a config deploys, so only a panic, a post-marker failure, or a nondeterministic plan counts. Every PR runs a small no-panic window; drift checking is opt-in (FUZZ_CHECK_DRIFT) since the fake server accepts configs the real backend rejects.task test-fuzzand the nightly job sweep a wider window (200 seeds, drift checking on) and report failures on the PR.Slow
mutatedeploy combos can't finish a large seed window inside the per-script Timeout, soFUZZ_TIME_BUDGETnow defaults on (900s, under the 20m Timeout): a slow-but-progressing variant stops cleanly and passes with a recordedLOG.budgetnote rather than being force-killed and read as a failure. SetFUZZ_TIME_BUDGET=0for an uncapped run.Testing
go test ./acceptance -run TestAccept/bundle/invariant— curated invariants across every config.go test ./acceptance -run TestAccept/bundle/invariant/fuzz— generate/mutate, deploy, run invariant, destroy; assert no panic and (in fuzz mode) deterministic plans.task test-fuzz— wide-window run with drift checking (nightly); reproduce a failure withFUZZ_SEED_START=<seed> FUZZ_SEED_COUNT=1 FUZZ_TARGET=<target> FUZZ_MODE=<generate|mutate> task test-fuzz.task test-fuzz-cover— run the corpus under coverage and report which packages it exercises.