Skip to content

rewrite of sei-load generator and sender.#56

Open
pompon0 wants to merge 37 commits into
mainfrom
gprusak-nonce-reset
Open

rewrite of sei-load generator and sender.#56
pompon0 wants to merge 37 commits into
mainfrom
gprusak-nonce-reset

Conversation

@pompon0

@pompon0 pompon0 commented Jun 26, 2026

Copy link
Copy Markdown
Collaborator

Primary reason for the rewrite is support for non-happy paths for autobahn loadtesting - rejected transactions/not successfully executed/etc. The idea is that if sending a tx fails, we need to check the account nonce before sending the next transaction. Note that this affects only accounts from the long-lived pools - accounts generated for the sake of sending just 1 transaction (aka "new accounts") do not benefit from this extra logic.

The nonces, as well as the txs waiting to be sent are managed by TxsQueue, which ensures that:

  • inserted (generated) transactions have expected nonces
  • transactions per account are sent sequentially (otherwise they will be rejected by autobahn)

TxsQueue is managed by ShardedSender.

Additionally

  • removed unnecessary abstractions in both sender and generator logic
  • made Generator.Run accept the sender which exposes Nonce(), Flush() and Send() calls
  • added Generator <-> Sender integration tests
  • removed open/closed loop logic (might need to be reintroduced later if needed)
  • removed utils/rng in favor of just using a single Rand synchronously (we don't have other use cases). If concurrent Rand usage is needed, a SplitRng method was introduced (slower but robust alternative to SplitMix).

@pompon0 pompon0 requested a review from bdchatham June 26, 2026 16:14
@pompon0 pompon0 marked this pull request as ready for review June 29, 2026 16:59
@cursor

cursor Bot commented Jun 29, 2026

Copy link
Copy Markdown

PR Summary

High Risk
Core load-generation and submission path is redesigned (nonce recovery, queue semantics, removal of open-loop scheduling), which can change throughput, replay behavior, and failure handling in production load tests.

Overview
Rewrites the load-test generator and sender so failed sends on long-lived accounts can recover via on-chain nonce checks, and strips several layers of indirection.

Sender: The dispatcher, open/closed-loop scheduler, and sharded worker queues are gone. ShardedSender feeds a new TxsQueue that enforces per-address nonce order, caps backlog with MaxInFlight, and on RPC failure for tracked senders refetches nonce and **Reset**s queued txs. One ethClient dials all endpoints and picks a client by address hash; sends run in spawned goroutines after the shared rate limiter.

Generator: Generator.Run(ctx, rng, txSender) drives the campaign: weighted scenarios are expanded and shuffled once, then each tick pulls Nonce from the sender, builds txs with an explicit *rand.Rand, and **Send**s. Prewarm lives on Generator.Prewarm. Gas/distribution sampling no longer uses SetStream / utils/rng—callers pass Rand into SampleIndex / GenerateGas.

Other: Scenarios return signed Transaction + errors instead of LoadTx; FundAccounts takes a flat address list; settings Validate only requires MaxInFlight > 0 (arrival-model checks removed). Large test suites for the old scheduler/dispatcher are deleted; integration tests cover generator ↔ sender nonce rewind behavior.

Reviewed by Cursor Bugbot for commit a28e7c0. Bugbot is set up for automated code reviews on this repo. Configure here.

Comment thread sender/sharded_sender.go
Comment thread sender/txs_queue.go
Comment thread main.go
Comment thread sender/txs_queue.go
if nonce != state.nextNonce {
// It is expected in case of send failure.
log.Printf("bad nonce for %v: got %v, want %v", addr, nonce, state.nextNonce)
return nil

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dropped txs look successful

Medium Severity

When TxsQueue.Push rejects a transaction for a nonce mismatch, it logs and returns nil. Generator.Run treats that as a successful send and immediately builds another transaction, which can spin at full speed without making progress until context cancel if the queue and generator stay desynchronized.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 91db965. Configure here.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WAI

Comment thread sender/sharded_sender.go
}, nil
cfg: cfg,
queue: NewTxsQueue(cfg.Settings.MaxInFlight),
limiter: limiter,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Buffer size no longer bounds queue

Medium Severity

NewShardedSender sizes TxsQueue only from Settings.MaxInFlight. Settings.BufferSize and LoadConfig.TotalQueueSize() are no longer used, so existing configs that tune bufferSize/--buffer-size do not limit queue depth or memory as before.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 34ba3fe. Configure here.

Comment thread sender/metrics.go
attribute.String("chain_id", stats.ChainID),
))
}
}*/

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Worker queue metric disabled

Low Severity

The worker_queue_length observable gauge callback no longer records observations: the ShardStats loop is commented out and only assigns _ = ss. Scrapers always see an empty series after the sender rewrite removed per-shard queues.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 34ba3fe. Configure here.

Comment thread generator/generator.go Outdated
Comment thread generator/generator.go
Comment thread generator/generator.go
}
ltx := &types.LoadTx{EthTx: tx, IntendedSendTime: time.Now(), Scenario: scenario}
if err := txSender.Send(ctx, ltx); err != nil {
return err

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale nonce before async send

Medium Severity

Run reads txSender.Nonce, builds and signs the tx, then calls Send without holding sender/queue state. A concurrent send failure and Reset can advance the queue nonce before Push, producing signed txs the queue rejects.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit b83e882. Configure here.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rejection is silent and it is expected, since we send the txs asynchronously.

Comment thread main.go

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

There are 5 total unresolved issues (including 4 from previous reviews).

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit a28e7c0. Configure here.

Comment thread main.go
// Start the sender (starts all workers)
s.SpawnBgNamed("sender", func() error { return sharedSender.Run(ctx) })
log.Printf("✅ Connected to %d endpoints", len(cfg.Endpoints))
snd = sharedSender

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Workers setting no longer applied

Medium Severity

The send path now runs a single ShardedSender background task and no longer uses Settings.TasksPerEndpoint (workers) or LoadConfig.GetNumShards, even though those settings remain in config, CLI, and tests. Raising --workers does not increase RPC parallelism as documented.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit a28e7c0. Configure here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants