TestDistributedTracing goldfile approach

[dsmiley]

Goal: make it easy & insightful to maintain TestDistributedTracing
Risk: determinism is hard

I hate maintaining TestDistributedTracing... it kills my motivation to work on tracing. One might say nowadays just let Claude do it but that's a cop-out. I want to test observability by examining observability in data form (e.g. json), asserting that the data is what we expect it to be. I do not want to maintain bespoke unit test assertions that try to look at certain aspects of the data.

I recognize this has challenges... basically determinism. But we can combat that -- fix the seed, lay out the replicas in a specified order (deterministic leader not random)... and we can scrub/normalize values that are likely to vary (e.g. containing host/port). And this whole approach requires some test infra -- some non-trivial code. Personally, I'd rather maintain that any day over the status quo. Furthermore, if somebody makes some change that breaks this test, it needs to be super easy to know how to regenerate it. I think the test infra here shows that it's easy. I've done a similar setup conceptually at my day job for something different.

Testing against data leads to insights, positive and negative, that are absent when we only have bespoke assertions. Seeing data sheds a light on what's great and what needs work. For example, it's apparent we don't have enough information in some of the spans to understand what that request was all about (e.g. a distributed search "purpose" or what indexing may or may not have happened on an /update).

[dsmiley]

Gonna merge this Monday if no feedback.