chore: Update test projects to use xunit v3

[filzrev]

This PR migrate test projects to use Microsoft.Testing.Platform (xUnit.v3 and TUnit)

What's changed in this PR

1. Update test projects

• Migrate project to use xUnit v3 and MTP v2 (xunit.v3 v4.x)
• Change OutputType to Exe
• Remove Microsoft.NET.Test.Sdk and xunit.runner.visualstudio packages

2. Modify namespaces
Remove Xunit.Abstractions namespace and add additional using.

3. Modify FactEnvSpecificAttribute/TheoryEnvSpecificAttribute
It's required to resolve xUnit3003 analyzer errors

4. Remove Tests project dependency from IntegrationTests.
Instead, add <Compile> file references.

5. Modify TheoryData related codes.
Modify incompatible code between xUnit v2/v3.

6. Add global.json at solution root
It's required to run tests with Microsoft.Testing.Platform with dotnet test

7. Add tests/.editorconfig‎
It's required to suppress following warnings.

xUnit1051: Calls to methods which accept CancellationToken should use TestContext.Current.CancellationToken to allow test cancellation to be more responsive.

8. Modify InlineDataEnvSpecific.cs and remove InlineDataEnvSpecificDiscoverer.cs
xUnit v3 don't support reflection based discover and needs migrate code.
https://xunit.net/docs/getting-started/v3/migration#removal-of-reflection-abstractions

9. Fix Wasm related tests
After migrated to xUnit.v3 some Wasm related tests failed by AppHost check.
So it need to add workaround code by adding Directory.Build.targets file.
And add <UseAppHost>false</UseAppHost setting to wasm project template.

10. Add settings to suppress warning MSB3277

11. Modify BenchmarkDotNet.IntegrationTests.ManualRunning.MultipleFrameworks.csproj‎
Migrate test framework to use TUnit.
It's required because xUnit v3 don't support .NET 4.6.2.
And It can't mix VsTest/MTP based projects

12. Update BenchmarkDotNet.Build project

• Update Cake.Frosting package
• Add PathType = DotNetTestPathType.Auto option. Because when using dotnet test with MTP mode. It need explicitly specify --project.
• Modify logger parameters for trx.

13. Disable IsTestingPlatformApplication property for auto-generated project templates

When Microsoft.Testing.Platform.MSBuild package is referenced transitively.
It auto-generate Main entry point by default and cause conflicts.
So it need to suppress auto-generated entry point.

14. Modify run-tests-selected workflow to support dotnet test with MTP

15. Add <PreferNativeArm64> setting for .NET Framework tests on windows-11-arm
When migrated tests to MTP and run tests with dotnet test.
It seems x64 emulation is used by default.
So it need enable <PreferNativeArm64> setting to keep existing behaviors.

16. Temporary dissable disassemble tests for macos
This changes are reverted after migrated to ClrMD 4.x (#3081)

17. Add SkipTestWithoutData=true setting to avoid no test data exception

18. Modify test to use dotnet run instead of dotnet test
When running tests with dotnet test. A lot of extra logs are outputted when using --output Detailed
So replace Cake.Fronsting setting to use dotnet run instead.

19. Add testconfig.json and disable telemetry.
If telemetry is enabled. AllocationQuantumIsNotAnIssueForNetCore21Plus test failed with extra memory allocation (792 bytes) on Windows
It looks like some telemetry operation is running background.
So I've explicitly disabled MTP telemetry by setting environment variable.

On windows(arm64) environment, it still failed with extra allocations.
So I've add setting to skip test on this environment.

20. Add --progress off setting
On windows(arm64) environment, TieredJitShouldNotInterfereAllocationResults test failed with 1-2 bytes extra memory allocations.

21. Add setting to skip AwaitingTasksShouldNotInterfereAllocationResults in-process test on Windows(arm64)+.NETFramework
Add skip setting as temporary workaround.
And create separated issue #3203

TODO:

Following tasks are expected to be handled on another PRs.

• Migrate VSTestAdapter project to use MTP (It's implemented on another PR)
• Re-enable following test

  BenchmarkDotNet/tests/BenchmarkDotNet.IntegrationTests/MemoryDiagnoserTests.cs

  Lines 39 to 42 in 2365829

  	// xunit v2 allocates every 100ms on a background timer that makes the tests flaky on Mac/Linux (MSTestEnableParentProcessQuery).
  	// TODO: remove the guard when the test framework is updated.
  	if (OsDetector.IsWindows())
  	yield return [InProcessEmitToolchain.Default];

  
  ThreadingDiagnoserTests failed randomly when using InProcessToolchain on macos(arm64/x64) #3098

• Pass xUnit CancellationToken to BenchmarkDotNet. (It need to migrate async-based API)

[azure-pipelines]

Azure Pipelines:
3 pipeline(s) were filtered out due to trigger conditions.

[azure-pipelines]

Azure Pipelines:
3 pipeline(s) were filtered out due to trigger conditions.

[timcassell]

#3199 Revealed a background timer in xunit v2 causes allocations every 100ms that disappears in xunit v3. What's the status on this @filzrev?

[filzrev]

Extra memory allocation issue is occurred on another tests. (On Windows(x64)+.NET Framework)

failed EngineShouldNotIntroduceBoxing(toolchain: InProcessEmitToolchain) (31s 545ms)
  Assert.Equal() Failure: Values differ
  Expected: 0
  Actual:   16384

[filzrev]

On PR #3169, following setting is removed from IntegrationTests project.

  <!-- Disable EventSource to stabilize MemoryDiagnoserTests. https://github.com/dotnet/BenchmarkDotNet/pull/2562#issuecomment-2081317379 -->
    <RuntimeHostConfigurationOption Include="System.Diagnostics.Tracing.EventSource.IsSupported" Value="false" />

Extra memory allocation might be caused by event source subscription.
It might be better to skip these instable tests (InProcessEmitToolchain tests on Windows) like #1925.

[timcassell]

On PR #3169, following setting is removed from IntegrationTests project.

  <!-- Disable EventSource to stabilize MemoryDiagnoserTests. https://github.com/dotnet/BenchmarkDotNet/pull/2562#issuecomment-2081317379 -->
    <RuntimeHostConfigurationOption Include="System.Diagnostics.Tracing.EventSource.IsSupported" Value="false" />

Extra memory allocation might be caused by event source subscription. It might be better to skip these instable tests (InProcessEmitToolchain tests on Windows) like #1925.

Did you observe that to be an issue? It was added previously without verification that it was actually an issue.

[filzrev]

Did you observe that to be an issue? It was added previously without verification that it was actually an issue.

Currently, It occurred on CI and it's not occurred when running flakey tests multiple times with run-tests-selected.yaml workflow.
Attritionary, it can't reproduced on local build. (At least, it can't reproduced with Until Failed feature of VS)

Memory allocation is captured by using GC.GetTotalAllocatedBytes(on .NET8) so if there are active background thread exists.
It'll report extra memory allocation when using InProcessToolchain.

[timcassell]

Because there are potentially a lot of background workers that we may have little or no control over in testing frameworks, I think it's reasonable to remove InProcess tests that measure process-wide to avoid flaky results. That includes memory, threading, and exception diagnoser tests.

[timcassell]

11. Modify BenchmarkDotNet.IntegrationTests.ManualRunning.MultipleFrameworks.csproj‎
Migrate test framework to use TUnit.
It's required because xUnit v3 don't support .NET 4.6.2.
And It can't mix VsTest/MTP based projects

Any reason not to use TUnit everywhere?

[filzrev]

Any reason not to use TUnit everywhere?

Migration cost reason.
Once it's migrated to xUnit.v3. It can possible migrate to TUnit per project basis.

[filzrev]

Because there are potentially a lot of background workers that we may have little or no control over in testing frameworks,
I think it's reasonable to remove InProcess tests that measure process-wide to avoid flaky results.
That includes memory, threading, and exception diagnoser tests.

I don't known why Windows OS only affected by xUnit.v3 migration.
And it need to determine what's background worker of xUnit.v3/Microsoft.Testing.Platform affects results.

So for the time being, add temporary workaround for flakey test instead of completely exclude InProcessEmitToolchain from memory/threading/exception diagnoser tests.