Run Microsoft.Testing.Platform test apps under vstest.console and datacollector#16201
Conversation
…ence
Add ExecutionPreference {Default, MicrosoftTestingPlatform} to ObjectModel and
SourceDetail. Detect MTP apps in AssemblyMetadataProvider by reading the
assembly-level AssemblyMetadata("Microsoft.Testing.Platform.Application", "true")
attribute, expose it through InferHelper.DetectExecutionPreference, and thread a
per-source ExecutionPreference map through TestRequestManager into SourceDetail.
TestEngine now groups unique run configurations by ExecutionPreference as well as
framework/architecture, and forces isolation for MTP sources so they never run
in-process. This is the detection/plumbing groundwork for routing MTP sources to
an MTP-protocol proxy.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Implements the Microsoft.Testing.Platform (MTP) execution path in CrossPlatEngine: - MtpServerConnection: launches an MTP app in `--server` mode, opens a loopback TCP listener that the app dials back into, and speaks JSON-RPC 2.0 with Content-Length framing. Raises testUpdates/log events. - MtpTestNodeConverter: converts pure MTP test nodes (uid, display-name, execution-state, time.duration-ms, error.*, location.*, traits) into vstest TestCase/TestResult. vstest.* bridge props are used only as optional enrichment so an app with no vstest dependency still converts. - MtpProxyDiscoveryManager / MtpProxyExecutionManager: IProxyDiscoveryManager / IProxyExecutionManager implementations that drive an MTP app per source, translate node updates into discovery/run events, and collect attachments. - TestEngine: routes discovery and execution to the MTP proxies when a source's ExecutionPreference is MicrosoftTestingPlatform. The MTP code is guarded with `#if NETCOREAPP` and uses System.Text.Json, matching the CommunicationUtilities pattern. CrossPlatEngine now also targets net8.0 so the runner loads an MTP-enabled flavor on modern .NET; net462 falls back to the normal path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
net8.0 Exe with EnableMSTestRunner=true so it builds as a Microsoft.Testing.Platform app (carries the Microsoft.Testing.Platform.Application metadata attribute that vstest's MTP detection keys on). Four tests: two pass, one fails, one skipped - used to validate outcome mapping through the new MTP proxies. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ected GetTestRuntimeProvidersForUniqueConfigurations looked up a vstest ITestRuntimeProvider (testhost) for every source. MTP sources have none - they're driven directly over the MTP protocol - so the group got a null-Type TestRuntimeProviderInfo, which tripped the 'No suitable test runtime provider' guard and made the parallel managers treat the workload as non-runnable (HasProvider checks Type != null). For MTP source groups, register a sentinel TestRuntimeProviderInfo(typeof(ITestRuntimeProvider)) under NETCOREAPP so the guard passes and the workload is runnable; the discovery/execution manager creators then route to MtpProxyDiscoveryManager / MtpProxyExecutionManager based on ExecutionPreference. First full end-to-end: vstest.console (net8.0) runs a pure-MTP MSTest app over the MTP protocol - discovery lists all tests, execution reports Passed 2 / Failed 1 / Skipped 1 with error message and stack trace mapped through to the console reporter. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
MtpProxyExecutionManager can now own an IProxyDataCollectionManager. Before the run it calls BeforeTestRunStart to spin up datacollector.exe and merges the profiler environment variables it returns into the env vars injected into the MTP application launch. After each MTP app connects, TestHostLaunched is called with the app's process id so the collector can track it. When the run completes, AfterTestRunEnd is called and its attachments (e.g. the .coverage file) and invoked data collectors are merged into TestRunCompleteEventArgs. The MTP proxy drives HandleTestRunComplete directly rather than via raw ExecutionComplete messages, so the data-collection lifecycle is handled inline instead of reusing DataCollectionTestRunEventsHandler (which hooks the raw-message path). TestEngine routes MTP sources to the data-collection-enabled MtpProxyExecutionManager when data collection is enabled in runsettings. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Plumb ExecutionPreference through RunConfiguration (parse/emit) and into the per-source runsettings so DefaultTestHostManager and DotnetTestHostManager return false for Microsoft.Testing.Platform sources in CanExecuteCurrentRunConfiguration. A vstest testhost can no longer claim an MTP app; routing falls through to the MTP proxies instead. Also regenerate expected-dll-frameworks.json from a clean Release pack: the netcore console now ships the net8.0 CrossPlatEngine build (the one that carries the MTP path) rather than the netstandard2.0 one. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
MtpPureProject references only Microsoft.Testing.Platform. It hand-rolls its own ITestFramework and reports test nodes over the MTP protocol directly, with no MSTest, no VSTestBridge, and no Microsoft.TestPlatform.ObjectModel anywhere in its closure. Deployed output carries just Microsoft.Testing.Platform.dll + the app itself (the MSTest MTP asset drags in 4 vstest DLLs: ObjectModel, VSTestBridge, CoreUtilities, PlatformAbstractions). Proven end-to-end under vstest.console via the MTP provider: - detected as MTP through the Microsoft.Testing.Platform.Application assembly metadata - Failed 1 / Passed 2 / Skipped 1 / Total 4 (matches the native MTP run) - canonical --collect:"Code Coverage" produced a real .coverage: Calculator.Add 2/2 blocks, Multiply 5/5, Divide 0/5 (uncovered by design) This is the ultimate demonstration that a framework with no vstest lineage runs under vstest.console purely over MTP, including code coverage. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
GetTestHostManagerByRunConfiguration already received the sources list but threw it away (the param was named `_`). Use it: providers that implement the new internal ISourceAwareTestRuntimeProvider get first refusal based on the actual sources, before the source-blind providers that only match by target framework. This lets a more specific provider (e.g. the upcoming MTP provider) claim a source by its shape and win without a global ordering scheme, and without relying on the other providers to decline. Providers that don't implement the interface keep their existing source-blind behaviour, and passing null sources skips the first pass entirely, so existing runs are unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
TestEngine already has the source (SourceDetail.Source) at every point where it decides how to route a run, so let it derive whether a source is a Microsoft.Testing.Platform app from the assembly itself instead of reading a pre-baked ExecutionPreference that had to be computed upstream and threaded through SourceDetail/RunConfiguration. The MTP detection (the [assembly: AssemblyMetadata(...)] PEReader probe) moves into a shared MicrosoftTestingPlatformDetector in CoreUtilities, which already uses PEReader and is befriended by CrossPlatEngine, vstest.console and the TestHostRuntimeProvider. vstest.console's AssemblyMetadataProvider now delegates to it instead of carrying its own copy of the (subtle) attribute-blob parsing. TestEngine memoizes the result per source path (concurrent, since the parallel manager creators run on multiple threads). ExecutionPreference is still set upstream for now; TestEngine simply no longer depends on it. Grouping, isolation, the sentinel provider and the MTP proxy routing all key on the source instead. Verified: pure-MTP asset still runs Failed1/Passed2/Skipped1/Total4 under vstest.console; CrossPlatEngine.UnitTests (665) and Common.UnitTests (396) green. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
TestEngine and the runtime-provider selection now detect Microsoft.Testing.Platform apps straight from the source (the earlier two commits), so the ExecutionPreference enum and all the plumbing that carried it through runsettings is dead weight. Delete the enum, its SourceDetail/RunConfiguration properties (parse + emit), the InferRunSettingsHelper node, InferHelper.DetectExecutionPreference and the sourceToExecutionPreferenceMap threaded through TestRequestManager, plus the inert "decline MTP" branches in the Default/Dotnet host managers. The ExecutionPreference public API entries were unshipped, so this is a clean delete. Build green, pure-MTP E2E still Failed1/Passed2/Skipped1/Total4, and ObjectModel, Utilities, vstest.console, CrossPlatEngine and Common unit suites pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
A source-aware provider that declined by source in the first pass was still re-evaluated in the second, source-blind pass -- redundant work, and it could be wrongly re-admitted by matching only the target framework. Exclude ISourceAwareTestRuntimeProvider from the second pass so the manager owns the first-refusal contract instead of relying on each provider to return false when asked the source-blind question. The test double now returns true from its source-blind method to prove the exclusion is enforced by the manager. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Adds proof-of-concept support for running Microsoft.Testing.Platform (MTP) test applications directly under vstest.console/datacollector by detecting MTP-marked sources and routing discovery/execution through new MTP JSON-RPC proxy managers (while keeping classic vstest testhost-based projects working, including mixed runs).
Changes:
- Introduces MTP source detection via assembly metadata and uses it to route CrossPlatEngine discovery/execution to new MTP proxy managers.
- Adds MTP JSON-RPC client/protocol implementation and node-to-vstest result conversion for TRX + data collection interop.
- Adds new MTP-only and MTP+MSTest test assets plus unit tests for the new source-aware runtime provider selection path.
Reviewed changes
Copilot reviewed 24 out of 24 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| test/vstest.ProgrammerTests/Fakes/FakeAssemblyMetadataProvider.cs | Updates fake to satisfy new metadata-provider surface. |
| test/TestAssets/MtpPureProject/PureTestFramework.cs | Adds a pure MTP framework asset emitting nodes over MTP. |
| test/TestAssets/MtpPureProject/Program.cs | MTP app entry point registering the pure test framework. |
| test/TestAssets/MtpPureProject/MtpPureProject.csproj | Defines a net8.0 “pure MTP” test application asset and stamps MTP detection metadata. |
| test/TestAssets/MtpPureProject/Calculator.cs | Simple code-under-test to validate execution + coverage. |
| test/TestAssets/MtpMSTestProject/UnitTests.cs | Adds MSTest tests for the MTP+MSTest asset. |
| test/TestAssets/MtpMSTestProject/MtpMSTestProject.csproj | Defines an MSTest-based MTP app using MTP MSBuild integration. |
| test/Microsoft.TestPlatform.Common.UnitTests/Hosting/TestHostProviderManagerTests.cs | Adds coverage for source-aware provider preference and fallback behavior. |
| src/vstest.console/CommandLine/Interfaces/IAssemblyMetadataProvider.cs | Extends interface with MTP-app detection API. |
| src/vstest.console/CommandLine/AssemblyMetadataProvider.cs | Implements MTP-app detection using shared CoreUtilities detector. |
| src/Microsoft.TestPlatform.ObjectModel/Host/ISourceAwareTestRuntimeProvider.cs | Adds internal source-aware runtime-provider interface to avoid new public API. |
| src/Microsoft.TestPlatform.ObjectModel/Friends.cs | Grants Common.UnitTests access to ObjectModel internals for testing. |
| src/Microsoft.TestPlatform.Common/Hosting/TestRunTimeProviderManager.cs | Adds 2-pass provider selection: source-aware first, legacy fallback second. |
| src/Microsoft.TestPlatform.Common/PublicAPI/PublicAPI.Shipped.txt | Updates shipped API listing for the manager signature/param name. |
| src/Microsoft.TestPlatform.CoreUtilities/Helpers/MicrosoftTestingPlatformDetector.cs | Adds shared assembly-metadata detector for MTP marker. |
| src/Microsoft.TestPlatform.CrossPlatEngine/TestEngine.cs | Routes MTP sources to MTP proxy managers and prevents in-proc/no-isolation for MTP sources. |
| src/Microsoft.TestPlatform.CrossPlatEngine/Microsoft.TestPlatform.CrossPlatEngine.csproj | Adds a netcoreapp TFM for hosting NETCOREAPP-only MTP client code. |
| src/Microsoft.TestPlatform.CrossPlatEngine/Client/MTP/MtpConstants.cs | Defines MTP server-mode JSON-RPC constants. |
| src/Microsoft.TestPlatform.CrossPlatEngine/Client/MTP/MtpClientHelpers.cs | Adds shared helpers for timeouts, message parsing, log-level mapping. |
| src/Microsoft.TestPlatform.CrossPlatEngine/Client/MTP/MtpServerConnection.cs | Implements TCP loopback JSON-RPC transport and process lifecycle management. |
| src/Microsoft.TestPlatform.CrossPlatEngine/Client/MTP/MtpTestNodeConverter.cs | Converts MTP nodes into vstest TestCase/TestResult. |
| src/Microsoft.TestPlatform.CrossPlatEngine/Client/MTP/MtpProxyDiscoveryManager.cs | Implements MTP-backed discovery via MTP protocol. |
| src/Microsoft.TestPlatform.CrossPlatEngine/Client/MTP/MtpProxyExecutionManager.cs | Implements MTP-backed execution, including data collection integration. |
| eng/expected-dll-frameworks.json | Updates expected framework classification due to CrossPlatEngine now producing net8/net* outputs. |
|
There is nicer way to abstract the mtp stuff than bunch of IFs but did not want to go that far, can also be implemented for .net framework ofc. Main part of the impelentation is something that looks like a "MTP" client that we already had mentioned in few places, and where the issue was recently closed as having no real usecase so maybe this is one. microsoft/testfx#5667 |
Rewrite the MTP client (MtpServerConnection, MtpProxy* managers, MtpTestNodeConverter, MtpClientHelpers) to serialize/parse with Jsonite instead of System.Text.Json, and remove the #if NETCOREAPP guards so the client compiles on net462, netstandard2.0 and net8.0. Jsonite is already compiled into CommunicationUtilities on every TFM and made visible to CrossPlatEngine via InternalsVisibleTo, so no extra dependency is needed and there is no System.Text.Json binding-redirect fallout on the .NET Framework runner. Add MtpJson, a small set of Jsonite DOM accessors, to centralize number coercion (int/long/double/decimal) and object/array casting. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… ifs The MTP proof routed Microsoft.Testing.Platform sources by branching inside TestEngine on #if NETCOREAPP + IsMicrosoftTestingPlatformSource. That leaks protocol awareness into the engine and doesn't compile the MTP path on netfx. Replace it with a registered runtime provider and a small proxy-factory seam: - MtpTestRuntimeProvider (in the TestHostRuntimeProvider assembly) claims MTP sources via ISourceAwareTestRuntimeProvider and produces its own discovery/ execution proxy managers via a new IProxyManagerFactory. - TestEngine asks the resolved host manager `is IProxyManagerFactory` and uses it, so the engine no longer knows anything about MTP. The #if NETCOREAPP and IsMicrosoftTestingPlatformSource branches are gone. - MtpProxyManagerFactory (public, CrossPlatEngine) creates the internal MTP proxies so the out-of-assembly provider can build them without exposing the proxy classes themselves. - Group unique run configs by the source-aware provider type so MTP and classic sources split into separate hosts in a mixed run. Builds clean Debug + Release across all TFMs (net462/netstandard2.0/net8.0). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add three acceptance tests proving MTP apps run under vstest.console end-to-end through the packaged runners the harness uses: - RunMtpApplicationExecutesTestsOverMtpProtocol (2 pass/1 fail/1 skip) - RunMixedClassicAndMtpApplicationsInSingleRun (3/2/2) - RunMixedClassicAndMtpApplicationsWritesSingleTrx (3/2/2 + single TRX) Multi-target the MtpMSTestProject asset net8.0;net11.0 so the harness resolves net11.0 while manual E2E scripts keep net8.0. Register the asset in TestAssets.slnx so it builds in CI. All 6 matrix cells (3 tests x netfx+netcore console) pass locally through the real harness. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Coverage packaging — notes from the discussion with the owner of code coverageTalked this through, capturing where we landed so the "known limitation" above isn't just parked. Framing we agreed on: in this mode vstest owns everything above the testhost — datacollector, loggers, TRX. The MTP app is only the testhost. So vstest collects coverage with the The only real question is how the collector binaries reach the run once the project stops referencing That gives three options:
Leaning 1 for the proof (opt-in, no packaging changes) and 2 as the shipping answer for the zero-dep story, with the multi-version precedence handled in vstest. In this PR the collector is grafted into the runner |
The provider assembly registers MtpTestRuntimeProvider as a third TestExtensionTypes entry, so GetTypesToLoad returns three full names. The acceptance meta-test still listed two and failed 2 vs 3. Added the third expected name. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Proof-of-concept / RFC. Companion draft: microsoft/testfx#9546, you don't need both (see below).
vstest.consoleanddatacollectorcan now discover and run a Microsoft.Testing.Platform (MTP) app directly over the MTP protocol, instead of only launching a classic vstest testhost. An MTP app is its own host, so vstest detects it from the source ([assembly: AssemblyMetadata("Microsoft.Testing.Platform.Application", "true")]) and drives it with MTP-specific proxies rather than the vstest testhost JSON protocol. Classic vstest-based projects keep working unchanged, and the two can run in the same invocation.Why
Test frameworks can drop their vstest dependencies and still be run by vstest in a mixed run. That makes migrating off vstest incremental instead of all-or-nothing: port some tests to MTP, then all of them, then move the infra (reporting, CI) off vstest when it's convenient, instead of replacing test infra, reporting, and every test in one shot. And no dynamic code loading. The companion testfx draft (microsoft/testfx#9546) is the other half: MSTest shedding
Microsoft.NET.Test.SdksoMicrosoft.TestPlatform.ObjectModelis its only vstest-lineage dep. Neither PR needs the other to land. Together they show the future is reachable.What changed
MicrosoftTestingPlatformDetector(CoreUtilities) andAssemblyMetadataProvider.IsMicrosoftTestingPlatformApp(vstest.console) read the MTP assembly-metadata marker from the source.CrossPlatEngine/Client/MTP/adds a JSON-RPC client (MtpServerConnection), a node to vstest result converter, andMtpProxyDiscoveryManager/MtpProxyExecutionManagerthat implement the existingIProxyDiscoveryManager/IProxyExecutionManagerbut speak MTP. The wire is serialized withJsonite, notSystem.Text.Json, so the client runs on every TFM we ship includingnet462, with no binding-redirect fallout in hosts that run without them.MtpTestRuntimeProvider(assemblyMicrosoft.TestPlatform.TestHostRuntimeProvider) is a first-classITestRuntimeProviderthat claims MTP sources by their shape throughISourceAwareTestRuntimeProvider. It implements the newIProxyManagerFactory, soTestEngineasks the resolved provider for its discovery and execution managers instead of branching on the source. A two-passTestRuntimeProviderManagerresolves it so the "no runtime provider" guard no longer rejects MTP sources.TestEnginehas no#if NETCOREAPPorIsMicrosoftTestingPlatformSourcebranches anymore, MTP is named only in explanatory comments.TestEnginewraps code-coverage datacollection around the MTP execution path, the same way it does for the classic path.ExecutionPreferenceswitch, detection is purely source-based.What I tested
The capstone: one
vstest.consoleinvocation over a classic vstest-based project and an MTP-only project together, with--collect:"Code Coverage"and--logger:trx:MSTestProject1.dllMtpMSTestProject.dllMtpMSTestProject72.73% blocks,MSTestProject150%). Stable across 3 back-to-back runs.MtpUnderVstestTests): a pure-MTP run, a mixed classic+MTP run, and a mixed run that writes a single TRX. Green through the real acceptance harness across both console axes (netfxvstest.console.exeand netcoredotnet vstest), 6/6, in Debug and Release.Microsoft.TestPlatform.Common.UnitTests396 passing / 0 failing, clean Release build and pack, smoke suite green.Known limitation: coverage packaging
The vstest "Code Coverage" datacollector binaries ship in
Microsoft.CodeCoverageand normally reach vstest.console throughMicrosoft.NET.Test.Sdk(Microsoft.CodeCoverage.propssetsTraceDataCollectorDirectoryPath, which the SDK's VSTest task forwards asVSTestTraceDataCollectorDirectoryPath). A test project that has dropped its vstest deps won't restore those DLLs, so vstest-datacollector coverage on it needs another source for them, either the project referencesMicrosoft.CodeCoverage, or vstest bundles the collector in its runner. In this proof I grafted the collector into the runner'sExtensions/(the "vstest bundles it" option). I wrote up the options in a comment below, pinned for a design discussion with the owner of code coverage. MTP-native coverage (Microsoft.Testing.Extensions.CodeCoverage) is a separate, unaffected path.Note for reviewers
MTP routing is a first-class runtime provider now, not a set of inline branches.
TestEngineknows only theIProxyManagerFactoryabstraction: if the resolved provider implements it, the engine asks it for the discovery and execution manager, otherwise the standard testhost proxies wrap the provider as before. The#if NETCOREAPPandIsMicrosoftTestingPlatformSourcebranches called out in the earlier inline threads are gone, MTP is named only in explanatory comments.Co-authored with GitHub Copilot.