Add `SuccessfulTests` to ensure something runs in every non-Helix run (#22406)

* Remove extra `[SkipOnHelix]` attribute - only need the one compiled into Microsoft.AspNetCore.Testing - update the documentation to reflect this - nit: address Markdown warnings that VS Code showed * Add `SuccessfulTests` to ensure something runs in every non-Helix runs - #22241 - cleans up hundreds of warnings but leaves a couple for the Blazor tests assembly - see comments about xUnit runner command line in the new class
2020-06-02 12:19:28 -07:00 · 2020-06-02 12:19:28 -07:00 · 256045729a
parent bcd4fc0598
commit 256045729a
37 changed files with 60 additions and 108 deletions
--- a/docs/Helix.md
+++ b/docs/Helix.md
@ -1,5 +1,4 @@
-Helix testing in ASP.NET Core
-==============================
+# Helix testing in ASP.NET Core

 Helix is the distributed test platform that we use to run tests.  We build a helix payload that contains the publish directory of every test project that we want to test
 send a job with with this payload to a set of queues for the various combinations of OS that we want to test
@ -11,16 +10,17 @@ For more info about helix see: [SDK](https://github.com/dotnet/arcade/blob/maste

 To run Helix tests for one particular test project:

-```
+``` powershell
 .\eng\scripts\RunHelix.ps1 -Project path\mytestproject.csproj
 ```

 This will restore, and then publish all the test project including some bootstrapping scripts that will install the correct dotnet runtime/sdk before running the test assembly on the helix machine(s), and upload the job to helix.

-
 ## How do I look at the results of a helix run on Azure Pipelines?
+
 There's a link embedded in the build.cmd log of the helix target on Azure Pipelines, near the bottom right that will look something like this:
-```
+
+``` text
 2019-02-07T21:55:48.1516089Z   Results will be available from https://mc.dot.net/#/user/aspnetcore/pr~2Faspnet~2Faspnetcore/ci/20190207.34
 2019-02-07T21:56:43.2209607Z   Job 0dedeef6-210e-4815-89f9-fd07513179fe is completed with 108 finished work items.
 2019-02-07T21:56:43.5091018Z   Job 4c45a464-9464-4321-906c-2503320066b0 is completed with 108 finished work items.
@ -36,11 +36,13 @@ There's a link embedded in the build.cmd log of the helix target on Azure Pipeli

 The link will take you to an overview of all the tests with clickable links to the logs and each run broken down by queue.

-All of the helix runs for aspnetcore can be found here https://mc.dot.net/#/user/aspnetcore/builds
+All of the helix runs for aspnetcore can be found here <https://mc.dot.net/#/user/aspnetcore/builds>.

 ## What do I do if a test fails?
+
 You can simulate how most tests run locally:
-```
+
+``` powershell
 dotnet publish
 cd <the publish directory>
 dotnet vstest My.Tests.dll
@ -49,11 +51,12 @@ dotnet vstest My.Tests.dll
 If that doesn't help, you can try the Get Repro environment link from mission control and try to debug that way.

 ## Differences from running tests locally
+
 Most tests that don't just work on helix automatically are ones that depend on the source code being accessible. The helix payloads only contain whatever is in the publish directories, so any thing else that test depends on will need to be included to the payload.

 This can be accomplished by using the `HelixContent` property like so.

-```
+``` msbuild
 <ItemGroup>
  <HelixContent Include="$(RepoRoot)src\KeepMe.js"/>
  <HelixContent Include="$(RepoRoot)src\Project\**"/>
@ -62,7 +65,7 @@ This can be accomplished by using the `HelixContent` property like so.

 By default, these files will be included in the root directory. To include these files in a different directory, you can use either the `Link` or `LinkBase` attributes to set the included path.

-```
+``` msbuild
 <ItemGroup>
  <HelixContent Include="$(RepoRoot)src\KeepMe.js" Link="$(MSBuildThisFileDirectory)\myassets\KeepMe.js"/>
  <HelixContent Include="$(RepoRoot)src\Project\**" LinkBase="$(MSBuildThisFileDirectory)\myassets"/>
@ -70,8 +73,10 @@ By default, these files will be included in the root directory. To include these
 ```

 ## How to skip tests on helix
+
 There are two main ways to opt out of helix
+
 - Skipping the entire test project via `<BuildHelixPayload>false</BuildHelixPayload>` in csproj (the default value for this is IsTestProject).
- Skipping an individual test via `[SkipOnHelix("url to github issue")]` which might require including a compile reference to: `<Compile Include="$(SharedSourceRoot)test\SkipOnHelixAttribute.cs" />`
+- Skipping an individual test via `[SkipOnHelix("url to github issue")]`.

 Make sure to file an issue for any skipped tests and include that in a comment next to either of these
--- a/eng/helix/content/RunTests/TestRunner.cs
+++ b/eng/helix/content/RunTests/TestRunner.cs
@ -272,7 +272,7 @@ namespace RunTests

                    // Filter syntax: https://github.com/Microsoft/vstest-docs/blob/master/docs/filter.md
                    var result = await ProcessUtil.RunAsync($"{Options.DotnetRoot}/dotnet",
-                        commonTestArgs + " --TestCaseFilter:\"Quarantined!=true\"",
+                        commonTestArgs + " --TestCaseFilter:\"Quarantined!=true|Quarantined=false\"",
                        environmentVariables: EnvironmentVariables,
                        outputDataReceived: Console.WriteLine,
                        errorDataReceived: Console.Error.WriteLine,
--- a/eng/targets/CSharp.Common.props
+++ b/eng/targets/CSharp.Common.props
@ -42,6 +42,7 @@
    <Reference Include="Microsoft.AspNetCore.Testing" />
    <Reference Include="Moq" />
    <Reference Include="NETStandard.Library" />
+    <Compile Include="$(SharedSourceRoot)test\SuccessfulTests.cs" LinkBase="SharedTests" />
  </ItemGroup>

  <Import Project="$(RepoRoot)src\Testing\src\build\Microsoft.AspNetCore.Testing.props" Condition=" '$(IsTestProject)' == 'true' " />
--- a/src/Analyzers/Analyzers/test/Microsoft.AspNetCore.Analyzers.Test.csproj
+++ b/src/Analyzers/Analyzers/test/Microsoft.AspNetCore.Analyzers.Test.csproj
@ -7,7 +7,6 @@
  </PropertyGroup>

  <ItemGroup>
-    <Compile Include="$(SharedSourceRoot)test\SkipOnHelixAttribute.cs" />
    <Content Include="TestFiles\**\*.*" CopyToOutputDirectory="PreserveNewest" />
  </ItemGroup>

--- a/src/Components/Analyzers/test/Microsoft.AspNetCore.Components.Analyzers.Tests.csproj
+++ b/src/Components/Analyzers/test/Microsoft.AspNetCore.Components.Analyzers.Tests.csproj
@ -16,7 +16,6 @@
  </ItemGroup>

  <ItemGroup>
-    <Compile Include="$(SharedSourceRoot)test\SkipOnHelixAttribute.cs" />
    <Content Include="TestFiles\**\*.*" CopyToPublishDirectory="PreserveNewest" />
  </ItemGroup>
 </Project>
--- a/src/Components/Authorization/test/Microsoft.AspNetCore.Components.Authorization.Tests.csproj
+++ b/src/Components/Authorization/test/Microsoft.AspNetCore.Components.Authorization.Tests.csproj
@ -12,7 +12,6 @@

  <ItemGroup>
    <Compile Include="$(ComponentsSharedSourceRoot)test\**\*.cs" LinkBase="Helpers" />
-    <Compile Include="$(SharedSourceRoot)test\SkipOnHelixAttribute.cs" />
  </ItemGroup>

 </Project>
--- a/src/Components/Components/test/Microsoft.AspNetCore.Components.Tests.csproj
+++ b/src/Components/Components/test/Microsoft.AspNetCore.Components.Tests.csproj
@ -12,7 +12,6 @@

  <ItemGroup>
    <Compile Include="$(ComponentsSharedSourceRoot)test\**\*.cs" LinkBase="Helpers" />
-    <Compile Include="$(SharedSourceRoot)test\SkipOnHelixAttribute.cs" />
  </ItemGroup>

 </Project>
--- a/src/Components/Forms/test/Microsoft.AspNetCore.Components.Forms.Tests.csproj
+++ b/src/Components/Forms/test/Microsoft.AspNetCore.Components.Forms.Tests.csproj
@ -11,7 +11,6 @@

  <ItemGroup>
    <Compile Include="$(ComponentsSharedSourceRoot)test\**\*.cs" LinkBase="Helpers" />
-    <Compile Include="$(SharedSourceRoot)test\SkipOnHelixAttribute.cs" />
  </ItemGroup>

 </Project>
--- a/src/Components/Web/test/Microsoft.AspNetCore.Components.Web.Tests.csproj
+++ b/src/Components/Web/test/Microsoft.AspNetCore.Components.Web.Tests.csproj
@ -13,7 +13,6 @@

  <ItemGroup>
    <Compile Include="$(ComponentsSharedSourceRoot)test\**\*.cs" LinkBase="Helpers" />
-    <Compile Include="$(SharedSourceRoot)test\SkipOnHelixAttribute.cs" />
  </ItemGroup>

 </Project>
--- a/src/Components/WebAssembly/Build/test/Microsoft.AspNetCore.Components.WebAssembly.Build.Tests.csproj
+++ b/src/Components/WebAssembly/Build/test/Microsoft.AspNetCore.Components.WebAssembly.Build.Tests.csproj
@ -34,7 +34,6 @@

  <ItemGroup>
    <ProjectReference Include="..\..\testassets\StandaloneApp\StandaloneApp.csproj" />
-    <Compile Include="$(SharedSourceRoot)test\SkipOnHelixAttribute.cs" />
    <Compile Include="$(SharedSourceRoot)CommandLineUtils\**\*.cs" />
  </ItemGroup>

--- a/src/Components/WebAssembly/Build/test/RuntimeDependenciesResolverTest.cs
+++ b/src/Components/WebAssembly/Build/test/RuntimeDependenciesResolverTest.cs
@ -7,7 +7,7 @@ using System.IO;
 using System.Linq;
 using System.Reflection;
 using System.Text;
-using Microsoft.AspNetCore.Testing.xunit;
+using Microsoft.AspNetCore.Testing;
 using Xunit;

 namespace Microsoft.AspNetCore.Components.WebAssembly.Build
--- a/src/DataProtection/Extensions/test/Microsoft.AspNetCore.DataProtection.Extensions.Tests.csproj
+++ b/src/DataProtection/Extensions/test/Microsoft.AspNetCore.DataProtection.Extensions.Tests.csproj
@ -7,7 +7,6 @@
  <ItemGroup>
    <Compile Include="..\..\shared\test\*.cs" />
    <Content Include="TestFiles\**\*" CopyToOutputDirectory="PreserveNewest" />
-    <Compile Include="$(SharedSourceRoot)test\SkipOnHelixAttribute.cs" />
  </ItemGroup>

  <ItemGroup>
--- a/src/Hosting/Hosting/test/Microsoft.AspNetCore.Hosting.Tests.csproj
+++ b/src/Hosting/Hosting/test/Microsoft.AspNetCore.Hosting.Tests.csproj
@ -5,7 +5,6 @@
  </PropertyGroup>

  <ItemGroup>
-    <Compile Include="$(SharedSourceRoot)test\SkipOnHelixAttribute.cs" />
    <Compile Include="$(SharedSourceRoot)EventSource.Testing\TestEventListener.cs" />
    <Compile Include="$(SharedSourceRoot)EventSource.Testing\TestCounterListener.cs" />
    <Content Include="testroot\**\*" CopyToOutputDirectory="PreserveNewest" CopyToPublishDirectory="PreserveNewest" />
--- a/src/Http/Http/test/BindingAddressTests.cs
+++ b/src/Http/Http/test/BindingAddressTests.cs
@ -2,11 +2,9 @@
 // Licensed under the Apache License, Version 2.0. See License.txt in the project root for license information.

 using System;
-using System.Collections.Generic;
-using System.Text;
 using Microsoft.AspNetCore.Testing;
-using Microsoft.AspNetCore.Testing.xunit;
 using Xunit;
+
 namespace Microsoft.AspNetCore.Http.Tests
 {
    public class BindingAddressTests
--- a/src/Identity/ApiAuthorization.IdentityServer/test/Microsoft.AspNetCore.ApiAuthorization.IdentityServer.Tests.csproj
+++ b/src/Identity/ApiAuthorization.IdentityServer/test/Microsoft.AspNetCore.ApiAuthorization.IdentityServer.Tests.csproj
@ -12,7 +12,6 @@
    <FrameworkReference Remove="Microsoft.AspNetCore.App" />

    <Reference Include="Microsoft.AspNetCore.ApiAuthorization.IdentityServer" />
-    <Compile Include="$(SharedSourceRoot)test\SkipOnHelixAttribute.cs" />
  </ItemGroup>

  <ItemGroup>
--- a/src/Identity/test/Identity.Test/Microsoft.AspNetCore.Identity.Test.csproj
+++ b/src/Identity/test/Identity.Test/Microsoft.AspNetCore.Identity.Test.csproj
@ -6,7 +6,6 @@

  <ItemGroup>
    <Compile Include="$(IdentityTestSharedSourceRoot)**\*.cs" />
-    <Compile Include="$(SharedSourceRoot)test\SkipOnHelixAttribute.cs" />
  </ItemGroup>

  <ItemGroup>
--- a/src/Middleware/Diagnostics.EntityFrameworkCore/test/FunctionalTests/Diagnostics.EFCore.FunctionalTests.csproj
+++ b/src/Middleware/Diagnostics.EntityFrameworkCore/test/FunctionalTests/Diagnostics.EFCore.FunctionalTests.csproj
@ -15,8 +15,6 @@
    <!-- Avoid CS1705 errors due to mix of assemblies brought in transitively. -->
    <Reference Include="Microsoft.Extensions.DependencyInjection.Abstractions" />
    <Reference Include="Microsoft.Extensions.Logging.Abstractions" />
-    
-    <Compile Include="$(SharedSourceRoot)test\SkipOnHelixAttribute.cs" />
  </ItemGroup>

 </Project>
--- a/src/Middleware/NodeServices/test/Microsoft.AspNetCore.NodeServices.Tests.csproj
+++ b/src/Middleware/NodeServices/test/Microsoft.AspNetCore.NodeServices.Tests.csproj
@ -8,7 +8,6 @@
  <ItemGroup>
    <Reference Include="Microsoft.AspNetCore.NodeServices" />
    <Reference Include="Microsoft.AspNetCore.TestHost" />
-    <Compile Include="$(SharedSourceRoot)test\SkipOnHelixAttribute.cs" />
    <Content Include="js\**\*" />
  </ItemGroup>

--- a/src/Mvc/Mvc.Analyzers/test/Mvc.Analyzers.Test.csproj
+++ b/src/Mvc/Mvc.Analyzers/test/Mvc.Analyzers.Test.csproj
@ -8,7 +8,6 @@

  <ItemGroup>
    <None Include="xunit.runner.json" CopyToOutputDirectory="PreserveNewest" />
-    <Compile Include="$(SharedSourceRoot)test\SkipOnHelixAttribute.cs" />
    <Compile Remove="TestFiles\TagHelpersInCodeBlocksAnalyzerTest\*.*" />
    <Content Include="TestFiles\**\*.*" CopyToPublishDirectory="PreserveNewest" />
  </ItemGroup>
--- a/src/Mvc/Mvc.Api.Analyzers/test/Mvc.Api.Analyzers.Test.csproj
+++ b/src/Mvc/Mvc.Api.Analyzers/test/Mvc.Api.Analyzers.Test.csproj
@ -8,7 +8,6 @@

  <ItemGroup>
    <Compile Include="..\..\Mvc.Analyzers\test\Infrastructure\MvcDiagnosticAnalyzerRunner.cs" Link="Infrastructure\MvcDiagnosticAnalyzerRunner.cs" />
-    <Compile Include="$(SharedSourceRoot)test\SkipOnHelixAttribute.cs" />
    <Content Include="TestFiles\**\*.*" CopyToPublishDirectory="PreserveNewest" />
  </ItemGroup>

--- a/src/ProjectTemplates/test/ProjectTemplates.Tests.csproj
+++ b/src/ProjectTemplates/test/ProjectTemplates.Tests.csproj
@ -27,7 +27,6 @@
  <ItemGroup>
    <EmbeddedResource Include="template-baselines.json" />
    <Compile Include="$(SharedSourceRoot)Process\*.cs" LinkBase="shared\Process" />
-    <Compile Include="$(SharedSourceRoot)test\SkipOnHelixAttribute.cs" />
    <Compile Include="..\Shared\**" LinkBase="Helpers" />
  </ItemGroup>

--- a/src/Security/Authentication/Negotiate/test/Negotiate.FunctionalTest/Microsoft.AspNetCore.Authentication.Negotiate.FunctionalTest.csproj
+++ b/src/Security/Authentication/Negotiate/test/Negotiate.FunctionalTest/Microsoft.AspNetCore.Authentication.Negotiate.FunctionalTest.csproj
@ -5,10 +5,6 @@
    <IsWindowsOnlyTest>true</IsWindowsOnlyTest>
  </PropertyGroup>

-  <ItemGroup>
-    <Compile Include="$(SharedSourceRoot)test\SkipOnHelixAttribute.cs" />
-  </ItemGroup>
-
  <ItemGroup>
    <Reference Include="Microsoft.AspNetCore.Authentication.Negotiate" />
    <Reference Include="Microsoft.AspNetCore.Routing" />
--- a/src/Security/Authentication/Negotiate/test/Negotiate.Test/Microsoft.AspNetCore.Authentication.Negotiate.Test.csproj
+++ b/src/Security/Authentication/Negotiate/test/Negotiate.Test/Microsoft.AspNetCore.Authentication.Negotiate.Test.csproj
@ -4,10 +4,6 @@
    <TargetFrameworks>$(DefaultNetCoreTargetFramework)</TargetFrameworks>
  </PropertyGroup>

-  <ItemGroup>
-    <Compile Include="$(SharedSourceRoot)test\SkipOnHelixAttribute.cs" />
-  </ItemGroup>
-
  <ItemGroup>
    <Reference Include="Microsoft.AspNetCore.Authentication.Negotiate" />
    <Reference Include="Microsoft.AspNetCore.Diagnostics" />
--- a/src/Security/Authentication/test/Microsoft.AspNetCore.Authentication.Test.csproj
+++ b/src/Security/Authentication/test/Microsoft.AspNetCore.Authentication.Test.csproj
@ -33,10 +33,6 @@
    <EmbeddedResource Include="OpenIdConnect\wellknownkeys.json" />
  </ItemGroup>

-  <ItemGroup>
-    <Compile Include="$(SharedSourceRoot)test\SkipOnHelixAttribute.cs" />
-  </ItemGroup>
-
  <ItemGroup>
    <Reference Include="Microsoft.AspNetCore.Authentication.Certificate" />
    <Reference Include="Microsoft.AspNetCore.Authentication.Cookies" />
--- a/src/Servers/IIS/IIS/test/FunctionalTest.props
+++ b/src/Servers/IIS/IIS/test/FunctionalTest.props
@ -5,8 +5,6 @@
  </PropertyGroup>

  <ItemGroup>
-    <Compile Include="$(SharedSourceRoot)test\SkipOnHelixAttribute.cs" />
-
    <Content Include="..\Common.FunctionalTests\AppHostConfig\*.config" CopyToOutputDirectory="PreserveNewest" />
  </ItemGroup>

--- a/src/Servers/Kestrel/Kestrel/test/Microsoft.AspNetCore.Server.Kestrel.Tests.csproj
+++ b/src/Servers/Kestrel/Kestrel/test/Microsoft.AspNetCore.Server.Kestrel.Tests.csproj
@ -7,7 +7,6 @@
  <ItemGroup>
    <Compile Include="$(SharedSourceRoot)NullScope.cs" />
    <Compile Include="$(KestrelSharedSourceRoot)test\*.cs" LinkBase="shared" />
-    <Compile Include="$(SharedSourceRoot)test\SkipOnHelixAttribute.cs" />
    <Content Include="$(KestrelSharedSourceRoot)test\TestCertificates\*.pfx" LinkBase="shared\TestCertificates" CopyToOutputDirectory="PreserveNewest" />
    <Content Include="$(KestrelRoot)Core\src\Internal\Http\HttpHeaders.Generated.cs" LinkBase="shared\GeneratedContent" CopyToOutputDirectory="PreserveNewest" />
    <Content Include="$(KestrelRoot)Core\src\Internal\Http\HttpProtocol.Generated.cs" LinkBase="shared\GeneratedContent" CopyToOutputDirectory="PreserveNewest" />
--- a/src/Servers/Kestrel/test/InMemory.FunctionalTests/InMemory.FunctionalTests.csproj
+++ b/src/Servers/Kestrel/test/InMemory.FunctionalTests/InMemory.FunctionalTests.csproj
@ -10,7 +10,6 @@
  <ItemGroup>
    <Compile Include="$(SharedSourceRoot)NullScope.cs" />
    <Compile Include="$(KestrelSharedSourceRoot)test\*.cs" LinkBase="shared" />
-    <Compile Include="$(SharedSourceRoot)test\SkipOnHelixAttribute.cs" />
    <Content Include="$(KestrelSharedSourceRoot)test\TestCertificates\*.pfx" LinkBase="shared\TestCertificates" CopyToOutputDirectory="PreserveNewest" />
    <Compile Include="$(RepoRoot)src\Shared\Buffers.MemoryPool\*.cs" LinkBase="MemoryPool" />
    <Compile Include="$(KestrelSharedSourceRoot)\CorrelationIdGenerator.cs" Link="Internal\CorrelationIdGenerator.cs" />
@ -27,4 +26,4 @@
    <Reference Include="Newtonsoft.Json" />
  </ItemGroup>

-</Project>
+</Project>
--- a/src/Servers/Kestrel/test/Interop.FunctionalTests/Interop.FunctionalTests.csproj
+++ b/src/Servers/Kestrel/test/Interop.FunctionalTests/Interop.FunctionalTests.csproj
@ -17,7 +17,6 @@
    <Compile Include="$(KestrelSharedSourceRoot)test\TestConstants.cs" LinkBase="shared" />
    <Compile Include="$(KestrelSharedSourceRoot)test\TestResources.cs" LinkBase="shared" />
    <Content Include="$(KestrelSharedSourceRoot)test\TestCertificates\*.pfx" LinkBase="shared\TestCertificates" CopyToOutputDirectory="PreserveNewest" />
-    <Compile Include="$(SharedSourceRoot)test\SkipOnHelixAttribute.cs" />
  </ItemGroup>

  <ItemGroup>
--- a/src/Servers/Kestrel/test/Libuv.BindTests/Libuv.BindTests.csproj
+++ b/src/Servers/Kestrel/test/Libuv.BindTests/Libuv.BindTests.csproj
@ -5,7 +5,7 @@
    <ServerGarbageCollection>true</ServerGarbageCollection>
    <TestGroupName>Libuv.BindTests</TestGroupName>
    <!-- https://github.com/dotnet/aspnetcore/issues/22114 -->
-    <SkipHelixQueues>Windows.10.Arm64;Windows.10.Arm64.Open</SkipHelixQueues>   
+    <SkipHelixQueues>Windows.10.Arm64;Windows.10.Arm64.Open</SkipHelixQueues>
  </PropertyGroup>

  <ItemGroup>
@ -14,7 +14,6 @@
  </ItemGroup>

  <ItemGroup>
-    <Compile Include="$(SharedSourceRoot)test\SkipOnHelixAttribute.cs" />
    <Compile Include="..\BindTests\**\*.cs" />
    <Compile Include="..\Libuv.FunctionalTests\TransportSelector.cs" />
    <Compile Include="$(SharedSourceRoot)NullScope.cs" />
--- a/src/Servers/Kestrel/test/Libuv.FunctionalTests/Libuv.FunctionalTests.csproj
+++ b/src/Servers/Kestrel/test/Libuv.FunctionalTests/Libuv.FunctionalTests.csproj
@ -14,7 +14,6 @@
  <ItemGroup>
    <Compile Include="..\FunctionalTests\**\*.cs" />
    <Compile Include="$(SharedSourceRoot)NullScope.cs" />
-    <Compile Include="$(SharedSourceRoot)test\SkipOnHelixAttribute.cs" />
    <Compile Include="$(KestrelSharedSourceRoot)test\*.cs" LinkBase="shared" />
    <Compile Include="$(KestrelSharedSourceRoot)test\TransportTestHelpers\*.cs" LinkBase="shared\TransportTestHelpers" />
    <Content Include="$(KestrelSharedSourceRoot)test\TestCertificates\*.pfx" LinkBase="shared\TestCertificates" CopyToOutputDirectory="PreserveNewest" />
--- a/src/Servers/Kestrel/test/Sockets.BindTests/Sockets.BindTests.csproj
+++ b/src/Servers/Kestrel/test/Sockets.BindTests/Sockets.BindTests.csproj
@ -7,7 +7,6 @@
  </PropertyGroup>

  <ItemGroup>
-    <Compile Include="$(SharedSourceRoot)test\SkipOnHelixAttribute.cs" />
    <Compile Include="..\BindTests\**\*.cs" />
    <Compile Include="..\Sockets.FunctionalTests\TransportSelector.cs" />
    <Compile Include="$(SharedSourceRoot)NullScope.cs" />
--- a/src/Servers/Kestrel/test/Sockets.FunctionalTests/Sockets.FunctionalTests.csproj
+++ b/src/Servers/Kestrel/test/Sockets.FunctionalTests/Sockets.FunctionalTests.csproj
@ -10,7 +10,6 @@
  <ItemGroup>
    <Compile Include="..\FunctionalTests\**\*.cs" />
    <Compile Include="$(SharedSourceRoot)NullScope.cs" />
-    <Compile Include="$(SharedSourceRoot)test\SkipOnHelixAttribute.cs" />
    <Compile Include="$(KestrelSharedSourceRoot)test\*.cs" LinkBase="shared" />
    <Compile Include="$(KestrelSharedSourceRoot)test\TransportTestHelpers\*.cs" LinkBase="shared\TransportTestHelpers" />
    <Content Include="$(KestrelSharedSourceRoot)test\TestCertificates\*.pfx" LinkBase="shared\TestCertificates" CopyToOutputDirectory="PreserveNewest" />
--- a/src/Shared/test/SkipOnHelixAttribute.cs
+++ b/src/Shared/test/SkipOnHelixAttribute.cs
@ -1,50 +0,0 @@
-// Copyright (c) .NET Foundation. All rights reserved.
-// Licensed under the Apache License, Version 2.0. See License.txt in the project root for license information.
-
-using System;
-using System.Linq;
-
-namespace Microsoft.AspNetCore.Testing.xunit
-{
-    /// <summary>
-    /// Skip test if running on helix (or a particular helix queue).
-    /// </summary>
-    [AttributeUsage(AttributeTargets.Method | AttributeTargets.Class, AllowMultiple = false)]
-    public class SkipOnHelixAttribute : Attribute, ITestCondition
-    {
-        public SkipOnHelixAttribute(string issueUrl)
-        {
-            if (string.IsNullOrEmpty(issueUrl))
-            {
-                throw new ArgumentException();
-            }
-            IssueUrl = issueUrl;
-        }
-
-        public string IssueUrl { get; }
-
-        public bool IsMet
-        {
-            get
-            {
-                var skip = OnHelix() && (Queues == null || Queues.ToLowerInvariant().Split(";").Contains(GetTargetHelixQueue().ToLowerInvariant()));
-                return !skip;
-            }
-        }
-
-        // Queues that should be skipped on, i.e. "Windows.10.Amd64.ClientRS4.VS2017.Open;OSX.1012.Amd64.Open"
-        public string Queues { get; set; }
-
-        public string SkipReason
-        {
-            get
-            {
-                return $"This test is skipped on helix";
-            }
-        }
-
-        public static bool OnHelix() => !string.IsNullOrEmpty(GetTargetHelixQueue());
-
-        public static string GetTargetHelixQueue() => Environment.GetEnvironmentVariable("helix");
-    }
-}
--- a/src/Shared/test/SuccessfulTests.cs
+++ b/src/Shared/test/SuccessfulTests.cs
@ -0,0 +1,39 @@
+// Copyright (c) .NET Foundation. All rights reserved.
+// Licensed under the Apache License, Version 2.0. See License.txt in the project root for license information.
+
+using Microsoft.AspNetCore.Testing;
+using Xunit;
+
+namespace AlwaysTestTests
+{
+    /// <summary>
+    /// Tests for every test assembly to ensure quarantined and unquarantined runs report at least one test execution.
+    /// </summary>
+    public class SuccessfulTests
+    {
+        /// <summary>
+        /// Test that executes in quarantined runs and always succeeds.
+        /// </summary>
+        [Fact]
+        [QuarantinedTest]
+        public void GuaranteedQuarantinedTest()
+        {
+        }
+
+        /// <summary>
+        /// Test that executes in unquarantined runs and always succeeds.
+        /// </summary>
+        /// <remarks>
+        /// <see cref="TraitAttribute"/> applied to ensure test runs even if assembly is quarantined overall.
+        /// <c>dotnet test --filter</c>, <c>dotnet vstest --testcasefilter</c>, and
+        /// <c>vstest.console.exe --testcasefilter</c> handle the "no Quarantined=true trait OR Quarantined=false" case
+        /// e.g. <c>"Quarantined!=true|Quarantined=false</c>. But, <c>xunit.console.exe</c> doesn't have a syntax to
+        /// make it work (yet?).
+        /// </remarks>
+        [Fact]
+        [Trait("Quarantined", "false")]
+        public void GuaranteedUnquarantinedTest()
+        {
+        }
+    }
+}
--- a/src/Tools/FirstRunCertGenerator/test/Microsoft.AspNetCore.DeveloperCertificates.XPlat.Tests.csproj
+++ b/src/Tools/FirstRunCertGenerator/test/Microsoft.AspNetCore.DeveloperCertificates.XPlat.Tests.csproj
@ -6,7 +6,6 @@

  <ItemGroup>
  	<Reference Include="Microsoft.AspNetCore.DeveloperCertificates.XPlat" />
-    <Compile Include="$(SharedSourceRoot)test\SkipOnHelixAttribute.cs" />
  </ItemGroup>

 </Project>
--- a/src/Tools/Microsoft.dotnet-openapi/test/dotnet-microsoft.openapi.Tests.csproj
+++ b/src/Tools/Microsoft.dotnet-openapi/test/dotnet-microsoft.openapi.Tests.csproj
@ -12,7 +12,6 @@

  <ItemGroup>
    <Compile Include="$(ToolSharedSourceRoot)TestHelpers\**\*.cs" />
-    <Compile Include="$(SharedSourceRoot)test\SkipOnHelixAttribute.cs" />
    <Content Include="TestContent\*" LinkBase="TestContent\" CopyToOutputDirectory="PreserveNewest" />
    <Compile Include="$(SharedSourceRoot)Process\ProcessExtensions.cs" />
  </ItemGroup>
--- a/src/Tools/dotnet-watch/test/dotnet-watch.Tests.csproj
+++ b/src/Tools/dotnet-watch/test/dotnet-watch.Tests.csproj
@ -9,7 +9,6 @@

  <ItemGroup>
    <Compile Include="$(ToolSharedSourceRoot)TestHelpers\**\*.cs" />
-    <Compile Include="$(SharedSourceRoot)test\SkipOnHelixAttribute.cs" />
    <Content Include="TestProjects\**\*" CopyToOutputDirectory="PreserveNewest" />
  </ItemGroup>