diff --git a/docs/Helix.md b/docs/Helix.md index 256908b8ed..762f89dc2d 100644 --- a/docs/Helix.md +++ b/docs/Helix.md @@ -2,12 +2,30 @@ Helix testing in ASP.NET Core ============================== Helix is the distributed test platform that we use to run tests. We build a helix payload that contains the publish directory of every test project that we want to test -send a job with with this payload to a set of queues for the various combinations of OS that we want to test +send a job with with this payload to a set of queues for the various combinations of OS that we want to test for example: `Windows.10.Amd64.ClientRS4.VS2017.Open`, `OSX.1012.Amd64.Open`, `Ubuntu.1810.Amd64.Open`. Helix takes care of unzipping, running the job, and reporting results. -For more info about helix see: [SDK](https://github.com/dotnet/arcade/blob/master/src/Microsoft.DotNet.Helix/Sdk/Readme.md), [JobSender](https://github.com/dotnet/arcade/blob/master/src/Microsoft.DotNet.Helix/Sdk/Readme.md) +For more info about helix see: [SDK](https://github.com/dotnet/arcade/blob/master/src/Microsoft.DotNet.Helix/Sdk/Readme.md), [JobSender](https://github.com/dotnet/arcade/blob/master/src/Microsoft.DotNet.Helix/Sdk/Readme.md) -## How do I look at the results of a helix run? +## Running helix tests locally + +To run Helix tests for one particular test project: + +``` +cd src/MyCode/test +dotnet build /t:Helix +``` + +To run tests for the entire repo, run: + +``` +.\build.cmd /t:Helix +``` + +This will restore, and then publish all of the test projects including some bootstrapping scripts that will install the correct dotnet runtime/sdk before running the test assemblies on the helix machine, and upload the job to helix, it won't wait for the jobs to complete, but you can go to https://mc.dot.net/#/user/$(your user name)/builds. + + +## How do I look at the results of a helix run on Azure Pipelines? There's a link embedded in the build.cmd log of the helix target on Azure Pipelines, near the bottom right that will look something like this: ``` 2019-02-07T21:55:48.1516089Z Results will be available from https://mc.dot.net/#/user/aspnetcore/pr~2Faspnet~2Faspnetcore/ci/20190207.34 @@ -23,7 +41,7 @@ There's a link embedded in the build.cmd log of the helix target on Azure Pipeli 2019-02-07T22:06:33.6898567Z Job 82f27d4c-9099-4f0e-b383-870c24d8dc2c is completed with 108 finished work items. ``` -The link will take you to an overview of all the tests with clickable links to the logs and each run broken down by queue. +The link will take you to an overview of all the tests with clickable links to the logs and each run broken down by queue. All of the helix runs for aspnetcore can be found here https://mc.dot.net/#/user/aspnetcore/builds @@ -40,11 +58,6 @@ If that doesn't help, you can try the Get Repro environment link from mission co ## Differences from running tests locally Most tests that don't just work on helix automatically are ones that depend on the source code being accessible. The helix payloads only contain whatever is in the publish directories, so any thing else that test depends on will need to be included to the payload (TBD how to do this). -## Running helix tests locally -`.\build.cmd /t:Helix /p:IsHelixJob=true` - -This will restore, and then publish all of the test projects including some bootstrapping scripts that will install the correct dotnet runtime/sdk before running the test assemblies on the helix machine, and upload the job to helix, it won't wait for the jobs to complete, but you can go to https://mc.dot.net/#/user/aspnetcore/builds and look for a source that matches private-yourusername - ## How to skip tests on helix There are two main ways to opt out of helix - Skipping the entire test project via `false` in csproj (the default value for this is IsTestProject). diff --git a/docs/ProjectProperties.md b/docs/ProjectProperties.md index a05fa69bde..1a5a1c4eb4 100644 --- a/docs/ProjectProperties.md +++ b/docs/ProjectProperties.md @@ -7,3 +7,4 @@ Property name | Meaning -------------------|-------------------------------------------------------------------------------------------- IsShippingPackage | When set to `true`, the package produced by from project is intended for use by customers. Defaults to `false`, which means the package is intended for internal use only by Microsoft teams. IsAspNetCoreApp | Set to `true` when the assembly is part of the [Microsoft.AspNetCore.App shared framework](./SharedFramework.md) and is not available as a NuGet package (unless IsShippingPackage is also set to `true`). +TestDependsOnMssql | Set to `true` when your tests depends on SQL Server. This will ensure distribute tests on Helix install LocalDB ([more information on Helix](./Helix.md)). diff --git a/eng/helix/content/README.md b/eng/helix/content/README.md new file mode 100644 index 0000000000..9afae21e76 --- /dev/null +++ b/eng/helix/content/README.md @@ -0,0 +1,3 @@ +# Helix content + +The content of this folder is included in the test payload for each Helix work item. The code here is mean to be used alongside test binaries. diff --git a/eng/scripts/RunPowershell.cmd b/eng/helix/content/RunPowershell.cmd similarity index 93% rename from eng/scripts/RunPowershell.cmd rename to eng/helix/content/RunPowershell.cmd index d80b1776d0..f76939d603 100644 --- a/eng/scripts/RunPowershell.cmd +++ b/eng/helix/content/RunPowershell.cmd @@ -7,5 +7,5 @@ SET POWERSHELL=%windir%\System32\WindowsPowerShell\v1.0\powershell.exe rem Force 64bit powershell if /i "%PROCESSOR_ARCHITEW6432%" EQU "AMD64" SET POWERSHELL=%windir%\sysnative\WindowsPowerShell\v1.0\powershell.exe - +echo "PS: Running '%~dp0%1' %_TAIL%" %POWERSHELL% -NoProfile -NoLogo -ExecutionPolicy unrestricted -Command "[System.Threading.Thread]::CurrentThread.CurrentCulture = ''; [System.Threading.Thread]::CurrentThread.CurrentUICulture = ''; try { & '%~dp0%1' %_TAIL%; exit $LASTEXITCODE } catch { write-host $_; exit 1 }" diff --git a/eng/helix/content/mssql/InstallSqlServerLocalDB.ps1 b/eng/helix/content/mssql/InstallSqlServerLocalDB.ps1 new file mode 100644 index 0000000000..f4c5995f70 --- /dev/null +++ b/eng/helix/content/mssql/InstallSqlServerLocalDB.ps1 @@ -0,0 +1,34 @@ +<# +.SYNOPSIS + Installs SQL Server 2016 Express LocalDB on a machine. +.DESCRIPTION + This script installs Microsoft SQL Server 2016 Express LocalDB on a machine. +.LINK + https://docs.microsoft.com/en-us/sql/database-engine/configure-windows/sql-server-2016-express-localdb?view=sql-server-2016 + https://docs.microsoft.com/en-us/sql/database-engine/install-windows/install-sql-server-from-the-command-prompt?view=sql-server-2016 +#> + +$ErrorActionPreference = 'Stop' +$ProgressPreference = 'SilentlyContinue' # Workaround PowerShell/PowerShell#2138 +Set-StrictMode -Version 1 + +$intermedateDir = "$PSScriptRoot\obj" +mkdir $intermedateDir -ErrorAction Ignore | Out-Null + +Write-Host "Installing SQL Server 2016 Express LocalDB" -f Magenta + +# Download SqlLocalDB.msi. +$installerFilename = "SqlLocalDB.msi" +$installerPath = "$intermedateDir\$installerFilename" +Write-Host "" +Write-Host "Downloading '$installerFilename' to '$installerPath'." +Invoke-WebRequest -OutFile $installerPath -UseBasicParsing ` + -Uri 'https://download.microsoft.com/download/9/0/7/907AD35F-9F9C-43A5-9789-52470555DB90/ENU/SqlLocalDB.msi' + +# Install LocalDB. +$arguments = '/package', "`"$installerPath`"", '/NoRestart', '/Passive', ` + 'IACCEPTSQLLOCALDBLICENSETERMS=YES', 'HIDEPROGRESSBAR=YES' +Write-Host "" +Write-Host "Running 'msiexec $arguments'." +$process = Start-Process msiexec.exe -ArgumentList $arguments -NoNewWindow -PassThru -Verbose -Wait +exit $process.ExitCode diff --git a/eng/helix/vstest/runtests.cmd b/eng/helix/content/runtests.cmd similarity index 89% rename from eng/helix/vstest/runtests.cmd rename to eng/helix/content/runtests.cmd index 647e482027..8a8a158739 100644 --- a/eng/helix/vstest/runtests.cmd +++ b/eng/helix/content/runtests.cmd @@ -1,10 +1,12 @@ +@echo off REM Disable "!Foo!" expansions because they break the filter syntax setlocal disableextensions set target=%1 -set sdkVersion=%2 -set runtimeVersion=%3 -set helixQueue=%4 +set targetFrameworkIdentifier=%2 +set sdkVersion=%3 +set runtimeVersion=%4 +set helixQueue=%5 set DOTNET_HOME=%HELIX_CORRELATION_PAYLOAD%\sdk set DOTNET_ROOT=%DOTNET_HOME%\x64 @@ -19,13 +21,18 @@ powershell.exe -NoProfile -ExecutionPolicy unrestricted -Command "[Net.ServicePo set HELIX=%helixQueue% +if (%targetFrameworkIdentifier%==.NETFramework) ( + xunit.console.exe %target% -xml testResults.xml + exit /b %ERRORLEVEL% +) + %DOTNET_ROOT%\dotnet vstest %target% -lt >discovered.txt find /c "Exception thrown" discovered.txt REM "ERRORLEVEL is not %ERRORLEVEL%" https://blogs.msdn.microsoft.com/oldnewthing/20080926-00/?p=20743/ if not errorlevel 1 ( echo Exception thrown during test discovery. 1>&2 type discovered.txt 1>&2 - exit 1 + exit /b 1 ) set exit_code=0 @@ -51,5 +58,5 @@ if errorlevel 1 ( REM DO NOT EXIT and DO NOT SET EXIT_CODE to 1 ) -exit %exit_code% +exit /b %exit_code% diff --git a/eng/helix/vstest/runtests.sh b/eng/helix/content/runtests.sh similarity index 100% rename from eng/helix/vstest/runtests.sh rename to eng/helix/content/runtests.sh diff --git a/eng/helix/helix.proj b/eng/helix/helix.proj index 2173d06a65..9274e5a6df 100644 --- a/eng/helix/helix.proj +++ b/eng/helix/helix.proj @@ -1,7 +1,7 @@ - - + + @@ -9,17 +9,27 @@ pr/aspnet/aspnetcore - ci private-$(USERNAME) private-$(USER) - $(BUILD_BUILDNUMBER) - true - true - true true - aspnetcore true - 4 + 2 + + + + ci + aspnetcore + $(BUILD_BUILDNUMBER) + true + true + true + + + + dev + $(USERNAME) + $(USER) + $([System.DateTime]::Now.ToString('yyyyMMdd HH:mm')) diff --git a/eng/helix/xunit/runtests.cmd b/eng/helix/xunit/runtests.cmd deleted file mode 100644 index 2c58f15196..0000000000 --- a/eng/helix/xunit/runtests.cmd +++ /dev/null @@ -1,3 +0,0 @@ -set target=%1 -set helix=%4 -xunit.console.exe %target% -xml testResults.xml diff --git a/eng/targets/Helix.Common.props b/eng/targets/Helix.Common.props index 09fdfee74f..f1c5ccb26d 100644 --- a/eng/targets/Helix.Common.props +++ b/eng/targets/Helix.Common.props @@ -12,6 +12,8 @@ + + diff --git a/eng/targets/Helix.props b/eng/targets/Helix.props index c4723b3a34..c33a6d4b54 100644 --- a/eng/targets/Helix.props +++ b/eng/targets/Helix.props @@ -1,38 +1,38 @@ - + - - - Never - Always - - + + + Never + Always + + - - true - 00:30:00 - $(HelixTargetQueue.Contains('Windows')) - $(MSBuildProjectName)-$(TargetFramework) - false - true - + + true + 00:30:00 + false + true + $(MSBuildProjectName)/$(TargetFramework) + false + true + - - + + + - - + + + - - + + + - - - - - - - + + + diff --git a/eng/targets/Helix.targets b/eng/targets/Helix.targets index 2768e42a85..bc94dc345e 100644 --- a/eng/targets/Helix.targets +++ b/eng/targets/Helix.targets @@ -2,13 +2,25 @@ - + + + + + + @@ -44,10 +56,8 @@ - - @@ -64,7 +74,7 @@ $(TargetFileName) @(HelixPreCommand) @(HelixPostCommand) - call runtests.cmd $(TargetFileName) $(NETCoreSdkVersion) $(MicrosoftNETCoreAppPackageVersion) $(HelixTargetQueue) + call runtests.cmd $(TargetFileName) $(TargetFrameworkIdentifier) $(NETCoreSdkVersion) $(MicrosoftNETCoreAppPackageVersion) $(HelixTargetQueue) ./runtests.sh $(TargetFileName) $(NETCoreSdkVersion) $(MicrosoftNETCoreAppPackageVersion) $(HelixTargetQueue) $(HelixTimeout) diff --git a/src/Identity/EntityFrameworkCore/test/EF.Test/Microsoft.AspNetCore.Identity.EntityFrameworkCore.Test.csproj b/src/Identity/EntityFrameworkCore/test/EF.Test/Microsoft.AspNetCore.Identity.EntityFrameworkCore.Test.csproj index d024b3dffe..229157c810 100644 --- a/src/Identity/EntityFrameworkCore/test/EF.Test/Microsoft.AspNetCore.Identity.EntityFrameworkCore.Test.csproj +++ b/src/Identity/EntityFrameworkCore/test/EF.Test/Microsoft.AspNetCore.Identity.EntityFrameworkCore.Test.csproj @@ -2,6 +2,7 @@ netcoreapp3.0 + true true diff --git a/src/Middleware/Diagnostics.EntityFrameworkCore/test/FunctionalTests/Diagnostics.EFCore.FunctionalTests.csproj b/src/Middleware/Diagnostics.EntityFrameworkCore/test/FunctionalTests/Diagnostics.EFCore.FunctionalTests.csproj index 732ae020e9..911a6b5d11 100644 --- a/src/Middleware/Diagnostics.EntityFrameworkCore/test/FunctionalTests/Diagnostics.EFCore.FunctionalTests.csproj +++ b/src/Middleware/Diagnostics.EntityFrameworkCore/test/FunctionalTests/Diagnostics.EFCore.FunctionalTests.csproj @@ -4,6 +4,7 @@ netcoreapp3.0 Diagnostics.EFCore.FunctionalTests + true diff --git a/src/Middleware/Diagnostics/test/FunctionalTests/Diagnostics.FunctionalTests.csproj b/src/Middleware/Diagnostics/test/FunctionalTests/Diagnostics.FunctionalTests.csproj index a76552bbf8..4c8951746c 100644 --- a/src/Middleware/Diagnostics/test/FunctionalTests/Diagnostics.FunctionalTests.csproj +++ b/src/Middleware/Diagnostics/test/FunctionalTests/Diagnostics.FunctionalTests.csproj @@ -4,6 +4,7 @@ netcoreapp3.0 false Diagnostics.FunctionalTests + true diff --git a/src/Servers/HttpSys/test/Directory.Build.props b/src/Servers/HttpSys/test/Directory.Build.props new file mode 100644 index 0000000000..bb2a02a75b --- /dev/null +++ b/src/Servers/HttpSys/test/Directory.Build.props @@ -0,0 +1,9 @@ + + + + + + + + + diff --git a/src/Servers/HttpSys/test/FunctionalTests/Microsoft.AspNetCore.Server.HttpSys.FunctionalTests.csproj b/src/Servers/HttpSys/test/FunctionalTests/Microsoft.AspNetCore.Server.HttpSys.FunctionalTests.csproj index 0186303746..77499f443a 100644 --- a/src/Servers/HttpSys/test/FunctionalTests/Microsoft.AspNetCore.Server.HttpSys.FunctionalTests.csproj +++ b/src/Servers/HttpSys/test/FunctionalTests/Microsoft.AspNetCore.Server.HttpSys.FunctionalTests.csproj @@ -8,7 +8,6 @@ - @@ -16,4 +15,10 @@ 214124cd-d05b-4309-9af9-9caa44b2b74a + + + + + + diff --git a/src/Servers/IIS/IIS/test/Common.FunctionalTests/ClientCertificateTests.cs b/src/Servers/IIS/IIS/test/Common.FunctionalTests/ClientCertificateTests.cs index 7ba740f0c5..8cdd0f102c 100644 --- a/src/Servers/IIS/IIS/test/Common.FunctionalTests/ClientCertificateTests.cs +++ b/src/Servers/IIS/IIS/test/Common.FunctionalTests/ClientCertificateTests.cs @@ -35,6 +35,7 @@ namespace Microsoft.AspNetCore.Server.IIS.FunctionalTests [ConditionalTheory] [MemberData(nameof(TestVariants))] + [OSSkipCondition(OperatingSystems.Windows, WindowsVersions.Win7, WindowsVersions.Win2008R2)] public Task HttpsNoClientCert_NoClientCert(TestVariant variant) { return ClientCertTest(variant, sendClientCert: false); @@ -42,6 +43,7 @@ namespace Microsoft.AspNetCore.Server.IIS.FunctionalTests [ConditionalTheory] [MemberData(nameof(TestVariants))] + [OSSkipCondition(OperatingSystems.Windows, WindowsVersions.Win7, WindowsVersions.Win2008R2)] public Task HttpsClientCert_GetCertInformation(TestVariant variant) { return ClientCertTest(variant, sendClientCert: true); diff --git a/src/Servers/IIS/IIS/test/Common.FunctionalTests/HttpsTests.cs b/src/Servers/IIS/IIS/test/Common.FunctionalTests/HttpsTests.cs index b2053b5cb1..7a5158f9bf 100644 --- a/src/Servers/IIS/IIS/test/Common.FunctionalTests/HttpsTests.cs +++ b/src/Servers/IIS/IIS/test/Common.FunctionalTests/HttpsTests.cs @@ -31,6 +31,7 @@ namespace Microsoft.AspNetCore.Server.IISIntegration.FunctionalTests [ConditionalTheory] [MemberData(nameof(TestVariants))] + [OSSkipCondition(OperatingSystems.Windows, WindowsVersions.Win7, WindowsVersions.Win2008R2)] public async Task HttpsHelloWorld(TestVariant variant) { var port = TestPortHelper.GetNextSSLPort(); diff --git a/src/Servers/IIS/IIS/test/Common.FunctionalTests/Inprocess/StartupTests.cs b/src/Servers/IIS/IIS/test/Common.FunctionalTests/Inprocess/StartupTests.cs index 5f2a0f70aa..d55fc4bf4b 100644 --- a/src/Servers/IIS/IIS/test/Common.FunctionalTests/Inprocess/StartupTests.cs +++ b/src/Servers/IIS/IIS/test/Common.FunctionalTests/Inprocess/StartupTests.cs @@ -113,6 +113,7 @@ namespace Microsoft.AspNetCore.Server.IISIntegration.FunctionalTests [SkipIfNotAdmin] [RequiresNewShim] [RequiresIIS(IISCapability.PoolEnvironmentVariables)] + [SkipOnHelix] [Flaky("https://github.com/aspnet/AspNetCore-Internal/issues/2221", FlakyOn.Helix.All)] public async Task StartsWithDotnetInstallLocation(RuntimeArchitecture runtimeArchitecture) { diff --git a/src/Servers/IIS/IIS/test/FunctionalTest.props b/src/Servers/IIS/IIS/test/FunctionalTest.props index f9fb3067bb..9150054fc4 100644 --- a/src/Servers/IIS/IIS/test/FunctionalTest.props +++ b/src/Servers/IIS/IIS/test/FunctionalTest.props @@ -16,15 +16,17 @@ + + + - + - - - + + diff --git a/src/Servers/IIS/IIS/test/IIS.Shared.FunctionalTests/MofFileTests.cs b/src/Servers/IIS/IIS/test/IIS.Shared.FunctionalTests/MofFileTests.cs index 41d8404395..8cc6e8c9ac 100644 --- a/src/Servers/IIS/IIS/test/IIS.Shared.FunctionalTests/MofFileTests.cs +++ b/src/Servers/IIS/IIS/test/IIS.Shared.FunctionalTests/MofFileTests.cs @@ -16,6 +16,7 @@ namespace IIS.FunctionalTests [OSSkipCondition(OperatingSystems.Linux | OperatingSystems.MacOSX)] [RequiresIIS(IISCapability.TracingModule)] [Flaky("https://github.com/aspnet/AspNetCore-Internal/issues/2222", FlakyOn.Helix.All)] + [SkipOnHelix] public void CheckMofFile() { var path = Path.Combine(TestPathUtilities.GetSolutionRootDirectory("IISIntegration"), "aspnetcoremodulev2", "aspnetcore", "ancm.mof"); diff --git a/src/Servers/IIS/tools/TestCert.pfx b/src/Servers/IIS/tools/TestCert.pfx new file mode 100644 index 0000000000..7d00f5563b Binary files /dev/null and b/src/Servers/IIS/tools/TestCert.pfx differ diff --git a/src/Servers/IIS/tools/UpdateIISExpressCertificate.ps1 b/src/Servers/IIS/tools/UpdateIISExpressCertificate.ps1 index 9034cf8f75..9185181e68 100644 --- a/src/Servers/IIS/tools/UpdateIISExpressCertificate.ps1 +++ b/src/Servers/IIS/tools/UpdateIISExpressCertificate.ps1 @@ -1,20 +1,24 @@ -$cert = New-SelfSignedCertificate -DnsName "localhost", "localhost" -CertStoreLocation "cert:\LocalMachine\My" -NotAfter (Get-Date).AddYears(5) -$thumb = $cert.GetCertHashString() +$ErrorActionPreference = 'Stop' -$Store = New-Object -TypeName System.Security.Cryptography.X509Certificates.X509Store -ArgumentList 'root', 'LocalMachine' -$Store.Open([System.Security.Cryptography.X509Certificates.OpenFlags]::ReadWrite) -$Store.Add($cert) -$Store.Close() - -$tempFile = [System.IO.Path]::GetTempFileName(); -$content = ""; - -for ($i=44300; $i -le 44399; $i++) { - $content += "http delete sslcert ipport=0.0.0.0:$i`n"; - $content += "http add sslcert ipport=0.0.0.0:$i certhash=$thumb appid=`{214124cd-d05b-4309-9af9-9caa44b2b74a`}`n"; +if (-not $PSScriptRoot) { + # Older versions of Powershell do not define this variable + $PSScriptRoot = Split-Path $MyInvocation.MyCommand.Path -Parent } -[IO.File]::WriteAllLines($tempFile, $content) +# The certificate thumbprint of TestCert.pfx +$thumb = 'CBA8EF428446072286B8201C2877F5EF0EA3B804' -netsh -f $tempFile -Remove-Item $tempFile; \ No newline at end of file +& certutil -f -v -p testpassword -importpfx "$PSScriptRoot\TestCert.pfx" +if ($lastexitcode -ne 0) { + throw 'Failed to import test certificate into machine root store. This is required for IIS Express tests.' +} + +$tempFile = [System.IO.Path]::GetTempFileName() + +for ($i=44300; $i -le 44399; $i++) { + Add-Content -Path $tempFile "http delete sslcert ipport=0.0.0.0:$i" + Add-Content -Path $tempFile "http add sslcert ipport=0.0.0.0:$i certhash=$thumb appid=`{214124cd-d05b-4309-9af9-9caa44b2b74a`}" +} + +& netsh -f $tempFile +Remove-Item $tempFile -ErrorAction Ignore