Commit Graph

70 Commits

Author SHA1 Message Date
Doug Bunting 477fa8ce4c
Find or install Tar on CI
- work around dotnet/core-eng#7970
- install Git in repo if Tar can't be found in usual locations
- use found or installed Tar in Microsoft.AspNetCore.App.Ref project
  - copy into repo from wherever it's found
- add lots of `Write-Host` debugging

nit: clean up / comment on VS Code warnings about build.ps1
2019-09-30 16:09:12 -07:00
Javier Calvarro Nelson 2e9484b44e
[Infrastructure] Separate dump capturing and upload from log updating and upload (#14198)
* Separates the dumps into its on set of artifacts.
* Prioritizes uploading dumps for hanging builds before any other artifact.
2019-09-20 21:07:28 +02:00
Doug Bunting 7d4b691d8b Provide more information when pipelines fail
- restore publication of flaky test results to AzDO Tests tab
- upload logs and test results for forks too
2019-09-18 15:50:30 -07:00
Javier Calvarro Nelson 85af1fd66b
[Infrastructure] Adds support for capturing process dumps for hanging builds on Windows (#13912)
* Downloads and installs ProcDump as part of the build.
* Registers a scheduled job that wakes up in 160 minutes.
* Upon waking up, the background job lists all the processes it
  received as candidates.
* For each process it captures a full memory dump.
* At the end of the build, a separate step checks on whether the job
  ran (the build hanged) or everything is correct and displays statistics.
* If dumps are found, they are collected and made available as artifacts
  under Windows(_Templates)_Tests_Logs.
2019-09-16 21:54:20 +02:00
Doug Bunting 04705ee4f1
Use stages pipeline (#13040)
- #11924
- change ci.yml to use stages and post-build.yml
  - add use of publish-build-assets.yml and post-build.yml
  - create manifests and push to artifacts in last build step of each job
    - pass more MSBuild properties into those builds
    - use Arcade to publish installers
  - use distinct `$(AssetManifestFileName)` values per job
    - set global property to override what's hard-coded in Publish.proj
- change codesign-xplat.yml to use empty.proj and normal Arcade signing and publication process
  - remove XPlatPackageSigner.proj
- change default-build.yml to use job.yml
  - remove unused parameters e.g. `matrix`, `poolName`, `variables`
  - use `enableMicrobuild` and `enablePublishTestResults` to eliminate duplicate build steps
- add .dll's and .exe's as files to sign w/ Microsoft400
  - add signcheck exclusions
- remove custom manifest generation i.e. the `GenerateBuildAssetManifest` target and related artifacts
- update docker infrastructure to use same paths in and out of the container
  - avoids problems adding to artifacts from within the builds
- correct typo in build.sh
- use `$env:DOTNET_INSTALL_DIR` in `DotNetCommands`
  - relax expectations that an arch-specific folder exists under (say) `$env:DOTNET_HOME`
  - avoids need to define `$env:DOTNET_HOME` in all jobs on CI
- update dependencies from dotnet/arcade build '20190908.2'
  - upgrade to eg. Arcade SDK '1.0.0-beta.19458.2' package version
  - pick up dotnet/arcade@dd593acc8b fix
    - enable use of `%(PublishFlatContainer)` metadata and correct signing validation issues
- use `$(DotNetFinalVersionKind)` in preparation for servicing builds
  - set `$(IsStableBuild)` for use in Arcade infrastructure
- disable signing validation for now (see #13864)

nits:
- upload logs in first artifact
- remove attempts to package non-existent VSIX
  - follow-up to 29cf7ecb80
- respect verbosity setting in build.sh
- add more information to Artifacts.md
- enable test signing in internal PRs
2019-09-10 20:43:24 -07:00
Ryan Nowak 706309f266 Increase cancelTimeoutInMinutes
This is a setting AZDO supports to control how long you have after a
build is canceled before all of the work is terminated.

When a build reaches a timeout, usually the `build.{cmd|sh}` step will
be terminated, and then you have cancelTimeoutInMinutes number of
minutes to do everything else before everything shuts down.

We need to a value larger than the default of 5 minutes because it takes
us *about* 5 minutes to upload all of our logs and artifacts in a
success case.

This should give us a sufficient grace period to upload all of our logs
in the case that a build fails due to a test hang.
2019-08-02 22:24:43 -07:00
John Luo 7f3e2acdac
Use Enterprise Queue to reduce provisioning issues (#11434) 2019-06-20 22:50:25 -07:00
Nate McMaster 4fde84a7ad Replace KoreBuild with Arcade (#11122)
* Replace usages of $(CI) with $(ContinuousIntegrationBuild)

* Move RepoTasks to eng/tools/

* Remove dead code

* Update build.sh/ps1 to invoke Arcade instead of KoreBuild

* build/repo.props => eng/Build.props

* build/CodeSign.{props,targets} => eng/Signing.props

* build/repo.targets => eng/Build.props, eng/CodeGen.proj, and eng/AfterSolutionBuild.targets

* Remove AddAllProjectRefsToSolution.ps1

* Put selenium process tracking into artifacts/tmp/

* build/docker => eng/docker

* Move content from korebuild.json to global.json

* Remove cibuild.{sh,cmd} and update ci.yml

* Ensure vswhere detects BuildTools installations

* Remove sign check exclusions (Arcade does not support)

* list VS required components in global.json

* Update CodeCheck.ps1

* Update arcade (#11143)

* Fixups to broken build

* capture test results in xunit form
* attempt to fix code check
* restore before linux build
* remove duplicate signinfos

* More build fixes

* exclude node_modules from unique project check
* fixup signing props

* More build fixes

* Remove unused NoWarns
* Skip building site extension
* Suppress xunit color in console output
* Install x86 runtime

* Run KillProcesses.ps1 at end of build on CI

* activate env vars in codecheck.ps1

* Fix signing and generate build manifests

* Fixup code check and linux installers build

* Remove custom restore targets

* Import flaky test props

* Restore original manifest generation code and set warnAsErrors=false

* Alwasy restore the toolset

* More attempts at build fixes

* run restore before building deb/rpm packages
* pre-install Yarn.MSBuild
* Remove WIX workaround

* Attempt to fix RPM installers and x86 build

* Fix helix tests, java auto-detection, and code gen steps

* Globally disable MSBuild node resuse and capture MSbuild crash logs

* Cleanup build invocation for installers

* Set max time on xunit tests to 15 min

* Update helix-test.yml

* Fix Hosting tests

* Fix TestHost

* Fix Negotiate.Test

* Fix Mvc tests

* Capture results

* Fix Hosting tests

* Fix Kestrel tests

* Fix ServerComparison tests

* Fix DefaultBuilder functional tests

* Skip template tests

Need to run pack before running tests on mac/linux now

* Fix Identity test

* Fix ServerComparison tests

* Verify nginx installation on ubuntu

* Verify nginx installation?

* Try adding to path

* Try symlink

* Sudo?

* Woot it works

* Disable timeout

Template tests require more than 15 mins

* list env vars

* Use correct dotnet_home

* Prepent path instead

* Fix npmproj tests

* Fix CORS tests

* Debug signalr tests

* Debug signalr

* Capture bin

* Try building templates sequentially

* Debug signalr

* Cleanup

* Disable signalr npm tests

* Debug template test

* Try to fix templates again and make steps more reliable

* Cleanup

* Feedback
2019-06-19 10:03:38 -07:00
John Luo af2f647d63
Run signcheck if previous steps succeeded an try publishing build artifacts before uploading test results (#11237) 2019-06-14 15:38:11 -07:00
Nate McMaster 194b0b4523
Update install JDK script an instructions to make it easier to use for local development (#10306) 2019-05-20 08:48:54 -07:00
John Luo 818dc3b74a
Update log directory (#10301)
* Update log directory

This will now include functional test logs

* There were a few more places logs was used instead of log
2019-05-17 09:02:34 -07:00
Nate McMaster fa73edb9bf
Improve detection of JDK on Windows (#10294) 2019-05-16 12:41:50 -07:00
Doug Bunting ed886c47fe
Merge branch 'release/2.2' -> 'master' 2019-05-05 11:14:25 -07:00
Doug Bunting 8cb331f8e1
Merge branch 'release/2.1' into release/2.2 2019-05-04 22:43:55 -07:00
Doug Bunting 6a6a119c9a Switch to non `-Int` pools and clear NuGet caches
- aspnet/AspNetCore-Internal#2436
- aspnet/AspNetCore#9767
2019-05-04 22:23:33 -07:00
Doug Bunting 2e94885ecd
Merge branch 'release/2.2' into merge/release/2.1-to-release/2.2 2019-04-19 22:49:13 -07:00
Doug Bunting e96a53f9e7
Support internal pull requests (#9578)
- aspnet/AspNetCore-Internal#2231
- use internal pools for all internal builds
- do not sign in builds for internal pull requests

nits:
- VSTS -> Azure DevOps
- restore higher `maxParallel` values for E2E tests on Linux and Windows
2019-04-19 18:23:54 -07:00
Nate McMaster 52f3f79263 Use VS2019 Enterprise queues for testing (#9571) 2019-04-19 14:54:52 -07:00
Nate McMaster d4eaeec5a5
Stop running InstalVisualStudio.ps1 during CI builds (#9515) 2019-04-19 07:51:51 -07:00
Mikael Mengistu 4964b338ab
Report Java client test results (#9214) 2019-04-10 16:02:40 -07:00
Doug Bunting 7d9916d0f7
Merge branch 'release/2.2' => 'master' 2019-04-05 17:33:21 -07:00
BrennanConroy 074b1757d7
Report JUnit test results for SignalR TS tests (#9058) 2019-04-04 08:21:19 -07:00
Doug Bunting ec82890133
Merge branch 'release/2.1' into release/2.2
- move to latest release/2.2 commit from aspnet/EntityFrameworkCore
2019-04-03 08:42:07 -07:00
Doug Bunting 4072f70b8a
Do not specify `name` of the default (hosted) pools (#9002)
- aspnet/AspNetCore#8862
    - `Hosted Linux Preview` pool name was also no longer valid; aspnet/AspNetCore-Internal#1880
- update e2e-test.yml to use modern AzDO job schema
    - https://github.com/dotnet/arcade/blob/master/Documentation/AzureDevOps/PhaseToJobSchemaChange.md
    - also update comments to refer to the job schema
- use BYOC queues on Windows; part of aspnet/AspNetCore-Internal#2033
- pick up latest release/2.1 commit for aspnet/EntityFrameworkCore for related changes
2019-04-02 16:37:37 -07:00
Doug Bunting 5fa1c241ec
Move to BYOC queues
- aspnet/AspNetCore-Internal#2033
- use conditions matching aspnet/Extensions and dotnet/Arcade
  - e.g. 05cb24592a/azure-pipelines.yml (L49-L54)
- install necessary Build Tools for Visual Studio 2019 components and workloads on CI
- revert part of 85ae18c723 because multiple editions of VS 2019 are now publicly available
  - that is, restore support for multiple editions in InstallVisualStudio.ps1
- update InstallVisualStudio.ps1 to
  - support BuildTools edition of VS 2019

nit: merge a couple of Windows build steps
  - support automatically updating latest version of a VS edition  on the machine
  - include `-Quiet` option for a completely non-interactive installation
2019-03-26 18:46:59 -07:00
Javier Calvarro Nelson 9f1a978230
[Infrastructure improvements] (#8275)
* Improved selenium start and tear down
  * Selenium is set up and torn down in an assembly fixture.
  * Selenium is initialized lazily and in a non-blocking way.
  * Selenium processes are tracked as part of the build and their pids
    written to a file on disk for cleanup in the event of unexpected
    termination of the test process.
  * Browser fixture retries with linear backoff to create a remote
    driver. Under heavy load (like when we are doing a simultaneous NPM
    restore) the selenium server can become unresponsive so we retry
    three times, with a longer comand timeout allowance each time up to
    a max of 3 minutes.
* Moved test project setup to build time instead of runtime.
  * Added target PrepareForTest to create the required files for testing
    * The template creation folder.
    * The template props file to use our built packages.
    * The folder for the custom hive.
  * Added assembly metadata attributes to find all the data we need to
    run the tests.
    * Path to the artifacts shipping packages folder.
    * Path to the artifacts non-shipping packages folder.
    * Path to the test templates creation folder.
    * Path to use for the custom templating hive used in tests.
  * Proper cleanup as part of the build
    * Remove the test templates creation folder.
    * Remove the test packages restore path.
    * Recreate the test templates creation folder.
    * Recreate the test packages restore path.
  * Generated Directory.Build.Props and Directory.Build.Targets in the
    test templates creation folder.
  * Cleaned up potentially stale templatetestsprops.
* Improved test flows
  * Initialization is done lazily and asynchronously.
    * Selenium
    * Browser fixture
    * Template initialization.
  * Flattened test flows to avoid assertions inside deep callstacks.
    * All assertions happen at the test level with improved error messages.
      * With the exception of the migrations assertions.
    * Assertions contain information about which step failed, for what
      project and what failure details.
  * Broke down tests to perform individual steps instead of mixing build
    and publish.
    * Publish project.
    * Build project. (Debug)
    * Run built project.
    * Run published project.
  * Concentrated build logic into the Project class.
    * Context between the different steps of a test is maintained in
      this class.
    * All operations that require coordination are performed within this
      class.
      * There is a lock for dotnet and a lock for nodejs. When building
        SPAs we acquire the nodejs lock to correctly prevent multiple
        runs of nodejs in parallel.

[ApiAuthorization template cleanups]
  * Fix preview3 issues with breaking changes on Entity framework by
    manually configuring the model in ApiAuthorizationDbContext.
  * Add app.db to the project file when using local db.
  * Fix linting errors on angular template.
  * Fix react tests
  * Add tests to cover new auth options in the SPA templates.
2019-03-20 08:44:20 +01:00
Andrew Stanton-Nurse 706778d4de
Add a separate test pass to run flaky tests that doesn't fail the build (#8486)
Similar to https://github.com/aspnet/Extensions/pull/1239.
2019-03-18 18:21:14 -07:00
Doug Bunting 0c6591baea
Merge release/2.1 up to commit d2a4435ac8 => release/2.2
- remove updates about baseline test (5661c41909 and d2a4435ac8)
  - already done (differently) in release/2.2
- add JDK installation -- missing in these images but not needed in `release/2.1`
- add packages needed to run Chrome
2019-03-13 19:17:05 -07:00
Javier Calvarro Nelson d7a9606040
[Templating][Components] Unify and improve E2E testing infrastructure (#8188)
* Unify the Templating and Components testing infrastructure.
* Enable test project discovery in the components E2E tests.
* Enable selectively disabling Selenium tests through build properties.
2019-03-11 18:40:18 +01:00
Doug Bunting c902b18b0d Bump up the AzDO default job timeout to 3 hours
- aspnet/AspNetCore-Internal#1909
2019-03-07 13:00:38 -08:00
N. Taylor Mullen 4093399e8c Skip first run experience on azure pipelines.
- Suppresses the 'Welcome to .NET Core!' output that times out tests and causes locked file issues. When using dotnet we're not guarunteed to run in an environment where the dotnet.exe has had its first run experience already invoked. This would cause our functional tests to time out and occasionally crash due to dotnet first run experience sentinels being locked.

aspnet/AspNetCore-Internal#1859
2019-03-01 10:35:04 -08:00
Doug Bunting 4f2a0edc97
Update Azure queues for Linux and MacOS builds (#7231)
- aspnet/AspNetCore-Internal#1717
- use a specific Linux image
2019-02-25 09:50:01 -08:00
Nate McMaster fbaa3e8b77
Always kill processes and remove direct reference to component governance task (#7793) 2019-02-21 09:29:42 -08:00
Nate McMaster e1d506e961
Fix codesign-xplat.yml to use the right API for setting custom build steps 2019-02-20 14:42:07 -08:00
Nate McMaster 85ae18c723
Make Visual Studio 2019 a prerequisite to building this repo (#7005)
Changes:

* Make Visual Studio 2019 a prerequisite for building this repo
* Update .sln files
* Update Windows SDK to 17134
* Update developer docs
* Disable ANCM tests
* Update to .NET Core SDK 3.0 Preview 2
* Use Microsoft.NET.Sdk.Razor as a package consistently accross the repo
* React to changes in metadata from Microsoft.NETCore.App
* React to changes in .NET Core SDK
* Attempt to workaround CodeCheck.ps1 failure which doesn't repro locally or on different agents. Possibly due to differences in the version of the PowerShell task?
* Remove dead YML file
* Rename usages of win7-{x64,x86} to win-{x64,x86}
* Update KoreBuild to 3.0.0-build-20190219.1
2019-02-20 09:53:53 -08:00
Doug Bunting 45ecfad074 Skip code signing job if dependencies aren't available 2019-02-08 09:04:15 -08:00
Doug Bunting f280465275
Merge branch 'release/2.2' 2019-02-04 22:04:46 -08:00
Doug Bunting af463acc38
Merge branch 'release/2.1' into release/2.2 2019-02-04 20:53:34 -08:00
Doug Bunting f5ca141e9d
Kill processes after completing build and before publishing test results or artifacts (#7228)
- aspnet/AspNetCore-Internal#1735
- port aspnet/Extensions@e2bb8ac1a0 fix into this repo
  - change `default-build.yml` instead of `azure-pipelines.yml`
  - make a couple of changes to `KillProcesses.ps1` (which I'll take back to Extensions)
- remove `ci-official.yml`
2019-02-04 16:19:31 -08:00
Justin Kotalik 1d2838ea2e
Upload artifact logs for CI Test runs (#7251) 2019-02-04 13:55:23 -08:00
Nate McMaster 4c4a1effcd
Only publish artifacts on successful builds and fix code-signing for the SiteExtension package (#7199) 2019-02-01 19:17:01 -08:00
Nate McMaster 7ee3399ce2
Fix code-signing for xplat packages and Windows installers (#7191) 2019-02-01 10:52:00 -08:00
Nate McMaster 6081fec894
Add new build.cmd/sh options for suppressing restore, NodeJS, setting target arch and OS name, and more (#7137)
Add new command line parameters for working with the project:

* `-NoBuild`, `-NoRestore` - these already existed, but users found it hard to discover this powershell syntax: '-build:$false'
* `-Arch`/`--arch` - set the target CPU architecture to build. Defaults to x64
* `--os-name` - on non-Windows builds, manually specify if the build should target Alpine. generic Linux, or MacOS
* Rename flags used to build specific project types. The pattern now is `--build-$(group)` or `--no-build-$(group)` (In PowerShell its `-Build$(Group)` or `-NoBuild$(Group). Example: -NoBuildJava

Changes to build definitions:
* Update the ci build definition to build all supported architectures
* Support publishing multiple artifacts per job

Other changes:
* `-NoBuild` implies `-NoRestore`
* Add new properties, `TargetArchitecture`, `TargetOsName`, and `TargetRuntimeIdentifier`
* Replace usages of `SharedFxRid` with these new properties
* To make `--no-build-nodejs` actually work, replaced Components.Browser.JS.csproj with Components.Browser.JS.npmproj
* Fix errors when building for win-arm on a clean machine
* Fix a few other project errors, like using the wrong syntax for DefaultItemExcludes, or using the wrong Platform value for x86
2019-01-31 15:47:47 -08:00
Pavel Krymets 5ba89945ea
Fix site-extension build step artifacts path (#7138) 2019-01-30 11:57:21 -08:00
Pavel Krymets 78fabbdb40
Build 3.0 AggregateSite extension (#6849) 2019-01-28 20:09:51 -08:00
Sébastien Ros 768e9052dd
Adding artifacts generation for forks (#6324) 2019-01-28 18:02:57 -08:00
Nate McMaster eef9f76759
Fix SignalR daily tests (#6960)
Ensure the TeamName variable is set. It's required for all internal builds.
2019-01-23 15:15:34 -08:00
Pavel Krymets 6cac028716
Switch IIS tests to Hosted queue (#6746) 2019-01-16 11:47:52 -08:00
Nate McMaster 089912cd1b
Switch queues used during Windows builds (#6667) 2019-01-14 10:07:12 -08:00
Nate McMaster b4f869b2ff
Switch agent queues used on Windows builds (#6627) 2019-01-11 16:23:13 -08:00