* Fix some post-build signing issues
This fixes some post-build signing issues that are present in the aspnetcore repo
1. Add the .msi extension to be signed by Microsoft400 - Msis must be signed. With in-build signing these get handled explicitly by the wixproj infrastructure. When we do post build signing, we must sign these files.
2. Remove the strong name exclusions. These exclusions are incorrect when applied in post-build and unnecessary for in-build signing. Most importantly, the aspnetcore PKT would not end up re-strong named (it doesn't need to be strong name signed by ESRP since it's strong named in-build) because the PKT doesn't match any of the StrongNameSignInfo specified in arcade. The rest of the entries seem to be mostly about optimization. I could not find any performance difference between these entries being present and not. I am not sure whether they actually even apply to any assets. Moreover, when doing post-build signing, they would conflict with the entries in runtime and other places.
Verification - I have a tool that I wrote which unpacks every file between two directories and compares the strong name, nuget, and authenticode certs between equivalent files. This is the same tool being used to verify post-build signing. This tool shows no difference in any aspnetcore produced asset.
Baseline: https://dev.azure.com/dnceng/internal/_build/results?buildId=836183&view=results
Diff: https://dev.azure.com/dnceng/internal/_build/results?buildId=837176&view=results
* Do not push VS packages for installers when PostBuildSign == true
* Output wix command packages to the installers output path
* Don't import microbuild signing targets from wix when PostBuildSign=true
* Tweaks:
- Don't sign wixpacks when not in post-build signing
- Generate a wixpack for both the original msi name (which the wixproj generates) AND the name we use in the final outputs. This is because while these files are the same, signing differentiates the certificate based on the file name, and wixpack lookup is also based on the file names. Aspnetcore and other repos have uses the final outputs (e.g. dotnet-aspnetcore-runtime-123.5..) as well as the internal names (e.g. AspNetCoreSharedFramework_x64.msi).
- Don't sign msi's when not post-build signing.
* Avoid generating sha512 files for wixpack zips
* Don't run xplat code sign jobs if PostBuildSign == true
* Change original target names
* Conditionalize codesign operations
* Add publishing flag for linux x64 and add deb sha512 generation
* Do not push the x64 linux runtime archive more than once
Changes WiX toolset used to 3.14 to support ARM64
Generates targeting pack from the x86/x64 leg, as it gets produced using a zip that gets generated there.
The ARM64 leg now produces all the necessary msi's, exe, and wixlib needed for the installer to generate a bundle.
- pin Microsoft.Net.Compilers.Toolset version to isolate us from Arcade
- the version now matches dotnet/runtime
- may move the pin to later version later in RC2 if needed
- double the macOS job max. length in our normal and quarantined PR runs
- have been seeing them timeout or come very close too often
* Grab binary logs for main Windows and Linux jobs
- in the Windows case, do not do this in official builds (logging slows build down)
nit: do not set variables with only two values three times
* Do not sign twice in Windows Code-sign build step
nit: correct wording in Signing.props
* !fixup! Don't grab the large x86 binary log
This pull request updates the following dependencies
[marker]: <> (Begin:2f3f3ff0-d34f-49bd-50aa-08d828f9655f)
## From https://github.com/dotnet/runtime
- **Subscription**: 2f3f3ff0-d34f-49bd-50aa-08d828f9655f
- **Build**: 20200722.7
- **Date Produced**: 7/23/2020 12:04 AM
- **Commit**: 3cab6dd440a5f3763bfbd2d582b36fe51095686a
- **Branch**: refs/heads/internal/release/5.0-preview8
[DependencyUpdate]: <> (Begin)
- **Updates**:
- **System.Diagnostics.DiagnosticSource**: from 5.0.0-preview.8.20361.2 to 5.0.0-preview.8.20372.7
- **System.Diagnostics.EventLog**: from 5.0.0-preview.8.20361.2 to 5.0.0-preview.8.20372.7
- **System.Drawing.Common**: from 5.0.0-preview.8.20361.2 to 5.0.0-preview.8.20372.7
- **System.IO.Pipelines**: from 5.0.0-preview.8.20361.2 to 5.0.0-preview.8.20372.7
- **System.ComponentModel.Annotations**: from 5.0.0-preview.8.20361.2 to 5.0.0-preview.8.20372.7
- **Microsoft.Extensions.Logging.Abstractions**: from 5.0.0-preview.8.20361.2 to 5.0.0-preview.8.20372.7
- **Microsoft.Extensions.Logging.Configuration**: from 5.0.0-preview.8.20361.2 to 5.0.0-preview.8.20372.7
- **Microsoft.Extensions.Logging.Console**: from 5.0.0-preview.8.20361.2 to 5.0.0-preview.8.20372.7
- **Microsoft.Extensions.Logging.Debug**: from 5.0.0-preview.8.20361.2 to 5.0.0-preview.8.20372.7
- **Microsoft.Extensions.Logging.EventLog**: from 5.0.0-preview.8.20361.2 to 5.0.0-preview.8.20372.7
- **Microsoft.Extensions.Logging.EventSource**: from 5.0.0-preview.8.20361.2 to 5.0.0-preview.8.20372.7
- **Microsoft.Extensions.Logging.TraceSource**: from 5.0.0-preview.8.20361.2 to 5.0.0-preview.8.20372.7
- **Microsoft.Extensions.Options**: from 5.0.0-preview.8.20361.2 to 5.0.0-preview.8.20372.7
- **Microsoft.Extensions.Options.ConfigurationExtensions**: from 5.0.0-preview.8.20361.2 to 5.0.0-preview.8.20372.7
- **Microsoft.Extensions.Options.DataAnnotations**: from 5.0.0-preview.8.20361.2 to 5.0.0-preview.8.20372.7
- **Microsoft.Extensions.Primitives**: from 5.0.0-preview.8.20361.2 to 5.0.0-preview.8.20372.7
- **Microsoft.Extensions.Logging**: from 5.0.0-preview.8.20361.2 to 5.0.0-preview.8.20372.7
- **Microsoft.Extensions.Internal.Transport**: from 5.0.0-preview.8.20361.2 to 5.0.0-preview.8.20372.7
- **Microsoft.Extensions.Hosting.Abstractions**: from 5.0.0-preview.8.20361.2 to 5.0.0-preview.8.20372.7
- **Microsoft.Extensions.Caching.Abstractions**: from 5.0.0-preview.8.20361.2 to 5.0.0-preview.8.20372.7
- **Microsoft.Extensions.Caching.Memory**: from 5.0.0-preview.8.20361.2 to 5.0.0-preview.8.20372.7
- **Microsoft.Extensions.Configuration**: from 5.0.0-preview.8.20361.2 to 5.0.0-preview.8.20372.7
- **Microsoft.Extensions.Configuration.Abstractions**: from 5.0.0-preview.8.20361.2 to 5.0.0-preview.8.20372.7
- **Microsoft.Extensions.Configuration.Binder**: from 5.0.0-preview.8.20361.2 to 5.0.0-preview.8.20372.7
- **Microsoft.Extensions.Configuration.CommandLine**: from 5.0.0-preview.8.203...
* Remove tests from official builds
- #22787
nit: _add_ dependency on Windows ARM64 build when publishing to the BAR
- not a major problem because this particular build rarely if ever fails
- the existence of the correct manifests is much more important
* Address nit: `Windows_64_build` -> `Windows_arm64_build`
* Build time changes
A few changes for build time
- Don't build tests with SkipTestBuild=true and use that for official
build legs. This cuts 40%-50% off the msbuild invocations for build.
The longest build leg drops by about 30 mins.
- Skip logging of some task parameters and their metadata.
This reduces overall binlog size, which is a major contributor to
build time.
Unfortunately, this does not mean we can yet turn binlogs back on. This
change can actually increase the overall binlog size due to logging of
more project started arguments. There is another optimization for this
in progress.
Co-authored-by: Doug Bunting <6431421+dougbu@users.noreply.github.com>
* Make `dotnet msbuild` the default on Windows too
- add step using desktop `msbuild` when native builds may be involved
- `-All` (without `-NoBuildNative`), `-BuildNative` or `-BuildInstallers` run this step
- but `-ForceCoreMsbuild` unconditionally skips this step
nits:
- add binary log for RepoTasks build if `$BinaryLog` (echoes the `dotnet msbuild` command)
- add blank lines between build steps
* Enable building managed projects depending on native assets
- splitting native builds out confuses these projects
- use `$(BuildNative)` less, only to control actual building (not bundling)
- build both native platforms in one `msbuild` invocation
* Adjust generation scripts to explicitly choose the MSBuild engine
- ensure native assets are included in GenerateReferenceAssemblies.ps1 build
- clean up the global state that tools.ps1 corrupts
* Revert move to VS2019.Pre queues
This reverts part of commit b67d161e03
- was "[release/5.0-preview5] Update dependencies from dotnet/aspnetcore-tooling (#21710)"
* Revert "!temporary! Require `msbuild` from VS2019 16.6"
- this reverts commit 58cf2304a6
* Reduce build duplication in pipelines
- build native assets and repo tasks once per CI job
- only cleanup framework references after packing managed projects
nits:
- wrap a few long lines
- remove extra `-forceCoreMsbuild` options in SiteExtensions' build.cmd
* Fix Helix jobs
- restore.cmd doesn't work well with `-projects`; script unconditionally adds `-all`
* !fixup! Reduce duplications further
- missed a couple of places `-noBuildRepoTasks` helps
* Cleanup: Remove a few dangling binary logs
* !fixup! Correct typos in generation scripts
* !fixup! Another typo in the generation scripts
* Avoid changing global state in CI runs II
- move `dotnet` tools into `$PWD/.dotnet/tools/`
- add `dotnet` tools to `$env:PATH`
- install `jq` in `$PWD/.tools/`
- install `nginx` in `$PWD/.tools/nginx/`
nits:
- remove an extra (incorrect) `$env:JAVA_HOME` setting
- build.ps1 finds a local JDK and sets this variable (and `$env:PATH`) properly
- avoid `'$(SELENIUMPROCESSTRACKINGFOLDER)' == '\artifacts\tmp\selenium\'`
- `$(BuildDirectory)` was normally the empty string (w/o fix)
- use `$(BuildDirectory)` a bit more
* Add files. disable artifact / test results publications to avoid warnings on CI
- artifacts/log is often empty when binary logs are disabled
- no tests create *.trx files
- don't have Java test results either
- only test jobs need test results published
* Update dependencies from https://github.com/dotnet/arcade build 20200511.9
- Microsoft.DotNet.Arcade.Sdk: 5.0.0-beta.20228.4 => 5.0.0-beta.20261.9
- Microsoft.DotNet.GenAPI: 5.0.0-beta.20228.4 => 5.0.0-beta.20261.9
- Microsoft.DotNet.Helix.Sdk: 5.0.0-beta.20228.4 => 5.0.0-beta.20261.9
* Update dependencies from https://github.com/dotnet/aspnetcore-tooling build 20200514.6
Microsoft.AspNetCore.Mvc.Razor.Extensions , Microsoft.AspNetCore.Razor.Language , Microsoft.CodeAnalysis.Razor , Microsoft.NET.Sdk.Razor
From Version 5.0.0-preview.5.20261.4 -> To Version 5.0.0-preview.6.20264.6
* Pre-emptively take -nobl change
* Disable binlogs in CI
* Fix build.sh to know about -nobl
* Align build.ps1|sh with latest Arcade parameters
- do not enable binary logs by default in CI builds
- leave `-binaryLog` and `-excludeCIBinaryLog` handling to eng/common/tools.ps1|sh
- was unnecessary since `-bl /bl:{some name}` worked fine, ignoring OOMs
nit: document `-excludeCIBinarylog` a bit more
* Do not pass unknown options into CodeCheck.ps1
* Pass `-ci -nobl` into remaining CI build jobs
* Switch default TFM to `net5.0`
* Update missing project templates tfms
* Add more `-ci -nobl`
- needed because _all_ builds in the pipeline are implicitly CI builds
- default-build.yml adds `-ci` when script wasn't explicit
* Default templates to net5.0
* PR feedback
* Update TFMs in explicit .nuspec files
* Update TFMs in test projects
* Update TFMs in test C# code
* Update TFMs in infrastructure files
* Future-proof a check for `net5.0` or later
- avoid comparisons involving `$(TargetFramework)` in .targets files
- fine to compare it with `''` or `$(DefaultNetCoreTargetFramework)`
* !fixup! Undo a couple of earlier fixes
- remove a duplicate `$()` setting
- correct the one remaining versioned `#if` define
- did not make it `#if NETCOREAPP` because benchmarks test numerous .NET Core TFMs
* Disable binary logs in CodeCheck.ps1
* Specify `-ci -nobl` just once when using `parameters.buildArgs`
* Restore `$binaryLog` default logic
Co-authored-by: Doug Bunting <6431421+dougbu@users.noreply.github.com>
Co-authored-by: Will Godbe <wigodbe@microsoft.com>
Co-authored-by: Viktor Hofer <viktor.hofer@microsoft.com>
- set `$env:DOTNET_CLI_HOME` because we need to install global tools in this repo
- without this, we see `dotnet-serve` installation failures on unclean machines
* Quick fix: Simplify devBuilds.yml
- remove an excess build step
* Quick fix: Move `SetupNugetSources` script invocations above `parameters.beforeBuild`
- ensure NuGet.config is ready for all internal builds
- remove now-duplicate `SetupNugetSources` invocations wherever default-build.yml is used
* Quick fix: Clean up SiteExtensions/build.cmd
- quote all rooted paths
- check `%ERRORLEVEL%` after every `CALL`
- nits:
- add a few more `ECHO` commands
- wrap long lines
* Quick fix: Ensure `$(BuildNative)` is always set correctly
- fix problems using `-all` or `/p:BuildAllProjects=true` without `-buildNative`
- ensure `$(BuildNative)` is `false` where it's not supported
- move some duplicated settings into eng/Common.props and `<Import />` the new file
- remove now-duplicated parts of conditions using `$(BuildNative)`
* Quick fix: Consistently use `--build-*`
- avoid `/p:Build*` on the command line (except with eng/scripts/ci-source-build.sh)
- nits:
- remove now-useless `-buildNative` with `-all`
- expand and correct a couple of related comments and messages
* Quick fix: Support `-all` together with `-projects`
- remove need to specify `/p:BuildAllProjects=true`
- nit: simplify some Boolean logic
- Use different Ubuntu pools for Ubuntu tests to avoid disk space issues.
- Updated the `default-build.yml` to have a new parameter specific to Ubuntu listed `useHostedUbuntu`. If we feel the need to expand the parameters usage outside of the Ubuntu configuration we can always rename to `useHosted`. Also didn't want to touch the `isTestingJob` because I wasn't sure of hte implications.
- Updated the Ubuntu test job to turn off hosted pools.
Fixes https://github.com/dotnet/aspnetcore-internal/issues/3574
- Hmm, are `queue` names case-sensitive?
- Correct `useHostedUbuntu: false` placement
- Correct placement of `useHostedUbuntu: true` default
- Try adding `--without-http_rewrite_module` to nginx configuration
- also add `set -euo pipefail` to the script for fast failure
Co-authored-by: Doug Bunting <6431421+dougbu@users.noreply.github.com>
* Always generate checksums as last part of publish job
* Initialize props correctly
* Fix wildcard
* Import Arcade SDK
* Add NoWarn MSB4011
* Make import conditional on GenerateChecksums
* Select specific files to checksum
* Respond to feedback
* AfterTargets -> BeforeTargets
- dotnet/aspnetcore-internal#3540
- nit: Consistently use `in` / `notin` with `Build.Reason`
- YAML was inconsistent and this aligns w/ the Arcade code
* WIP add interop tests
* Clean up
* Move test project into build infrastructure
* Clean up
* Remove hardcoded paths
* Clean up
* Fix build
* Add copyright notices
* Update azure template
* Fix build
* Fix build?
* Fix build?
* Add gRPC interop tests to CI
- Convert to using references managed by build infrastructure
- Use produced AspNetCore.App shared framework
- Save server logs
- Dynamically bind to ports
- Ensure InteropWebsite is built in the same configuration as the test project
* Cleanup
* Rebase fix
* Include tests assets in build directory for Helix
* Incorporate changes in ProcessEx
* Include Grpc test in regular build
* Fixup
* Test
* exe doesn't always exist
* Capture logs on helix
* Maybe this will work
* There are two application started messages
* Derp
* Cleanup
* Update directory
* Add interop tests to more pipelines
* mkdir
Co-authored-by: John Luo <johluo@microsoft.com>
Matt Mitchell
Epsitha Ananth
JC Aguilera
Juan Hoyos
Doug Bunting
Will Godbe
dotnet-bot
DotNet Bot
Pranav K
John Luo
Justin Kotalik
Brennan
Hao Kung
dotnet-maestro[bot]
Chris Ross
N. Taylor Mullen
Kevin Pilch
Ryan Nowak
James Newton-King