source-build: bundle NativeAOT libraries with the SDK. #41198

tmds · 2024-05-28T09:02:21Z

This includes the libraries that are needed for source-built NativeAOT with the source-built SDK:

runtime.<rid>.Microsoft.DotNet.ILCompiler extracted under 'packs'.
Microsoft.DotNet.ILCompiler and Microsoft.NET.ILLink.Tasks as NuGet packages under 'library-packs'.

Implements dotnet/source-build#1215 (comment)
Contributes to dotnet/source-build#1215

@dsplaisted @baronfel @ViktorHofer @jkotas ptal

tmds · 2024-05-28T09:04:42Z

src/Installer/redist-installer/targets/GenerateLayout.targets

+        <RelativeLayoutPath>packs/%(PackageName)/%(PackageVersion)</RelativeLayoutPath>
+      </BundledLayoutPackage>
+
+      <BundledLayoutLibraryPackage Include="$(SourceBuiltShippingPackagesDir)/../runtime/Microsoft.DotNet.ILCompiler.$(MicrosoftNETCoreAppRuntimePackageVersion).nupkg" Condition="'$(BundleNativeAotCompiler)' == 'true'" />


$(SourceBuiltShippingPackagesDir)/../runtime/ this moves from the sdk repo to the runtime repo shipping packages. Perhaps there's a better way to get to that directory.

Is it different than other runtime packs (apphost, corossgen2 etc.)?

These remain as .nupkg files which are stored in the library-packs directory.

The others are "restored" against a the target framework into the packs directory. You see the extracted package content.

Perhaps there's a better way to get to that directory.

I think you want to add two PackageDownload items for both packages so that they get downloaded in the context of dotnet/sdk and then construct the path to them from the cache (NuGetPackageRoot). The nuget package itself is part of the nuget cache.

FSharp and Roslyn do something similar but not when bundling the SDK but when publishing packages:

sdk/eng/Publishing.props

Line 85 in dad1976

<FSharpPackagesToPush Include="$(NuGetPackageRoot)\Microsoft.FSharp.Compiler\$(MicrosoftFSharpCompilerPackageVersion)\contentFiles\$(FSharpCorePath)\FSharp.Core.*.nupkg" />

cc @MiYanni @dsplaisted @marcpopMSFT (owner of the bundling logic in sdk)

For now, I've added the DotNetBuildOrchestrator property and I'll create a tracking issue for eliminating it.
I think what you are proposing sounds good. I don't know how to put it all together without digging deeper.

src/Installer/redist-installer/targets/GenerateLayout.targets

tmds · 2024-06-05T11:44:26Z

@baronfel @dsplaisted this is a working prototype. I need some feedback to improve it. Can you take a look?

dsplaisted

Looks good. I'm not sure if there's a better way to get the .nupkg files to copy into the library-packs directory or not.

dsplaisted · 2024-06-06T15:54:38Z

Also, I thought there might be some updates to the .targets files needed to handle the path to the pack when it is resolved in ProcessFrameworkReferences instead of coming from NuGet restore. However, if this is working then maybe that wasn't necessary.

baronfel · 2024-06-17T17:48:54Z

/azp run dotnet-sdk-public-ci,sdk-source-build,sdk-unified-build

azure-pipelines · 2024-06-17T17:49:13Z

Azure Pipelines successfully started running 3 pipeline(s).

src/Installer/redist-installer/targets/GenerateLayout.targets

This includes the libraries that are needed for source-built NativeAOT with the source-built SDK: - runtime.<rid>.Microsoft.DotNet.ILCompiler extracted under 'packs'. - Microsoft.DotNet.ILCompiler and Microsoft.NET.ILLink.Tasks as NuGet packages under 'library-packs'.

…iler.

tmds · 2024-06-18T14:59:20Z

@ViktorHofer the sdk-source-build job hasn't failed during the runtime build like in #41198 (comment). Since that run, I've added DotNetBuildOrchestrator and rebased the PR. I'd assume DotNetBuildOrchestrator=true is for this job, so that should be a no-op. Perhaps rebasing fixed the issue.

ViktorHofer · 2024-06-18T16:41:25Z

@dotnet/source-build-internal the VMR source-build leg times out during the test step (even after retrying the leg). Any idea what's going on?

mthalman · 2024-06-18T17:27:55Z

@dotnet/source-build-internal the VMR source-build leg times out during the test step (even after retrying the leg). Any idea what's going on?

Not sure. Seems related to these changes if it's consistent and only happens with this PR. @tmds - you'll probably need to locally run the test option of the build script to diagnose what's happening.

am11 · 2024-06-18T17:52:47Z

src/Installer/redist-installer/targets/GenerateLayout.targets

@@ -202,6 +202,17 @@
        <RelativeLayoutPath>packs/%(PackageName)/%(PackageVersion)</RelativeLayoutPath>
      </BundledLayoutPackage>

+      <BundledLayoutPackage Include="MicrosoftDotNetILCompilerPackNupkg" Condition="'$(BundleNativeAotCompiler)' == 'true'">
+        <PackageName>runtime.$(SharedFrameworkRid).Microsoft.DotNet.ILCompiler</PackageName>


(Just want to make sure we are aware of what to expect)
This will only provide ILCompiler package for current RID. Note that PublishAot=true downloads separate ILCompiler packages when we issue dotnet publish -p:PublishAot=true -r <some other rid> (e.g. fedora-arm64 cross-publish on fedora-x64 system), in which case it will download the Microsoft linux-arm64 package from nuget feed. If we don't like this behavior, then I think we have two options:

distro maintainers publish ilc packages with non-portable RID to nuget.org (or some dedicated nuget feed for dotnet distros).

In nativeaot BuildIntegration issue a warning message for source-build runtimes and continue, or error that cross-compile is not supported and stop. (former is probably less drastic)

Yes, this is the behavior we want. We are enabling publishing for the SDK rid with a packed ILCompiler for that rid only. For other rids, the SDK will download a package from nuget.org.

tmds · 2024-06-18T18:10:06Z

Not sure. Seems related to these changes if it's consistent and only happens with this PR. @tmds - you'll probably need to locally run the test option of the build script to diagnose what's happening.

The test log isn't giving away anything: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/711594/logs/45.

Can you restart it once more? I will try to reproduce it tomorrow if the issue persists.

tmds · 2024-06-19T04:59:20Z

@ViktorHofer If I have to debug this, it's gong to take me some hours to patch and build the vmr before I can eventually run the tests. Can you restart the job that fails once more so we see it time out once more before I start with that?

am11 · 2024-06-19T05:34:58Z

We can close and open the PR to rerun the CI.

ViktorHofer · 2024-06-19T07:15:15Z

I retried 3 times yesterday and it always timed out. It now fails again after you closed and re-opened the PR. Let's assume the timeout is related.

I'm also trying to repro this, in a code space. It's already building and I should have results in 1-2 hours.

ViktorHofer · 2024-06-19T08:36:58Z

The smoke tests are the ones that are now timing out

tmds · 2024-06-19T08:53:47Z

I doubt the test timeouts are related to the changes in this PR since the only scenario that we're changing is publish nativeaot with a source-built rid, and this is not a scenario that is used or validated by the tests (yet).

I doubt the test timeouts are related to the changes in this PR since the only scenario that we're changing is publish nativeaot with a source-built rid, and this is not a scenario that is used or validated by the tests (yet).

I'm also trying to repro this, in a code space. It's already building and I should have results in 1-2 hours.

Thank you!

My builds are on-going.

ViktorHofer · 2024-06-19T08:56:26Z

If I'm not mistaken, the smoke tests analyze the produced SDK layout. With your changes, the SDK archive now has two additional nuget packages and is getting bigger. There might be something in those nuget packages that's slowing down the tests. Still looking more closely. The test output is terrible to debug.

tmds · 2024-06-19T09:00:08Z

the SDK archive now has two additional nuget packages and is getting bigger.

Yes, and the ILCompiler pack itself is adding 100+MB. This may be the source of a test timeout.

tmds · 2024-06-19T11:05:00Z

@ViktorHofer it would be nice if we can have some information on the test run in the CI log. Would you have a suggestion on what change to make for that?

ViktorHofer · 2024-06-19T11:14:26Z

@ViktorHofer it would be nice if we can have some information on the test run in the CI log. Would you have a suggestion on what change to make for that?

Not really, no. I don't know much about this test library aside from that it tests different scenarios. I.e. sdk diff tests, license tests, etc. I submitted #41670 a few minutes ago to improve the output in general but that won't give you more information in the CI log.

As you already have a local repro, maybe take a look at the produced .trx file or so. That might provide more information on what's taking so long.

cc @dotnet/source-build-internal in case you know which test might be the reason here.

tmds · 2024-06-19T11:29:54Z

With .NET 9 there have been some changes to the SDK/msbuild/test runner output, and I wonder if these are affecting what shows up in these logs compared to .NET 8. I've added MSBUILDENSURESTDOUTFORTASKPROCESSES=1 to the CI job config, and I hope it will produce something useful with that.

The increase in the timeout is temporary to see if it has any effect.

ViktorHofer · 2024-06-19T11:43:43Z

FWIW here's a dump of my trx file:

# > 20s
<UnitTestResult testName="Microsoft.DotNet.SourceBuild.SmokeTests.WebScenarioTests.VerifyScenario(scenario: TestScenario { Commands = Build | Run | Publish, Language = CSharp, NoHttps = False, ScenarioName = &quot;WebScenarioTests&quot;, Template = BlazorWasm, ··· })" duration="00:00:25.4895771" startTime="2024-06-19T10:05:53.5950296+00:00" endTime="2024-06-19T10:05:53.5950299+00:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="8dea9045-c475-431c-808c-5a6ead336cac">
<UnitTestResult testName="Microsoft.DotNet.SourceBuild.SmokeTests.BasicScenarioTests.VerifyScenario(scenario: TestScenario { Commands = Build | Run | PublishComplex | PublishR2R, Language = VB, NoHttps = False, ScenarioName = &quot;BasicScenarioTests&quot;, Template = Console, ··· })" duration="00:00:21.3905591" startTime="2024-06-19T10:05:28.8068738+00:00" endTime="2024-06-19T10:05:28.8068740+00:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="47a3332a-d644-4113-b695-4a2eebf75013">
<UnitTestResult testName="Microsoft.DotNet.SourceBuild.SmokeTests.WebScenarioTests.VerifyScenario(scenario: TestScenario { Commands = Build | Run | Publish, Language = FSharp, NoHttps = True, ScenarioName = &quot;WebScenarioTests&quot;, Template = Mvc, ··· })" duration="00:00:22.2181871" startTime="2024-06-19T10:05:00.0459230+00:00" endTime="2024-06-19T10:05:00.0459233+00:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="29cc6a20-f65e-4452-8e4a-c08d1ea507d6">
<UnitTestResult testName="Microsoft.DotNet.SourceBuild.SmokeTests.BasicScenarioTests.VerifyScenario(scenario: TestScenario { Commands = Build | Run | PublishComplex | PublishR2R, Language = CSharp, NoHttps = False, ScenarioName = &quot;BasicScenarioTests&quot;, Template = Console, ··· })" duration="00:00:25.9489997" startTime="2024-06-19T10:03:56.1731589+00:00" endTime="2024-06-19T10:03:56.1731590+00:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="c5662788-7c31-4216-86e2-38e6a6f543e5">
<UnitTestResult testName="Microsoft.DotNet.SourceBuild.SmokeTests.BasicScenarioTests.VerifyScenario(scenario: TestScenario { Commands = Build | Run | PublishComplex | PublishR2R, Language = FSharp, NoHttps = False, ScenarioName = &quot;BasicScenarioTests&quot;, Template = Console, ··· })" duration="00:00:28.9336382" startTime="2024-06-19T10:04:43.2055750+00:00" endTime="2024-06-19T10:04:43.2055767+00:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="e2576edb-7c69-47e4-a209-ddf89292519f">
<UnitTestResult testName="Microsoft.DotNet.SourceBuild.SmokeTests.WebScenarioTests.VerifyScenario(scenario: TestScenario { Commands = Build | Run | PublishComplex, Language = CSharp, NoHttps = False, ScenarioName = &quot;WebScenarioTests&quot;, Template = Web, ··· })" duration="00:00:20.4631403" startTime="2024-06-19T10:03:50.6872729+00:00" endTime="2024-06-19T10:03:50.6872731+00:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="2baae30e-0982-4578-8e19-af8951f1f6c3">
<UnitTestResult testName="Microsoft.DotNet.SourceBuild.SmokeTests.WebScenarioTests.VerifyScenario(scenario: TestScenario { Commands = Build | Run | PublishComplex, Language = FSharp, NoHttps = False, ScenarioName = &quot;WebScenarioTests&quot;, Template = Web, ··· })" duration="00:00:24.5573507" startTime="2024-06-19T10:04:37.8276441+00:00" endTime="2024-06-19T10:04:37.8276444+00:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="f152b171-e682-490b-a784-ba441a848dd4">
<UnitTestResult testName="Microsoft.DotNet.SourceBuild.SmokeTests.SourcelinkTests.VerifySourcelinks" duration="00:00:21.7215431" startTime="2024-06-19T10:06:26.0297357+00:00" endTime="2024-06-19T10:06:26.0297363+00:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="17814606-0c01-446d-9adb-62363044d841">

# between
<UnitTestResult testName="Microsoft.DotNet.SourceBuild.SmokeTests.DebugTests.SourceBuiltSdkContainsNativeDebugSymbols" duration="00:00:15.3679113" startTime="2024-06-19T10:03:45.7065459+00:00" endTime="2024-06-19T10:03:45.7065462+00:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Failed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="6826ec33-9799-44ba-9e89-0f4b6ff23872">
<UnitTestResult testName="Microsoft.DotNet.SourceBuild.SmokeTests.WebScenarioTests.VerifyScenario(scenario: TestScenario { Commands = Build | Run | Publish, Language = FSharp, NoHttps = False, ScenarioName = &quot;WebScenarioTests&quot;, Template = WebApi, ··· })" duration="00:00:11.8570648" startTime="2024-06-19T10:05:11.9030679+00:00" endTime="2024-06-19T10:05:11.9030681+00:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="6f055358-e4c7-4954-b5dc-b5235e9a1a2d">
<UnitTestResult testName="Microsoft.DotNet.SourceBuild.SmokeTests.WebScenarioTests.VerifyScenario(scenario: TestScenario { Commands = Build | Run | Publish, Language = CSharp, NoHttps = False, ScenarioName = &quot;WebScenarioTests&quot;, Template = WebApi, ··· })" duration="00:00:08.4119039" startTime="2024-06-19T10:04:13.2701994+00:00" endTime="2024-06-19T10:04:13.2701996+00:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="2a811853-90c2-44db-b056-303fb9b55100">
<UnitTestResult testName="Microsoft.DotNet.SourceBuild.SmokeTests.BasicScenarioTests.VerifyScenario(scenario: TestScenario { Commands = Test, Language = FSharp, NoHttps = False, ScenarioName = &quot;BasicScenarioTests&quot;, Template = NUnit, ··· })" duration="00:00:05.5609766" startTime="2024-06-19T10:05:01.7239349+00:00" endTime="2024-06-19T10:05:01.7239351+00:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="cc0a1d9a-cdcf-4c17-ae10-ad3fc7a656c2">
<UnitTestResult testName="Microsoft.DotNet.SourceBuild.SmokeTests.WebScenarioTests.VerifyScenario(scenario: TestScenario { Commands = Build | Run | Publish, Language = CSharp, NoHttps = False, ScenarioName = &quot;WebScenarioTests&quot;, Template = Razor, ··· })" duration="00:00:16.2020463" startTime="2024-06-19T10:05:28.1052075+00:00" endTime="2024-06-19T10:05:28.1052078+00:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="e645235e-e666-4dbc-8446-39baa135a8b1">
<UnitTestResult testName="Microsoft.DotNet.SourceBuild.SmokeTests.DotNetWatchTests.WatchTests" duration="00:00:14.0497330" startTime="2024-06-19T10:03:44.2732813+00:00" endTime="2024-06-19T10:03:44.2732816+00:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="c2a3af72-9060-450f-a6bb-85b9a308e1f4">
<UnitTestResult testName="Microsoft.DotNet.SourceBuild.SmokeTests.WebScenarioTests.VerifyScenario(scenario: TestScenario { Commands = PublishSelfContained, Language = CSharp, NoHttps = False, ScenarioName = &quot;WebScenarioTests&quot;, Template = WebApp, ··· })" duration="00:00:08.8521001" startTime="2024-06-19T10:06:02.4470209+00:00" endTime="2024-06-19T10:06:02.4470212+00:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="d73115ce-280f-49c2-8ba6-1248b9a53c53">
<UnitTestResult testName="Microsoft.DotNet.SourceBuild.SmokeTests.WebScenarioTests.VerifyScenario(scenario: TestScenario { Commands = Build | Run | Publish, Language = CSharp, NoHttps = True, ScenarioName = &quot;WebScenarioTests&quot;, Template = Mvc, ··· })" duration="00:00:14.1708136" startTime="2024-06-19T10:04:04.8582175+00:00" endTime="2024-06-19T10:04:04.8582177+00:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="c45a7ac2-df02-4614-8fe9-3f82b30f7627">
<UnitTestResult testName="Microsoft.DotNet.SourceBuild.SmokeTests.DotNetFormatTests.FormatProject" duration="00:00:10.6515297" startTime="2024-06-19T10:03:40.8754780+00:00" endTime="2024-06-19T10:03:40.8754782+00:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="bfb0e68d-4150-4553-ac24-45066b7a814a">
<UnitTestResult testName="Microsoft.DotNet.SourceBuild.SmokeTests.BasicScenarioTests.VerifyScenario(scenario: TestScenario { Commands = Build | Publish, Language = FSharp, NoHttps = False, ScenarioName = &quot;BasicScenarioTests&quot;, Template = ClassLib, ··· })" duration="00:00:07.1129489" startTime="2024-06-19T10:04:50.3186349+00:00" endTime="2024-06-19T10:04:50.3186351+00:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="a60c60f0-cd5c-4e30-8349-fc39103c93e4">

# ~ 5 sec
<UnitTestResult testName="Microsoft.DotNet.SourceBuild.SmokeTests.BasicScenarioTests.VerifyScenario(scenario: TestScenario { Commands = Test, Language = FSharp, NoHttps = False, ScenarioName = &quot;BasicScenarioTests&quot;, Template = XUnit, ··· })" duration="00:00:05.8441612" startTime="2024-06-19T10:04:56.1628717+00:00" endTime="2024-06-19T10:04:56.1628719+00:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="403e6870-4855-4a64-8a33-5de30cfb0bdf">
<UnitTestResult testName="Microsoft.DotNet.SourceBuild.SmokeTests.BasicScenarioTests.VerifyScenario(scenario: TestScenario { Commands = Test, Language = CSharp, NoHttps = False, ScenarioName = &quot;BasicScenarioTests&quot;, Template = MSTest, ··· })" duration="00:00:04.2246271" startTime="2024-06-19T10:04:14.2718651+00:00" endTime="2024-06-19T10:04:14.2718653+00:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="a959a688-d3e0-4252-a33a-31fa61688d9b">
<UnitTestResult testName="Microsoft.DotNet.SourceBuild.SmokeTests.BasicScenarioTests.VerifyScenario(scenario: TestScenario { Commands = Build | Publish, Language = CSharp, NoHttps = False, ScenarioName = &quot;BasicScenarioTests&quot;, Template = ClassLib, ··· })" duration="00:00:04.5600032" startTime="2024-06-19T10:04:00.7332913+00:00" endTime="2024-06-19T10:04:00.7332915+00:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="3e5ceaca-7e9d-4db8-9497-a0d77f9569ae">
<UnitTestResult testName="Microsoft.DotNet.SourceBuild.SmokeTests.BasicScenarioTests.VerifyScenario(scenario: TestScenario { Commands = Test, Language = VB, NoHttps = False, ScenarioName = &quot;BasicScenarioTests&quot;, Template = MSTest, ··· })" duration="00:00:04.4162606" startTime="2024-06-19T10:05:46.7423195+00:00" endTime="2024-06-19T10:05:46.7423198+00:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="6a6adfc2-081e-4d15-85fd-57989de5b8bf">
<UnitTestResult testName="Microsoft.DotNet.SourceBuild.SmokeTests.BasicScenarioTests.VerifyScenario(scenario: TestScenario { Commands = Test, Language = VB, NoHttps = False, ScenarioName = &quot;BasicScenarioTests&quot;, Template = NUnit, ··· })" duration="00:00:04.5488726" startTime="2024-06-19T10:05:42.3259913+00:00" endTime="2024-06-19T10:05:42.3259915+00:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="0d631c7c-457e-48ef-b569-c2c97f22fc26">
<UnitTestResult testName="Microsoft.DotNet.SourceBuild.SmokeTests.BasicScenarioTests.VerifyScenario(scenario: TestScenario { Commands = Build | Publish, Language = VB, NoHttps = False, ScenarioName = &quot;BasicScenarioTests&quot;, Template = ClassLib, ··· })" duration="00:00:04.4917181" startTime="2024-06-19T10:05:33.2986808+00:00" endTime="2024-06-19T10:05:33.2986811+00:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="3d22ea1e-87b3-40c0-bb3b-3d34ee2dbd9c">
<UnitTestResult testName="Microsoft.DotNet.SourceBuild.SmokeTests.WebScenarioTests.VerifyScenario(scenario: TestScenario { Commands = None, Language = CSharp, NoHttps = False, ScenarioName = &quot;WebScenarioTests&quot;, Template = Worker, ··· })" duration="00:00:01.8600194" startTime="2024-06-19T10:06:04.3071510+00:00" endTime="2024-06-19T10:06:04.3071524+00:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="41c05e36-dbd9-4296-a1ec-ca85c245f097">
<UnitTestResult testName="Microsoft.DotNet.SourceBuild.SmokeTests.BasicScenarioTests.VerifyScenario(scenario: TestScenario { Commands = Test, Language = VB, NoHttps = False, ScenarioName = &quot;BasicScenarioTests&quot;, Template = XUnit, ··· })" duration="00:00:04.4782523" startTime="2024-06-19T10:05:37.7770524+00:00" endTime="2024-06-19T10:05:37.7770527+00:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="1056428a-443e-417b-a70a-ad378ed49585">
<UnitTestResult testName="Microsoft.DotNet.SourceBuild.SmokeTests.BasicScenarioTests.VerifyScenario(scenario: TestScenario { Commands = Test, Language = FSharp, NoHttps = False, ScenarioName = &quot;BasicScenarioTests&quot;, Template = MSTest, ··· })" duration="00:00:05.6920531" startTime="2024-06-19T10:05:07.4160633+00:00" endTime="2024-06-19T10:05:07.4160635+00:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="12b2184a-997e-4597-bf87-8ca57323b888">
<UnitTestResult testName="Microsoft.DotNet.SourceBuild.SmokeTests.BasicScenarioTests.VerifyScenario(scenario: TestScenario { Commands = Test, Language = CSharp, NoHttps = False, ScenarioName = &quot;BasicScenarioTests&quot;, Template = NUnit, ··· })" duration="00:00:04.4028153" startTime="2024-06-19T10:04:10.0471323+00:00" endTime="2024-06-19T10:04:10.0471326+00:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="4e3448d0-536f-4669-bff5-deeb2bbe656c">
<UnitTestResult testName="Microsoft.DotNet.SourceBuild.SmokeTests.BasicScenarioTests.VerifyScenario(scenario: TestScenario { Commands = Test, Language = CSharp, NoHttps = False, ScenarioName = &quot;BasicScenarioTests&quot;, Template = XUnit, ··· })" duration="00:00:04.9108043" startTime="2024-06-19T10:04:05.6441954+00:00" endTime="2024-06-19T10:04:05.6441960+00:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="9ee99710-578e-4c58-89ec-967c52f92efe">

# damn fast
<UnitTestResult testName="Microsoft.DotNet.SourceBuild.SmokeTests.SymbolsTests.VerifySdkSymbols" duration="00:00:00.7465457" startTime="2024-06-19T10:03:30.9727610+00:00" endTime="2024-06-19T10:03:30.9727612+00:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="f2fe75d1-4a1f-4225-bf46-2903954f4a87">
<UnitTestResult testName="Microsoft.DotNet.SourceBuild.SmokeTests.SourceBuiltArtifactsTests.VerifyVersionFile" duration="00:00:01.9214145" startTime="2024-06-19T10:03:32.1448484+00:00" endTime="2024-06-19T10:03:32.1448486+00:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="99fd6e74-8f40-44ab-b1c1-2e75a760f42c">

ViktorHofer · 2024-06-19T11:44:44Z

I see three PublishR2R tests in there. Could this be related?

tmds · 2024-06-19T11:49:59Z

I see three PublishR2R tests in there. Could this be related?

I don't expect that changes in this PR could/would make R2R slower.

I did another local test run, and now it shows a failure that is related to this PR:

[xUnit.net 00:00:19.93]     Microsoft.DotNet.SourceBuild.SmokeTests.DebugTests.SourceBuiltSdkContainsNativeDebugSymbols [FAIL]
[xUnit.net 00:00:19.93]       missing .debug_info section in /home/tmds/repos/dotnet/artifacts/bin/Microsoft.DotNet.SourceBuild.SmokeTests/Release/extracted-sdk/packs/runtime.fedora.39-x64.Microsoft.DotNet.ILCompiler/9.0.0-preview.6.24307.2/tools/ilc
[xUnit.net 00:00:19.93]       missing .debug_abbrev section in /home/tmds/repos/dotnet/artifacts/bin/Microsoft.DotNet.SourceBuild.SmokeTests/Release/extracted-sdk/packs/runtime.fedora.39-x64.Microsoft.DotNet.ILCompiler/9.0.0-preview.6.24307.2/tools/ilc
[xUnit.net 00:00:19.93]       missing FILE symbols in /home/tmds/repos/dotnet/artifacts/bin/Microsoft.DotNet.SourceBuild.SmokeTests/Release/extracted-sdk/packs/runtime.fedora.39-x64.Microsoft.DotNet.ILCompiler/9.0.0-preview.6.24307.2/tools/ilc
[xUnit.net 00:00:19.93]       
[xUnit.net 00:00:19.93]       Stack Trace:
[xUnit.net 00:00:19.93]         /home/tmds/repos/dotnet/test/Microsoft.DotNet.SourceBuild.SmokeTests/DebugTests.cs(64,0): at Microsoft.DotNet.SourceBuild.SmokeTests.DebugTests.SourceBuiltSdkContainsNativeDebugSymbols()
[xUnit.net 00:00:19.93]         
[xUnit.net 00:00:19.93]            at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
[xUnit.net 00:00:19.93]            at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)

I don't know why it didn't show up on my earlier test run.

afaik ilc is binary produced by NativeAOT. It doesn't seem to meet the expectations set by SourceBuiltSdkContainsNativeDebugSymbols.

I'm going to await the results from the CI run before making further changes.

ViktorHofer · 2024-06-19T11:52:09Z

microsoft/vstest#5091 might also be related somehow

I did another local test run, and now it shows a failure that is related to this PR:

Yes I see those on my box as well.

ViktorHofer · 2024-06-19T13:39:20Z

Interesting. Setting MSBUILDENSURESTDOUTFORTASKPROCESSES=1 really did help.

tmds · 2024-06-19T14:04:37Z

Setting MSBUILDENSURESTDOUTFORTASKPROCESSES=1 really did help.

Yes. There is useful output now from the test run.

And, the CI run finished in less than 4 minutes. So the timeout at 30min may well be microsoft/vstest#5091.

I'm going to change the timeout back to 30min, and skip ilc for the SourceBuiltSdkContainsNativeDebugSymbols test ( + create an issue for that when this PR is merged).

@ViktorHofer can you take a look at where I added MSBUILDENSURESTDOUTFORTASKPROCESSES=1 and see what may be a better place?

ViktorHofer · 2024-06-19T14:34:49Z

Yes, please add the export here for bash:

sdk/src/SourceBuild/content/build.sh

Line 222 in 00becb5

fi

and here for Windows:

sdk/src/SourceBuild/content/eng/build.ps1

Line 63 in 00becb5

}

Please also add a TODO marker and link to the vstest issue. This really sounds like a product bug.

ViktorHofer

Wow, that was a challenging experience. Thanks for fighting yourself through the hurdles.

ViktorHofer · 2024-06-19T17:03:32Z

@tmds this is in :) As I didn't see any tests in this PR, should we add some that verify that ILC now works in offline mode? I assume dotnet/scenario-tests would be a good place? cc @MichaelSimons

MichaelSimons · 2024-06-19T18:17:22Z

I assume dotnet/scenario-tests would be a good place?

Yes that would be a good place for an end to end test

tmds · 2024-06-20T04:32:42Z

@ViktorHofer yes, it is in! 🚀 Thanks for your help!

For test, I was going to add something to Microsoft.DotNet.SourceBuild.SmokeTests, like a test for a new PublishAot DotNetAction. Would that make a good test?

I didn't know about https://github.com/dotnet/scenario-tests. How does it tie into the CI pipeline?

ViktorHofer · 2024-06-20T08:55:38Z

For test, I was going to add something to Microsoft.DotNet.SourceBuild.SmokeTests, like a test for a new PublishAot DotNetAction. Would that make a good test?

scenario-tests from the scenario-tests repo are run as part of VMR's test action. See https://github.com/dotnet/dotnet/blob/2f26df4d144fb0687c9d657c3c1b93a6fd11dc97/test/tests.proj#L31

Related: dotnet/source-build#4384

mthalman · 2024-06-20T13:17:26Z

scenario-tests would be preferred if the test could be equally applicable to a Microsoft-built SDK. The tests in Microsoft.DotNet.SourceBuild.SmokeTests are meant for validation specific to source-build.

tmds · 2024-06-20T13:22:40Z

if the test could be equally applicable to a Microsoft-built SDK.

Both SDKs can publish NativeAOT.

The test should validate the source-built SDK can publish with the bundled ILCompiler and packages without needing additional packages from a NuGet source.

This is different from how the Microsoft SDK does NativeAOT (which requires having a NuGet source for these packages).

Does this mean the test should be in Microsoft.DotNet.SourceBuild.SmokeTests?

mthalman · 2024-06-20T13:31:52Z

Does this mean the test should be in Microsoft.DotNet.SourceBuild.SmokeTests?

Yes, in that case it sounds like the right location for such a test.

Conceivably, scenario-tests could also host a general NativeAOT publishing test but without the validation of how it gets additional packages (but that's outside the scope of what you're doing). Ah, and I see there's an existing PR for that: dotnet/scenario-tests#64

MichaelSimons · 2024-06-20T13:36:21Z

What do you think of adding conditioned source-build validation to the general NativeAOT publishing scenario tests? This would keep the tests in one location and avoid duplication.

dotnet-issue-labeler bot added Area-Infrastructure untriaged Request triage from a team member labels May 28, 2024

tmds commented May 28, 2024

View reviewed changes

src/Installer/redist-installer/targets/GenerateLayout.targets Show resolved Hide resolved

tmds mentioned this pull request May 31, 2024

Support building sdks that can build self-contained / ReadyToRun / NativeAOT applications without using pre-builts from nuget.org dotnet/source-build#1215

Closed

dsplaisted approved these changes Jun 6, 2024

View reviewed changes

ViktorHofer reviewed Jun 18, 2024

View reviewed changes

src/Installer/redist-installer/targets/GenerateLayout.targets Show resolved Hide resolved

tmds force-pushed the sb_bundle_aot branch from 7fe5ee2 to ec9a3f1 Compare June 18, 2024 12:40

Add DotNetBuildOrchestrator condition for setting BundleNativeAotComp…

3d1c224

…iler.

am11 reviewed Jun 18, 2024

View reviewed changes

tmds closed this Jun 19, 2024

tmds reopened this Jun 19, 2024

build-analysis bot mentioned this pull request Jun 19, 2024

VMR CI build failures during restore: feed download failed #41590

Open

Find out what happens when doubling the test timeout.

713a6ff

Try to get some more output from the CI test run.

cd188f3

build-analysis bot mentioned this pull request Jun 19, 2024

The "RestoreTask" task returned false but did not log an error dotnet/dnceng#3100

Open

3 tasks

tmds added 2 commits June 19, 2024 16:14

Skip 'ilc' in SourceBuiltSdkContainsNativeDebugSymbols.

71f5d98

Revert runTestsTimeout change.

7be0c03

Set MSBUILDENSURESTDOUTFORTASKPROCESSES in scripts when running tests.

22a593b

ViktorHofer approved these changes Jun 19, 2024

View reviewed changes

ViktorHofer merged commit f3ebfb5 into dotnet:main Jun 19, 2024
36 checks passed

am11 mentioned this pull request Jun 21, 2024

Portable RID PublishAOT doesn't work from source-build VMR .NET 9 Preview 5 #41727

Closed

ellahathaway mentioned this pull request Jun 21, 2024

Update Source-Build SDK Diff Tests Baselines and Exclusions #41716

Merged

source-build: bundle NativeAOT libraries with the SDK. #41198

source-build: bundle NativeAOT libraries with the SDK. #41198

Conversation

tmds commented May 28, 2024

tmds May 28, 2024

Choose a reason for hiding this comment

am11 May 28, 2024

Choose a reason for hiding this comment

tmds May 28, 2024

Choose a reason for hiding this comment

ViktorHofer Jun 18, 2024 • edited Loading

Choose a reason for hiding this comment

ViktorHofer Jun 18, 2024

Choose a reason for hiding this comment

tmds Jun 18, 2024

Choose a reason for hiding this comment

tmds commented Jun 5, 2024

dsplaisted left a comment

Choose a reason for hiding this comment

dsplaisted commented Jun 6, 2024

baronfel commented Jun 17, 2024

azure-pipelines bot commented Jun 17, 2024

tmds commented Jun 18, 2024 • edited Loading

ViktorHofer commented Jun 18, 2024

mthalman commented Jun 18, 2024

am11 Jun 18, 2024

Choose a reason for hiding this comment

tmds Jun 18, 2024

Choose a reason for hiding this comment

tmds commented Jun 18, 2024

tmds commented Jun 19, 2024

am11 commented Jun 19, 2024

ViktorHofer commented Jun 19, 2024 • edited Loading

ViktorHofer commented Jun 19, 2024

tmds commented Jun 19, 2024

ViktorHofer commented Jun 19, 2024

tmds commented Jun 19, 2024

tmds commented Jun 19, 2024

ViktorHofer commented Jun 19, 2024

tmds commented Jun 19, 2024

ViktorHofer commented Jun 19, 2024

ViktorHofer commented Jun 19, 2024

tmds commented Jun 19, 2024

ViktorHofer commented Jun 19, 2024

ViktorHofer commented Jun 19, 2024

tmds commented Jun 19, 2024

ViktorHofer commented Jun 19, 2024

ViktorHofer left a comment • edited Loading

Choose a reason for hiding this comment

ViktorHofer commented Jun 19, 2024

MichaelSimons commented Jun 19, 2024

tmds commented Jun 20, 2024

ViktorHofer commented Jun 20, 2024 • edited Loading

mthalman commented Jun 20, 2024

tmds commented Jun 20, 2024

mthalman commented Jun 20, 2024 • edited Loading

MichaelSimons commented Jun 20, 2024

ViktorHofer Jun 18, 2024 •

edited

Loading

tmds commented Jun 18, 2024 •

edited

Loading

ViktorHofer commented Jun 19, 2024 •

edited

Loading

ViktorHofer left a comment •

edited

Loading

ViktorHofer commented Jun 20, 2024 •

edited

Loading

mthalman commented Jun 20, 2024 •

edited

Loading