Skip to content

Commit bb4db54

Browse files
Update common Docker engineering infrastructure with latest
1 parent 87bbc76 commit bb4db54

8 files changed

Lines changed: 237 additions & 19 deletions

File tree

eng/docker-tools/CHANGELOG.md

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,35 @@ All breaking changes and new features in `eng/docker-tools` will be documented i
44

55
---
66

7+
## 2026-04-02: Extra Docker build options can be passed through ImageBuilder
8+
9+
- Pull request: [#2063](https://github.com/dotnet/docker-tools/pull/2063)
10+
11+
ImageBuilder's `build` command now accepts repeated `--build-option` arguments and forwards them directly to
12+
`docker build`. This allows repos to pass options such as `--ulimit nofile=65536:65536` or `--network host`
13+
through `imageBuilderBuildArgs`, in addition to standard Dockerfile `--build-arg` values.
14+
15+
**How to use:**
16+
17+
```yaml
18+
customBuildInitSteps:
19+
- powershell: |
20+
$args = '--build-option "--ulimit nofile=65536:65536"'
21+
echo "##vso[task.setvariable variable=imageBuilderBuildArgs]$args"
22+
```
23+
24+
Repeat `--build-option` for multiple Docker arguments, and quote values that contain spaces.
25+
26+
---
27+
28+
## 2026-03-25: Manifest list creation moved to Post_Build
29+
30+
- Issue: [#2002](https://github.com/dotnet/docker-tools/issues/2002)
31+
32+
Manifest lists are now created during `Post_Build` instead of during `Publish`. They are copied to the publish registry via ACR import along with the platform images, rather than being recreated from scratch during publish.
33+
34+
---
35+
736
## 2026-03-18: CG build template supports skipping .NET SDK installation
837

938
- Issue: [#2029](https://github.com/dotnet/docker-tools/issues/2029)

eng/docker-tools/DEV-GUIDE.md

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -370,7 +370,7 @@ To force a rebuild regardless of cache state, set the `noCache` parameter to `tr
370370

371371
### Pattern: Adding Build Arguments
372372

373-
Pass additional arguments to Docker builds via ImageBuilder:
373+
Pass Dockerfile `ARG` values via ImageBuilder:
374374

375375
```yaml
376376
customBuildInitSteps:
@@ -379,6 +379,15 @@ customBuildInitSteps:
379379
echo "##vso[task.setvariable variable=imageBuilderBuildArgs]$args"
380380
```
381381

382+
To pass raw options directly to `docker build`, use `--build-option`. Quote values that contain spaces:
383+
384+
```yaml
385+
customBuildInitSteps:
386+
- powershell: |
387+
$args = '--build-option "--ulimit nofile=65536:65536"'
388+
echo "##vso[task.setvariable variable=imageBuilderBuildArgs]$args"
389+
```
390+
382391
### Pattern: Re-running Stages with `stages` and `sourceBuildPipelineRunId`
383392

384393
A powerful pattern is combining the `stages` variable with the `sourceBuildPipelineRunId` pipeline parameter to run specific stages using artifacts from a previous build. This is useful for:
Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
#!/usr/bin/env pwsh
2+
# Lightweight wrapper for authenticated Azure DevOps REST API calls.
3+
# Uses `az account get-access-token` for bearer token auth.
4+
#
5+
# Usage:
6+
# . ./AzureDevOps.ps1
7+
# $response = Invoke-AzDORestMethod -Organization myorg -Project myproject `
8+
# -Endpoint "pipelines/42/runs" -Method POST -Body @{ resources = @{} }
9+
10+
$ErrorActionPreference = "Stop"
11+
12+
function Get-AzDOAccessToken {
13+
<#
14+
.SYNOPSIS
15+
Returns a bearer token for Azure DevOps.
16+
#>
17+
18+
# Well-known Entra ID application ID for Azure DevOps
19+
$tokenJson = az account get-access-token --resource "499b84ac-1321-427f-aa17-267ca6975798" 2>&1
20+
if ($LASTEXITCODE -ne 0) {
21+
throw "Failed to get access token. Run 'az login' first. Output: $tokenJson"
22+
}
23+
24+
$parsed = $tokenJson | ConvertFrom-Json
25+
return $parsed.accessToken
26+
}
27+
28+
function Invoke-AzDORestMethod {
29+
<#
30+
.SYNOPSIS
31+
Calls an Azure DevOps REST API endpoint with automatic authentication.
32+
.PARAMETER Organization
33+
Azure DevOps organization name (not the full URL).
34+
.PARAMETER Project
35+
Azure DevOps project name.
36+
.PARAMETER Endpoint
37+
API path after _apis/ (e.g. "pipelines/42/runs", "build/builds").
38+
.PARAMETER Method
39+
HTTP method. Defaults to GET.
40+
.PARAMETER Body
41+
Request body as a hashtable. Automatically converted to JSON.
42+
.PARAMETER ApiVersion
43+
API version. Defaults to 7.1.
44+
#>
45+
[CmdletBinding()]
46+
param(
47+
[Parameter(Mandatory)][string] $Organization,
48+
[Parameter(Mandatory)][string] $Project,
49+
[Parameter(Mandatory)][string] $Endpoint,
50+
[string] $Method = "GET",
51+
[hashtable] $Body,
52+
[string] $ApiVersion = "7.1"
53+
)
54+
55+
$token = Get-AzDOAccessToken
56+
$headers = @{
57+
Authorization = "Bearer $token"
58+
"Content-Type" = "application/json"
59+
}
60+
61+
$uri = "https://dev.azure.com/$Organization/$Project/_apis/$($Endpoint)?api-version=$ApiVersion"
62+
63+
$params = @{
64+
Uri = $uri
65+
Headers = $headers
66+
Method = $Method
67+
}
68+
69+
if ($Body) {
70+
$params.Body = $Body | ConvertTo-Json -Depth 10
71+
}
72+
73+
return Invoke-RestMethod @params
74+
}
Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
#!/usr/bin/env pwsh
2+
# Retrieves a build log by log ID.
3+
# Usage:
4+
# ./Get-BuildLog.ps1 -Organization dnceng -Project internal -BuildId 12345 -LogId 47
5+
6+
[CmdletBinding()]
7+
param(
8+
[Parameter(Mandatory)][string] $Organization,
9+
[Parameter(Mandatory)][string] $Project,
10+
[Parameter(Mandatory)][int] $BuildId,
11+
[Parameter(Mandatory)][int] $LogId
12+
)
13+
14+
$ErrorActionPreference = "Stop"
15+
16+
. "$PSScriptRoot/AzureDevOps.ps1"
17+
18+
Invoke-AzDORestMethod `
19+
-Organization $Organization `
20+
-Project $Project `
21+
-Endpoint "build/builds/$BuildId/logs/$LogId"
Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
#!/usr/bin/env pwsh
2+
# Prints the build timeline as an indented tree with result indicators.
3+
# Usage:
4+
# ./Show-BuildTimeline.ps1 -Organization dnceng -Project internal -BuildId 12345
5+
# ./Show-BuildTimeline.ps1 -Organization dnceng -Project internal -BuildId 12345 -ShowAllTasks
6+
7+
[CmdletBinding()]
8+
param(
9+
[Parameter(Mandatory)][string] $Organization,
10+
[Parameter(Mandatory)][string] $Project,
11+
[Parameter(Mandatory)][int] $BuildId,
12+
[switch] $ShowAllTasks
13+
)
14+
15+
$ErrorActionPreference = "Stop"
16+
17+
. "$PSScriptRoot/AzureDevOps.ps1"
18+
19+
$build = Invoke-AzDORestMethod `
20+
-Organization $Organization `
21+
-Project $Project `
22+
-Endpoint "build/builds/$BuildId"
23+
24+
Write-Host "# Build $BuildId - $($build.definition.name)"
25+
Write-Host ""
26+
Write-Host "- Status: $($build.status) $(if ($build.result) { "($($build.result))" })"
27+
Write-Host "- Branch: $($build.sourceBranch)"
28+
Write-Host "- Queued: $($build.queueTime)"
29+
Write-Host "- URL: $($build._links.web.href)"
30+
Write-Host ""
31+
32+
$timeline = Invoke-AzDORestMethod `
33+
-Organization $Organization `
34+
-Project $Project `
35+
-Endpoint "build/builds/$BuildId/timeline"
36+
37+
$records = $timeline.records
38+
39+
# Build a lookup of children grouped by parentId
40+
$childrenOf = @{}
41+
foreach ($record in $records) {
42+
$parentId = $record.parentId
43+
if (-not $parentId) { $parentId = "" }
44+
if (-not $childrenOf.ContainsKey($parentId)) {
45+
$childrenOf[$parentId] = [System.Collections.Generic.List[object]]::new()
46+
}
47+
$childrenOf[$parentId].Add($record)
48+
}
49+
50+
# Sort children by order within each group
51+
foreach ($key in @($childrenOf.Keys)) {
52+
$childrenOf[$key] = $childrenOf[$key] | Sort-Object { $_.order }
53+
}
54+
55+
function Write-TimelineNode([string] $nodeId, [int] $depth) {
56+
$children = $childrenOf[$nodeId]
57+
if (-not $children) { return }
58+
59+
foreach ($child in $children) {
60+
$isTask = $child.type -eq "Task"
61+
$isFailing = $child.result -in @("failed", "canceled", "abandoned") -or $child.state -eq "inProgress"
62+
if ($isTask -and -not $ShowAllTasks -and -not $isFailing) { continue }
63+
64+
$indent = " " * $depth
65+
$status = if ($child.result) { $child.result } else { $child.state }
66+
67+
$logId = $child.log.id
68+
$logLabel = if ($logId) { " #$logId" } else { "" }
69+
Write-Host "${indent}- $($child.type)$logLabel | $($child.name) | $status"
70+
Write-TimelineNode $child.id ($depth + 1)
71+
}
72+
}
73+
74+
Write-Host "## Build Timeline"
75+
Write-Host ""
76+
Write-TimelineNode "" 0
77+
Write-Host ""

eng/docker-tools/templates/jobs/post-build.yml

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,12 @@ jobs:
2020
dockerClientOS: linux
2121
customInitSteps: ${{ parameters.customInitSteps }}
2222
publishConfig: ${{ parameters.publishConfig }}
23+
- ${{ if parameters.publishConfig }}:
24+
- template: /eng/docker-tools/templates/steps/reference-service-connections.yml@self
25+
parameters:
26+
publishConfig: ${{ parameters.publishConfig }}
27+
usesRegistries:
28+
- ${{ parameters.publishConfig.BuildRegistry.server }}
2329
- template: /eng/docker-tools/templates/steps/download-build-artifact.yml@self
2430
parameters:
2531
targetPath: $(Build.ArtifactStagingDirectory)
@@ -75,6 +81,21 @@ jobs:
7581
$(manifestVariables)
7682
name: MergeImageInfoFiles
7783
displayName: Merge Image Info Files
84+
- ${{ if parameters.publishConfig }}:
85+
- template: /eng/docker-tools/templates/steps/run-imagebuilder.yml@self
86+
parameters:
87+
displayName: Create Manifest Lists
88+
internalProjectName: ${{ parameters.internalProjectName }}
89+
condition: and(succeeded(), ne(variables['MergeImageInfoFiles.noImageInfos'], 'true'), ne(variables['Build.Reason'], 'PullRequest'))
90+
args: >-
91+
createManifestList
92+
'$(imageInfosContainerDir)$(imageInfosOutputSubDir)/image-info.json'
93+
--repo-prefix '${{ parameters.publishConfig.BuildRegistry.repoPrefix }}'
94+
--os-type '*'
95+
--architecture '*'
96+
--manifest '$(manifest)'
97+
--registry-override '${{ parameters.publishConfig.BuildRegistry.server }}'
98+
$(manifestVariables)
7899
- template: /eng/docker-tools/templates/steps/publish-artifact.yml@self
79100
parameters:
80101
condition: and(succeeded(), ne(variables['MergeImageInfoFiles.noImageInfos'], 'true'))

eng/docker-tools/templates/jobs/publish.yml

Lines changed: 4 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -118,21 +118,6 @@ jobs:
118118
$(imageBuilder.pathArgs)
119119
$(imageBuilder.commonCmdArgs)
120120
121-
- template: /eng/docker-tools/templates/steps/run-imagebuilder.yml@self
122-
parameters:
123-
displayName: Publish Manifest
124-
internalProjectName: ${{ parameters.internalProjectName }}
125-
dockerClientOS: ${{ parameters.dockerClientOS }}
126-
args: >-
127-
publishManifest
128-
'$(imageInfoContainerDir)/image-info.json'
129-
--repo-prefix '${{ parameters.publishConfig.PublishRegistry.repoPrefix }}'
130-
--os-type '*'
131-
--architecture '*'
132-
$(dryRunArg)
133-
$(imageBuilder.pathArgs)
134-
$(imageBuilder.commonCmdArgs)
135-
136121
- template: /eng/docker-tools/templates/steps/publish-artifact.yml@self
137122
parameters:
138123
path: $(imageInfoHostDir)
@@ -256,6 +241,9 @@ jobs:
256241
#
257242
# https://github.com/dotnet/docker-tools/issues/1698 tracks making this command no longer depend
258243
# on individual step displayNames.
244+
#
245+
# Skipped for PR builds because manifest lists are not created in PR builds (see post-build.yml).
246+
# Without manifest lists present in image-info.json, the postPublishNotification fails with a NRE.
259247
- script: >
260248
$(runImageBuilderCmd) postPublishNotification
261249
'$(publishNotificationRepoName)'
@@ -270,7 +258,6 @@ jobs:
270258
'$(gitHubNotificationsRepoInfo.repo)'
271259
--repo-prefix '${{ parameters.publishConfig.PublishRegistry.repoPrefix }}'
272260
--task "🟪 Copy Images"
273-
--task "🟪 Publish Manifest"
274261
--task "🟪 Wait for Image Ingestion"
275262
--task "🟪 Publish Readmes"
276263
--task "🟪 Wait for MCR Doc Ingestion"
@@ -282,7 +269,7 @@ jobs:
282269
$(dryRunArg)
283270
$(imageBuilder.commonCmdArgs)
284271
displayName: Post Publish Notification
285-
condition: and(always(), eq(variables['publishNotificationsEnabled'], 'true'))
272+
condition: and(always(), eq(variables['publishNotificationsEnabled'], 'true'), ne(variables['Build.Reason'], 'PullRequest'))
286273
287274
- powershell: |
288275
# Default to current build number if parameter was not overridden

eng/docker-tools/templates/variables/docker-images.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
variables:
2-
imageNames.imageBuilderName: mcr.microsoft.com/dotnet-buildtools/image-builder:2928776
2+
imageNames.imageBuilderName: mcr.microsoft.com/dotnet-buildtools/image-builder:2941471
33
imageNames.imageBuilder: $(imageNames.imageBuilderName)
44
imageNames.imageBuilder.withrepo: imagebuilder-withrepo:$(Build.BuildId)-$(System.JobId)
55
imageNames.testRunner: mcr.microsoft.com/dotnet-buildtools/prereqs:azurelinux3.0-docker-testrunner

0 commit comments

Comments
 (0)