#763 wiki update for removal step (#833)

* #763 resource removal documentation

* Minor fixes

* helper folder

Co-authored-by: Alexander Sehr <ASehr@hotmail.de>

* Wave 1 comments integration

* Add custom removal action

* Move removal to pipeline section

* Level 1 titles

* Include subpages links

* remove extension from link

Co-authored-by: Alexander Sehr <ASehr@hotmail.de>

* Point to dedicated page for details

Co-authored-by: Erika Gressi <56914614+eriqua@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Erika Gressi <56914614+eriqua@users.noreply.github.com>

* Remove duplicate content

Co-authored-by: Simone Bertaccini <simonb@microsoft.com>
Co-authored-by: Alexander Sehr <ASehr@hotmail.de>
Co-authored-by: Erika Gressi <56914614+eriqua@users.noreply.github.com>

Author: 4 people

docs/wiki/Home.md

@@ -24,6 +24,8 @@ If you're unfamiliar with Infrastructure as Code, or wonder how you can use the
   - [Bicep to ARM conversion script](./UtilitiesConversionScript)
 - [Pipelines](./Pipelines)
   - [Design](./PipelinesDesign)
+    - [Parameter File Tokens](./ParameterFileTokens)
+    - [Removal action](./PipelineRemovalAction)
   - [Usage](./PipelinesUsage)
 - [Contribution Guide](./ContributionGuide)
 - [Known Issues](./KnownIssues)

docs/wiki/ModulesDesign.md

@@ -499,9 +499,6 @@ While exceptions might be needed, the following guidance should be followed as m
   > name: '${deployment().name}-Table-${index}'
   > ```
 
-
-
-
 ## Outputs
 
 - Output names are in camelCase, i.e `storageAccountResourceId`

docs/wiki/PipelineRemovalAction.md

@@ -0,0 +1,70 @@
+# Removal action
+
+This section describes how the removal of resources deployed by a module is performed and how to modify the default behaviour if a specific module or resource type needs it.
+
+---
+
+### _Navigation_
+
+- [Overview](#Overview)
+- [How it works](#how-it-works)
+- [Create a specialized removal procedure](#create-a-specialized-removal-procedure)
+
+---
+
+# Overview
+
+The Removal action is triggered after the deployment completes. This is used for several reasons:
+- Make sure to keep the validation subscription cost as low as possible.
+- Enable testing of the full module deployment at every run.
+
+The default removal procedure works fine for most of the modules created so far, so it's likely you won't have to change anything to make the module you're editing to be removed correctly after deployment.
+
+# How it works
+
+The removal process will remove all resources created during deployment. The list is identified by:
+
+1. Recursively fetching the list of resource IDs created through your deployment (resources created by deployments created by the parent one will be fetched too)
+1. Ordering the list based on resource IDs segment count (ensures child resources are removed first. E.g. `storageAccount/blobServices` comes before `storageAccount` as it has one more segments delimited by `/`)
+1. Filtering out from the list any resource used as dependencies for different modules (e.g. the commonly used Log Analytics workspace)
+1. Moving specific resource types to the top of the list (if a certain order is required). For example `vWAN` requires its `Virtual Hubs` to be removed first, even though they are no child-resources.
+
+After a resource is removed (this happens after each resource in the list), the script will execute, if defined, a **post removal operation**. This can be used for those resource types that requires a post processing, like purging a soft-deleted key vault.
+
+The procedure is initiated by the script `/utilities/pipelines/resourceRemoval/Initialize-DeploymentRemoval.ps1`, run during deployment by:
+- (Azure DevOps) `/.azuredevops/pipelineTemplates/module.jobs.deploy.yml`
+- (GitHub) `/.github/actions/templates/validateModuleDeployment/action.yml`
+
+It uses several helper scripts that can be found in the `/utilities/pipelines/resourceRemoval/helper` folder
+# Create a specialized removal procedure
+
+You can define a custom removal procedure by:
+1. influencing the **order** in which resources are removed by prioritizing specific resource types 
+    > **Example** Removing a _Virtual WAN_ resource requires related resources to be deleted in a specific order
+1. defining a **custom removal action** to remove a resource of a _specific resource type_ 
+    > **Example** A _Recovery Services Vault_ resource requires some protected items to be identified and removed beforehand
+1. defining a custom **post-removal action** to be run after removing a resource of a _specific resource type_
+    > **Example** A _Key Vault_ resource needs to be purged when soft deletion is enforced
+
+Those methods can be combined independently.
+
+> **Important**: _custom_ and _post-removal_ actions will be executed when a resource of the type you specify is removed **regardless** of which deployment triggered the deployment. Make sure you do not assume the resource is in a particular state defined by your module.
+
+To modify the resource types removal **order**:
+1. Open the `/utilities/pipelines/resourceRemoval/Initialize-DeploymentRemoval.ps1` file.
+1. Look for the following comment: `### CODE LOCATION: Add custom removal sequence here`
+1. Add a case value that matches your module name
+1. In the case block, update the `$removalSequence` variable value to accommodate your module requirements
+1. Remember to add the `break` statement.
+
+To defina a **custom removal** action:
+1. Open the `/utilities/pipelines/resourceRemoval/helper/Invoke-ResourceRemoval.ps1` file.
+1. Look for the following comment: `### CODE LOCATION: Add custom removal action here`
+1. Add a case value that matches the resource type you want to modify the removal action for
+1. In the case block, define the resource-type-specific removal action
+
+To add a **post-removal** step:
+1. Open the `/utilities/pipelines/resourceRemoval/helper/Invoke-ResourcePostRemoval.ps1` file.
+1. Look for the following comment: `### CODE LOCATION: Add custom post-removal operation here`
+1. Add a case value that matches the resource type you want to add a post-removal operation for
+1. In the case block, define the resource-type-specific post removal action

docs/wiki/PipelinesDesign.md

@@ -93,9 +93,9 @@ Note that, for the deployments we have to account for certain [prerequisites](#p
 
 #### Removal
 
-The removal phase is strongly coupled with the previous deployment phase. Fundamentally, we want to remove any test-deployed resource after its test concluded. If we would not, we would generate unnecessary costs and may temper with any subsequent test. Some resources may require a dedicated logic to be removed. This logic should be stored alongside the generally utilized removal script in the `.utilities/pipelines/resourceRemoval` folder and be referenced by the `Initialize-DeploymentRemoval.ps1` script that orchestrates the removal.
+The removal phase takes care of removing all resources deployed as part of the previous deployment phase. The reason is twofold: keeping validation subscriptions costs down and allow deployments from scratch at every run.
 
-Most of the removal scripts rely on the deployment name used during the preceding deployment step. Based on this name in combination with the template file path, the removal script find the corresponding deployment and removes all contained resources.
+For additional details on how removal works please refer to the dedicated [Removal action](PipelineRemovalAction) page.
 
 ### Publish
 
@@ -104,7 +104,7 @@ The publish phase concludes each module's pipeline. If all previous tests succee
 - _private bicep registry_
 
 By the time of this writing, the publishing experience works as follows:
-1. A user can optionally specific a specific version in the module's pipeline file, or during runtime. If the user does not, a default version is used
+1. A user can optionally specify a version in the module's pipeline file, or during runtime. If the user does not, a default version is used
 1. No matter what publishing location we enabled, the corresponding logic will
    1. Fetch the latest version of this module in the target location (if available)
    1. Compare it with any specified custom version the user optionally provided

utilities/pipelines/resourceRemoval/Initialize-DeploymentRemoval.ps1

@@ -72,6 +72,7 @@ function Initialize-DeploymentRemoval {
                     )
                     break
                 }
+                ### CODE LOCATION: Add custom removal sequence here
             }
 
             # Invoke removal

utilities/pipelines/resourceRemoval/helper/Get-ResourceIdsOfDeployment.ps1

@@ -51,9 +51,9 @@ function Get-ResourceIdsOfDeploymentInner {
         'resourcegroup' {
             if (Get-AzResourceGroup -Name $resourceGroupName -ErrorAction 'SilentlyContinue') {
                 [array]$deploymentTargets = (Get-AzResourceGroupDeploymentOperation -DeploymentName $name -ResourceGroupName $resourceGroupName).TargetResource | Where-Object { $_ -ne $null }
-                foreach ($deployment in ($deploymentTargets | Where-Object { $_ -notmatch '/deployments/' } )) {
-                    Write-Verbose ('Found deployment [{0}]' -f $deployment) -Verbose
-                    [array]$resultSet += $deployment
+                foreach ($resourceId in ($deploymentTargets | Where-Object { $_ -notmatch '/deployments/' } )) {
+                    Write-Verbose ('Found resource [{0}]' -f $resourceId) -Verbose
+                    [array]$resultSet += $resourceId
                 }
                 foreach ($deployment in ($deploymentTargets | Where-Object { $_ -match '/deployments/' } )) {
                     $name = Split-Path $deployment -Leaf

utilities/pipelines/resourceRemoval/helper/Invoke-ResourcePostRemoval.ps1

@@ -109,5 +109,6 @@ function Invoke-ResourcePostRemoval {
             # Undo a potential soft delete state change
             $null = Set-AzRecoveryServicesVaultProperty -VaultId $vaultId -SoftDeleteFeatureState $softDeleteStatus.TrimEnd('d')
         }
+        ### CODE LOCATION: Add custom post-removal operation here
     }
 }

utilities/pipelines/resourceRemoval/helper/Invoke-ResourceRemoval.ps1

@@ -95,6 +95,7 @@ function Invoke-ResourceRemoval {
             # --------------
             $null = Remove-AzResource -ResourceId $resourceId -Force -ErrorAction 'Stop'
         }
+        ### CODE LOCATION: Add custom removal action here
         Default {
             $null = Remove-AzResource -ResourceId $resourceId -Force -ErrorAction 'Stop'
         }

utilities/pipelines/resourceRemoval/helper/Remove-Deployment.ps1

@@ -22,7 +22,7 @@ Optional. The time to wait in between the search for resources via their remove
 Optional. The deployment name to use for the removal
 
 .PARAMETER TemplateFilePath
-Optional. The path to the deployment file
+Mandatory. The path to the deployment file
 
 .PARAMETER RemovalSequence
 Optional. The order of resource types to apply for deletion