Skip to content

Commit 8acc9b4

Browse files
authored
Merge pull request #3 from MicrosoftCloudEssentials-LearningHub/E2Eworkflow
E2E workflow guide
2 parents 4bb8e19 + f1570d0 commit 8acc9b4

13 files changed

Lines changed: 513 additions & 17 deletions

README.md

Lines changed: 36 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ Costa Rica
55
[![GitHub](https://img.shields.io/badge/--181717?logo=github&logoColor=ffffff)](https://github.com/)
66
[brown9804](https://github.com/brown9804)
77

8-
Last updated: 2025-02-20
8+
Last updated: 2025-02-21
99

1010
----------
1111

@@ -14,36 +14,59 @@ Last updated: 2025-02-20
1414
> [!IMPORTANT]
1515
> Disclaimer: Please note, this example of solution `it is not an official solution guide`. For official guidance, support, or more detailed information, please refer to Microsoft's official documentation or contact Microsoft directly: [Microsoft Sales and Support](https://support.microsoft.com/contactus?ContactUsExperienceEntryPointAssetId=S.HP.SMC-HOME)
1616
17-
18-
For this solution, you can set up your infrastructure using terraform:
19-
20-
[Infrastructure via Terraform](./terraform/): This approach focuses on `setting up the required infrastructure via Terraform`. It allows for source control of not only the solution code, connections, and setups `but also the infrastructure itself`.
21-
2217
<div align="center">
23-
<img src="https://github.com/user-attachments/assets/51809c83-47b5-44e0-b170-b7f22cd15975" alt="Centered Image" style="border: 2px solid #4CAF50; border-radius: 5px; padding: 5px; width: 700px;"/>
18+
<img src="https://github.com/user-attachments/assets/beaa73fb-e677-48d9-a3d9-a4367c3d7974" alt="Centered Image" style="border: 2px solid #4CAF50; border-radius: 5px; padding: 5px; width: 700px;"/>
2419
</div>
2520

26-
2721
## Prerequisites
2822

2923
- An `Azure subscription is required`. All other resources, including instructions for creating a Resource Group, are provided in this template.
3024
- If you choose to use the Terraform approach, please ensure that:
3125
- [Terraform is installed on your local machine](https://developer.hashicorp.com/terraform/tutorials/azure-get-started/install-cli#install-terraform).
3226
- [Install the Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli) to work with both Terraform and Azure commands.
3327

34-
3528
## Where to start?
3629

3730
Please follow as described below:
3831

39-
1. Please follow the [Terraform guide](./terraform/README.md) to deploy the necessary Azure resources for the workshop.
40-
2. Then, follow each [each section](#content).
32+
- If you're choosing the deploying the `infrastructure via Azure Portal`:
33+
1. Go through [each section](#content) `from the start`.
34+
- If you're choosing the [Infrastructure via Terraform](./terraform/) approach:
35+
1. Please follow the [Terraform guide](./terraform/README.md) to deploy the necessary Azure resources for the workshop.
36+
2. Then, follow each [each section](#content) but `skip creating any resources that you've already deployed with Terraform.`.
37+
38+
> [!NOTE]
39+
> About [Infrastructure via Terraform](./terraform/), Terraform is an infrastructure as code (IaC) tool that allows you to define and provision your infrastructure using a high-level configuration language. This approach `enables source control of the infrastructure itself, allowing you to manage not only the solution code but also the connections and configurations`. By using Terraform, you can ensure a consistent and reproducible environment for your deployments, automate infrastructure provisioning, and maintain version control over your infrastructure changes. Also, Microsoft provides other IaC tools such as Bicep and ARM templates. Bicep is a domain-specific language that uses declarative syntax to deploy Azure resources, offering a concise and easy-to-read alternative to JSON-based ARM templates. ARM templates are JSON files that define the infrastructure and configuration for your Azure solution. These tools provide flexibility and options to suit different preferences and requirements for managing Azure resources.
40+
41+
<div align="center">
42+
<img src="https://github.com/user-attachments/assets/51809c83-47b5-44e0-b170-b7f22cd15975" alt="Centered Image" style="border: 2px solid #4CAF50; border-radius: 5px; padding: 5px; width: 700px;"/>
43+
</div>
4144

4245
## Content
4346

44-
47+
- [Solution Overview](./instructions/0_Overview/README.md): Understand the core components and capabilities.
48+
- [Step 1: Creating a Resource Group in the Azure Portal](./instructions/1_CreateRG.md)
49+
- [Step 2: Set Up Azure Blob Storage](./instructions/2_CreateBlobStorage.md)
50+
- [Step 3: Set Up AI Services](./instructions/3_SetupAIServices.md)
51+
- [Step 4: Create a Log Analytics workspace in Azure](./instructions/4_CreateLogAnalyticsWS.md)
52+
- [Step 5: Setup the Key Vault and Store secrets/keys](./instructions/5_SetupKeyVault.md)
53+
54+
## Important Considerations for Production Environment
55+
56+
- **Public Network Site**: This example is based on a public network site and is intended for demonstration purposes only. It showcases how various Azure resources can work together to achieve the desired result.
57+
- **Private Network Configuration**: For enhanced security, consider configuring your Azure resources to operate within a private network. This can be achieved using Azure Virtual Network (VNet) to isolate your resources and control inbound and outbound traffic. Implementing private endpoints for services like Azure Blob Storage and Azure Functions can further secure your data by restricting access to your VNet.
58+
- **Security**: Ensure that you implement appropriate security measures when deploying this solution in a production environment. This includes:
59+
- **Securing Access**: Use Azure Entra ID (formerly known as Azure Active Directory or Azure AD) for authentication and role-based access control (RBAC) to manage permissions.
60+
- **Managing Secrets**: Store sensitive information such as connection strings and API keys in Azure Key Vault.
61+
- **Data Encryption**: Enable encryption for data at rest and in transit to protect sensitive information.
62+
- **Scalability**: While this example provides a basic setup, you may need to scale the resources based on your specific requirements. Azure services offer various scaling options to handle increased workloads. Consider using:
63+
- **Auto-scaling**: Configure auto-scaling for Azure Functions and other services to automatically adjust based on demand.
64+
- **Load Balancing**: Use Azure Load Balancer or Application Gateway to distribute traffic and ensure high availability.
65+
- **Cost Management**: Monitor and manage the costs associated with your Azure resources. Use Azure Cost Management and Billing to track usage and optimize resource allocation.
66+
- **Compliance**: Ensure that your deployment complies with relevant regulations and standards. Use Azure Policy to enforce compliance and governance policies across your resources.
67+
- **Disaster Recovery**: Implement a disaster recovery plan to ensure business continuity in case of failures. Use Azure Site Recovery and backup solutions to protect your data and applications.
68+
4569
<div align="center">
4670
<h3 style="color: #4CAF50;">Total Visitors</h3>
4771
<img src="https://profile-counter.glitch.me/brown9804/count.svg" alt="Visitor Count" style="border: 2px solid #4CAF50; border-radius: 5px; padding: 5px;"/>
4872
</div>
49-

instructions/0_Overview/README.md

Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
# E2E (End to End) Workflow: Overview
2+
3+
Costa Rica
4+
5+
[![GitHub](https://img.shields.io/badge/--181717?logo=github&logoColor=ffffff)](https://github.com/)
6+
[brown9804](https://github.com/brown9804)
7+
8+
Last updated: 2025-02-21
9+
10+
------------------------------------------
11+
12+
<details>
13+
<summary><b>List of references</b> (Click to expand)</summary>
14+
15+
- [Quickstart: Azure AI Vision v3.2 GA Read](https://learn.microsoft.com/en-us/azure/ai-services/computer-vision/quickstarts-sdk/client-library?tabs=windows%2Cvisual-studio&pivots=programming-language-python)
16+
- [Quickstart: Azure Blob Storage client library for Python](https://learn.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-python?tabs=managed-identity%2Croles-azure-portal%2Csign-in-azure-cli&pivots=blob-storage-quickstart-scratch)
17+
- [Create a function in Azure that's triggered by Blob storage](https://learn.microsoft.com/en-us/azure/azure-functions/functions-create-storage-blob-triggered-function)
18+
- [Quickstart: Azure Key Vault secret client library for Python](https://learn.microsoft.com/en-us/azure/key-vault/secrets/quick-create-python?tabs=azure-cli)
19+
- [Azure Application Insights SDK for Python](https://learn.microsoft.com/en-us/python/api/overview/azure/application-insights?view=azure-python)
20+
- [Create a Log Analytics workspace](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/quick-create-workspace?tabs=azure-portal)
21+
- [Quickstart: Use the Azure portal to create a virtual network](https://learn.microsoft.com/en-us/azure/virtual-network/quick-create-portal)
22+
23+
</details>
24+
25+
<details>
26+
<summary><b>List of Contents</b> (Click to expand)</summary>
27+
28+
- [Workflow](#workflow)
29+
- [Architecture: Components and Interactions](#architecture-components-and-interactions)
30+
31+
</details>
32+
33+
## Workflow
34+
35+
```mermaid
36+
graph TD
37+
A[Upload File to Blob Storage] -->|Trigger| B[Azure Function]
38+
B --> C[Read Key Table from Blob Storage]
39+
B --> D[Read Input File from Blob Storage]
40+
C --> E[Process Data]
41+
D --> E[Process Data]
42+
E --> F[Generate Summary]
43+
F --> G[Save Summary to Output Container]
44+
G --> H[Summary Available for Download]
45+
46+
subgraph Azure Resources
47+
A
48+
B
49+
C
50+
D
51+
G
52+
end
53+
54+
subgraph Processing Steps
55+
E
56+
F
57+
H
58+
end
59+
```
60+
61+
62+
1. **Upload File to Blob Storage**: Users upload files to the input container in Azure Blob Storage.
63+
2. **Azure Function**: The Azure Function is triggered by the file upload.
64+
3. **Read Key Table from Blob Storage**: The function reads the key table from the input container.
65+
4. **Read Input File from Blob Storage**: The function reads the uploaded input file from the input container.
66+
5. **Process Data**: The function processes the data by searching for key values and extracting relevant information.
67+
6. **Generate Summary**: The function generates a summary based on the processed data.
68+
7. **Save Summary to Output Container**: The function saves the summary to the output container in Azure Blob Storage.
69+
8. **Summary Available for Download**: The summary is available for download from the output container.
70+
71+
## Architecture: Components and Interactions
72+
73+
<div align="center">
74+
<img src="https://github.com/user-attachments/assets/beaa73fb-e677-48d9-a3d9-a4367c3d7974" alt="Centered Image" style="border: 2px solid #4CAF50; border-radius: 5px; padding: 5px; width: 700px;"/>
75+
</div>
76+
77+
<div align="center">
78+
<h3 style="color: #4CAF50;">Total Visitors</h3>
79+
<img src="https://profile-counter.glitch.me/brown9804/count.svg" alt="Visitor Count" style="border: 2px solid #4CAF50; border-radius: 5px; padding: 5px;"/>
80+
</div>
Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
<mxfile host="Electron" agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) draw.io/26.0.9 Chrome/128.0.6613.186 Electron/32.2.5 Safari/537.36" version="26.0.9">
2+
<diagram name="workflow" id="VzGhtupkYtY4OB6_fRJi">
3+
<mxGraphModel dx="2143" dy="696" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="850" pageHeight="1100" math="0" shadow="0">
4+
<root>
5+
<mxCell id="0" />
6+
<mxCell id="1" parent="0" />
7+
<mxCell id="UkH3Tmcc9s9qquvXN7uM-32" value="" style="rounded=0;whiteSpace=wrap;html=1;" parent="1" vertex="1">
8+
<mxGeometry x="-720" y="170" width="830" height="510" as="geometry" />
9+
</mxCell>
10+
<mxCell id="UkH3Tmcc9s9qquvXN7uM-30" value="Subscription" style="image;sketch=0;aspect=fixed;html=1;points=[];align=center;fontSize=12;image=img/lib/mscae/Key.svg;" parent="1" vertex="1">
11+
<mxGeometry x="-734.5" y="153.44" width="31" height="50" as="geometry" />
12+
</mxCell>
13+
<mxCell id="UkH3Tmcc9s9qquvXN7uM-31" value="Resource Group" style="image;sketch=0;aspect=fixed;html=1;points=[];align=center;fontSize=12;image=img/lib/mscae/ResourceGroup.svg;" parent="1" vertex="1">
14+
<mxGeometry x="-744" y="640" width="50" height="40" as="geometry" />
15+
</mxCell>
16+
<mxCell id="Vv6FFmmm0vw7hVMtTU7b-1" value="Blob Storage" style="image;sketch=0;aspect=fixed;html=1;points=[];align=center;fontSize=12;image=img/lib/mscae/BlobBlock.svg;" parent="1" vertex="1">
17+
<mxGeometry x="-385" y="265.64" width="58" height="48.72" as="geometry" />
18+
</mxCell>
19+
<mxCell id="Eqv_7Mpk2FUOEcddT1MY-4" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;" edge="1" parent="1" source="Vv6FFmmm0vw7hVMtTU7b-8" target="Vv6FFmmm0vw7hVMtTU7b-1">
20+
<mxGeometry relative="1" as="geometry" />
21+
</mxCell>
22+
<mxCell id="Eqv_7Mpk2FUOEcddT1MY-6" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;" edge="1" parent="1" source="Vv6FFmmm0vw7hVMtTU7b-8" target="3NK07bUQuaUZu8PJls8a-1">
23+
<mxGeometry relative="1" as="geometry" />
24+
</mxCell>
25+
<mxCell id="Eqv_7Mpk2FUOEcddT1MY-7" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;" edge="1" parent="1" source="Vv6FFmmm0vw7hVMtTU7b-8" target="3NK07bUQuaUZu8PJls8a-2">
26+
<mxGeometry relative="1" as="geometry" />
27+
</mxCell>
28+
<mxCell id="Vv6FFmmm0vw7hVMtTU7b-8" value="Function App" style="image;aspect=fixed;html=1;points=[];align=center;fontSize=12;image=img/lib/azure2/compute/Function_Apps.svg;" parent="1" vertex="1">
29+
<mxGeometry x="-395" y="457" width="68" height="60" as="geometry" />
30+
</mxCell>
31+
<mxCell id="Vv6FFmmm0vw7hVMtTU7b-12" value="Application&amp;nbsp;&lt;br&gt;Insights" style="image;sketch=0;aspect=fixed;html=1;points=[];align=center;fontSize=12;image=img/lib/mscae/Application_Insights.svg;" parent="1" vertex="1">
32+
<mxGeometry x="6" y="559.36" width="32" height="50" as="geometry" />
33+
</mxCell>
34+
<mxCell id="Eqv_7Mpk2FUOEcddT1MY-10" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;" edge="1" parent="1" source="Vv6FFmmm0vw7hVMtTU7b-13" target="UkH3Tmcc9s9qquvXN7uM-31">
35+
<mxGeometry relative="1" as="geometry" />
36+
</mxCell>
37+
<mxCell id="Vv6FFmmm0vw7hVMtTU7b-13" value="Log Analytics Workspace" style="image;aspect=fixed;html=1;points=[];align=center;fontSize=12;image=img/lib/azure2/management_governance/Log_Analytics_Workspaces.svg;" parent="1" vertex="1">
38+
<mxGeometry x="-540" y="576" width="64" height="64" as="geometry" />
39+
</mxCell>
40+
<mxCell id="3NK07bUQuaUZu8PJls8a-1" value="AI Services" style="image;aspect=fixed;html=1;points=[];align=center;fontSize=12;image=img/lib/azure2/ai_machine_learning/Cognitive_Services.svg;" parent="1" vertex="1">
41+
<mxGeometry x="-210" y="440" width="68" height="48" as="geometry" />
42+
</mxCell>
43+
<mxCell id="3NK07bUQuaUZu8PJls8a-2" value="Key Vault" style="image;aspect=fixed;html=1;points=[];align=center;fontSize=12;image=img/lib/azure2/security/Key_Vaults.svg;" parent="1" vertex="1">
44+
<mxGeometry x="-609" y="457" width="68" height="68" as="geometry" />
45+
</mxCell>
46+
<mxCell id="Eqv_7Mpk2FUOEcddT1MY-3" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;" edge="1" parent="1" source="Eqv_7Mpk2FUOEcddT1MY-1" target="Vv6FFmmm0vw7hVMtTU7b-1">
47+
<mxGeometry relative="1" as="geometry" />
48+
</mxCell>
49+
<mxCell id="Eqv_7Mpk2FUOEcddT1MY-1" value="User &lt;br&gt;Uploads File&amp;nbsp;" style="shape=umlActor;verticalLabelPosition=bottom;verticalAlign=top;html=1;outlineConnect=0;" vertex="1" parent="1">
50+
<mxGeometry x="-630" y="260" width="30" height="60" as="geometry" />
51+
</mxCell>
52+
<mxCell id="Eqv_7Mpk2FUOEcddT1MY-2" value="" style="whiteSpace=wrap;html=1;shape=mxgraph.basic.document" vertex="1" parent="1">
53+
<mxGeometry x="-590" y="250" width="30" height="30" as="geometry" />
54+
</mxCell>
55+
<mxCell id="Eqv_7Mpk2FUOEcddT1MY-11" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;entryX=0.446;entryY=0.874;entryDx=0;entryDy=0;entryPerimeter=0;" edge="1" parent="1" source="Vv6FFmmm0vw7hVMtTU7b-12" target="Vv6FFmmm0vw7hVMtTU7b-8">
56+
<mxGeometry relative="1" as="geometry" />
57+
</mxCell>
58+
</root>
59+
</mxGraphModel>
60+
</diagram>
61+
</mxfile>

instructions/1_CreateRG.md

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
# Step 1: Creating a Resource Group in the Azure Portal
2+
3+
Costa Rica
4+
5+
[![GitHub](https://img.shields.io/badge/--181717?logo=github&logoColor=ffffff)](https://github.com/)
6+
[brown9804](https://github.com/brown9804)
7+
8+
Last updated: 2025-02-21
9+
10+
------------------------------------------
11+
12+
> An Azure `Resource Group` is a `container that holds related resources for an Azure solution`.
13+
> It can include all the resources for the solution or only those you want to manage as a group.
14+
> Typically, resources that share the same lifecycle are added to the same resource group, allowing for easier deployment, updating, and deletion as a unit.
15+
> Resource groups also store metadata about the resources, and you can apply access control, locks, and tags to them for better management and organization.
16+
17+
<div align="center">
18+
<img src="https://github.com/user-attachments/assets/c02c3623-477f-4ab5-876a-9801191a34f8" alt="Centered Image" style="border: 2px solid #4CAF50; border-radius: 5px; padding: 5px; width: 200px;"/>
19+
</div>
20+
21+
22+
1. **Create an Azure Account**: If you don't have one, sign up for an Azure account.
23+
2. **Create a Resource Group**:
24+
- Go to the Azure portal.
25+
- Navigate to **Resource groups**.
26+
- Click **+ Create**.
27+
28+
<img width="550" alt="image" src="https://github.com/user-attachments/assets/56d1e99f-0a22-4492-bd6f-d4e3a76aedd8">
29+
30+
- Enter the Resource Group name (e.g., `RGFileUploadSummary`) and select a region (e.g., `East US 2`). You can add tags if needed.
31+
- Click **Review + create** and then **Create**.
32+
33+
<img width="550" alt="image" src="https://github.com/user-attachments/assets/44aefda0-a373-4682-b1dd-3ddf6e3a94c7">
34+
35+
<div align="center">
36+
<h3 style="color: #4CAF50;">Total Visitors</h3>
37+
<img src="https://profile-counter.glitch.me/brown9804/count.svg" alt="Visitor Count" style="border: 2px solid #4CAF50; border-radius: 5px; padding: 5px;"/>
38+
</div>

0 commit comments

Comments
 (0)