data loader updates

Author: Chenglong-MS

README.md

@@ -23,9 +23,9 @@ Transform data and create rich visualizations iteratively with AI 🪄. Try Data
 
 ## News 🔥🔥🔥
 
-- [05-13-2025] Data Formulator 0.2.1: External Data Loader 
+- [05-13-2025] Data Formulator 0.2.3: External Data Loader 
   - We introduced external data loader class to make import data easier. [Readme](https://github.com/microsoft/data-formulator/tree/main/py-src/data_formulator/data_loader) and [Demo](https://github.com/microsoft/data-formulator/pull/155)
-    - Example data loaders from MySQL and Azure Data Explorer (Kusto) are provided.
+    - Current data loaders: MySQL, Azure Data Explorer (Kusto), Azure Blob and Amazon S3 (json, parquet, csv).
   - Call for action [link](https://github.com/microsoft/data-formulator/issues/156):
     - Users: let us know which data source you'd like to load data from.
     - Developers: let's build more data loaders.

py-src/data_formulator/data_loader/azure_blob_data_loader.py

@@ -23,33 +23,39 @@ def list_params() -> List[Dict[str, Any]]:
 
     @staticmethod
     def auth_instructions() -> str:
-        return """**Authentication Options (choose one)**
+        return """Authentication Options (choose one)
 
-**Option 1 - Connection String (Simplest)**
-- Get connection string from Azure Portal > Storage Account > Access keys
-- Use `connection_string` parameter with full connection string
-- `account_name` can be omitted when using connection string
+Option 1 - Connection String (Simplest)
+    - Get connection string from Azure Portal > Storage Account > Access keys
+    - Use `connection_string` parameter with full connection string
+    - `account_name` can be omitted when using connection string
 
-**Option 2 - Account Key**
-- Get account key from Azure Portal > Storage Account > Access keys
-- Use `account_name` + `account_key` parameters
-- Provides full access to storage account
+Option 2 - Account Key
+    - Get account key from Azure Portal > Storage Account > Access keys
+    - Use `account_name` + `account_key` parameters
+    - Provides full access to storage account
 
-**Option 3 - SAS Token (Recommended for limited access)**
-- Generate SAS token from Azure Portal > Storage Account > Shared access signature
-- Use `account_name` + `sas_token` parameters
-- Can be time-limited and permission-scoped
+Option 3 - SAS Token (Recommended for limited access)
+    - Generate SAS token from Azure Portal > Storage Account > Shared access signature
+    - Use `account_name` + `sas_token` parameters
+    - Can be time-limited and permission-scoped
 
-**Option 4 - Credential Chain (Most Secure)**
-- Use `account_name` + `container_name` only (no explicit credentials)
-- Requires Azure CLI login (`az login`) or Managed Identity
-- Default chain: `cli;managed_identity;env`
-- Customize with `credential_chain` parameter
+Option 4 - Credential Chain (Most Secure)
+    - Use `account_name` + `container_name` only (no explicit credentials)
+    - Requires Azure CLI login (`az login` in terminal) or Managed Identity
+    - Default chain: `cli;managed_identity;env`
+    - Customize with `credential_chain` parameter
 
-**Additional Options**
-- `endpoint`: Custom endpoint (default: `blob.core.windows.net`)
-- For Azure Government: `blob.core.usgovcloudapi.net`
-- For Azure China: `blob.core.chinacloudapi.cn`"""
+Additional Options
+    - `endpoint`: Custom endpoint (default: `blob.core.windows.net`)
+    - For Azure Government: `blob.core.usgovcloudapi.net`
+    - For Azure China: `blob.core.chinacloudapi.cn`
+
+Supported File Formats:
+    - CSV files (.csv)
+    - Parquet files (.parquet) 
+    - JSON files (.json, .jsonl)
+"""
 
     def __init__(self, params: Dict[str, Any], duck_db_conn: duckdb.DuckDBPyConnection):
         self.params = params

py-src/data_formulator/data_loader/kusto_data_loader.py

@@ -26,38 +26,30 @@ def list_params() -> bool:
 
     @staticmethod
     def auth_instructions() -> str:
-        return """
-Azure Kusto Authentication Instructions:
-
-This data loader supports two authentication methods:
-
-**Method 1: Azure CLI Authentication (Recommended for development)**
-1. Install Azure CLI: https://docs.microsoft.com/en-us/cli/azure/install-azure-cli
-2. Run `az login` in your terminal to authenticate
-3. Ensure you have access to the specified Kusto cluster and database
-4. Leave client_id, client_secret, and tenant_id parameters empty
-
-**Method 2: Application Key Authentication (Recommended for production)**
-1. Register an Azure AD application in your tenant
-2. Generate a client secret for the application
-3. Grant the application appropriate permissions to your Kusto cluster:
-   - Go to your Kusto cluster in Azure Portal
-   - Navigate to Permissions > Add
-   - Add your application as a user with appropriate role (e.g., "AllDatabasesViewer" for read access)
-4. Provide the following parameters:
-   - client_id: Application (client) ID from your Azure AD app registration
-   - client_secret: Client secret value you generated
-   - tenant_id: Directory (tenant) ID from your Azure AD
-
-**Required Parameters:**
-- kusto_cluster: Your Kusto cluster URI (e.g., "https://mycluster.region.kusto.windows.net")
-- kusto_database: Name of the database you want to access
-
-**Troubleshooting:**
-- If authentication fails, ensure you have the correct permissions on the Kusto cluster
-- For CLI auth, make sure you're logged in with `az account show`
-- For app key auth, verify your client_id, client_secret, and tenant_id are correct
-        """
+        return """Azure Kusto Authentication Instructions
+
+Method 1: Azure CLI Authentication
+    1. Install Azure CLI: https://docs.microsoft.com/en-us/cli/azure/install-azure-cli
+    2. Run `az login` in your terminal to authenticate
+    3. Ensure you have access to the specified Kusto cluster and database
+    4. Leave client_id, client_secret, and tenant_id parameters empty
+
+Method 2: Application Key Authentication
+    1. Register an Azure AD application in your tenant
+    2. Generate a client secret for the application
+    3. Grant the application appropriate permissions to your Kusto cluster:
+        - Go to your Kusto cluster in Azure Portal
+        - Navigate to Permissions > Add
+        - Add your application as a user with appropriate role (e.g., "AllDatabasesViewer" for read access)
+    4. Provide the following parameters:
+        - client_id: Application (client) ID from your Azure AD app registration
+        - client_secret: Client secret value you generated
+        - tenant_id: Directory (tenant) ID from your Azure AD
+
+Required Parameters:
+    - kusto_cluster: Your Kusto cluster URI (e.g., "https://mycluster.region.kusto.windows.net")
+    - kusto_database: Name of the database you want to access
+"""
 
     def __init__(self, params: Dict[str, Any], duck_db_conn: duckdb.DuckDBPyConnection):
 

py-src/data_formulator/data_loader/mysql_data_loader.py

@@ -23,32 +23,26 @@ def auth_instructions() -> str:
         return """
 MySQL Connection Instructions:
 
-1. **Local MySQL Setup:**
+1. Local MySQL Setup:
    - Ensure MySQL server is running on your machine
    - Default connection: host='localhost', user='root'
    - If you haven't set a root password, leave password field empty
 
-2. **Remote MySQL Connection:**
+2. Remote MySQL Connection:
    - Obtain host address, username, and password from your database administrator
    - Ensure the MySQL server allows remote connections
    - Check that your IP is whitelisted in MySQL's user permissions
 
-3. **Common Connection Parameters:**
+3. Common Connection Parameters:
    - user: Your MySQL username (default: 'root')
    - password: Your MySQL password (leave empty if no password set)
    - host: MySQL server address (default: 'localhost')
    - database: Target database name to connect to
 
-4. **Troubleshooting:**
+4. Troubleshooting:
    - Verify MySQL service is running: `brew services list` (macOS) or `sudo systemctl status mysql` (Linux)
    - Test connection: `mysql -u [username] -p -h [host] [database]`
-   - Common issues: Wrong credentials, server not running, firewall blocking connection
-
-5. **Security Notes:**
-   - Use dedicated database users with limited privileges for applications
-   - Avoid using root user for application connections
-   - Consider using SSL connections for remote databases
-        """
+"""
 
     def __init__(self, params: Dict[str, Any], duck_db_conn: duckdb.DuckDBPyConnection):
         self.params = params

py-src/data_formulator/data_loader/s3_data_loader.py

@@ -22,65 +22,38 @@ def list_params() -> List[Dict[str, Any]]:
     @staticmethod
     def auth_instructions() -> str:
         return """
-To connect to Amazon S3, you'll need the following AWS credentials and configuration:
-
-**Required Parameters:**
+**Required AWS Credentials:**
 - **AWS Access Key ID**: Your AWS access key identifier
 - **AWS Secret Access Key**: Your AWS secret access key  
-- **Region Name**: The AWS region where your S3 bucket is located (e.g., 'us-east-1', 'us-west-2')
-- **Bucket**: The name of your S3 bucket
-
-**Optional Parameters:**
-- **AWS Session Token**: Required only if using temporary credentials (e.g., from AWS STS or IAM roles)
-
-**How to Get AWS Credentials:**
-
-1. **AWS IAM User (Recommended for programmatic access):**
-   - Go to AWS Console → IAM → Users
-   - Create a new user or select existing user
-   - Go to "Security credentials" tab
-   - Click "Create access key"
-   - Choose "Application running outside AWS"
-   - Save both the Access Key ID and Secret Access Key securely
-
-2. **Required S3 Permissions:**
-   Your IAM user/role needs these permissions for the target bucket:
-   ```json
-   {
-     "Version": "2012-10-17",
-     "Statement": [
-       {
-         "Effect": "Allow",
-         "Action": [
-           "s3:GetObject",
-           "s3:ListBucket"
-         ],
-         "Resource": [
-           "arn:aws:s3:::your-bucket-name",
-           "arn:aws:s3:::your-bucket-name/*"
-         ]
-       }
-     ]
-   }
-   ```
-
-3. **Finding Your Region:**
-   - Go to S3 Console → Select your bucket → Properties
-   - Look for "AWS Region" in the bucket overview
-
-**Security Best Practices:**
-- Never share your secret access key
-- Use IAM roles when possible instead of long-term access keys
-- Consider using temporary credentials with session tokens for enhanced security
-- Regularly rotate your access keys
-- Use the principle of least privilege for S3 permissions
+- **Region Name**: AWS region (e.g., 'us-east-1', 'us-west-2')
+- **Bucket**: S3 bucket name
+- **AWS Session Token**: Optional, for temporary credentials only
+
+**Getting Credentials:**
+1. AWS Console → IAM → Users → Select user → Security credentials → Create access key
+2. Choose "Application running outside AWS"
+
+**Required S3 Permissions:**
+```json
+{
+  "Version": "2012-10-17",
+  "Statement": [{
+    "Effect": "Allow",
+    "Action": ["s3:GetObject", "s3:ListBucket"],
+    "Resource": [
+      "arn:aws:s3:::your-bucket-name",
+      "arn:aws:s3:::your-bucket-name/*"
+    ]
+  }]
+}
+```
 
 **Supported File Formats:**
 - CSV files (.csv)
 - Parquet files (.parquet) 
 - JSON files (.json, .jsonl)
 
-The connector will automatically detect file types and load them appropriately using DuckDB's S3 integration.
+**Security:** Never share secret keys, rotate regularly, use least privilege permissions.
         """
 
     def __init__(self, params: Dict[str, Any], duck_db_conn: duckdb.DuckDBPyConnection):

py-src/data_formulator/tables_routes.py

@@ -728,7 +728,10 @@ def data_loader_list_data_loaders():
         return jsonify({
             "status": "success",
             "data_loaders": {
-                name:  data_loader.list_params()
+                name: {
+                    "params": data_loader.list_params(),
+                    "auth_instructions": data_loader.auth_instructions()
+                }
                 for name, data_loader in DATA_LOADERS.items()
             }
         })

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "data_formulator"
-version = "0.2.1.2"
+version = "0.2.1.3"
 
 requires-python = ">=3.9"
 authors = [