Databricks JSON Data Flattening

A lightweight and reusable PySpark framework for recursively flattening deeply nested JSON datasets in Databricks. The project converts complex semi-structured JSON into analytics-ready tabular data suitable for Delta Lake, SQL Warehouses, BI tools, and downstream data pipelines.

---

Features

• Recursive JSON flattening
• Handles nested StructType
• Handles nested ArrayType
• Supports arrays of structs
• Dynamic schema processing
• Databricks & Apache Spark compatible
• Delta Lake ready
• Reusable utility functions
• Suitable for batch and streaming workloads

---

Technology Stack

Technology	Purpose
Python	Programming Language
PySpark	Distributed Data Processing
Apache Spark	Processing Engine
Databricks	Execution Platform
Delta Lake	Storage Format
JSON	Source Data

---

Project Structure

databricks_json_data_flatten/
│
├── notebooks/
│   ├── amazon_json.py
│
├── data/
│   └── restaurant.json
│
├── requirements.txt
├── README.md
└── LICENSE

---

Dataset

The sample dataset contains restaurant information collected from the Zomato API. It includes multiple levels of nested objects and arrays, making it a practical example for recursive JSON flattening.

Nested Objects

• Restaurant
• Location
• User Rating
• Events
• Photos
• Offers

Nested Arrays

• Restaurants
• Events
• Photos
• Establishment Types

---

Sample Input

{
  "results_found": 17151,
  "restaurants": [
    {
      "restaurant": {
        "id": "308322",
        "name": "Hauz Khas Social",
        "cuisines": "Continental, American, Asian",
        "average_cost_for_two": 1600,
        "location": {
          "city": "New Delhi",
          "locality": "Hauz Khas Village"
        },
        "user_rating": {
          "aggregate_rating": "4.3",
          "votes": "7931"
        }
      }
    }
  ]
}

---

Flattened Output

restaurant_id	restaurant_name	city	locality	cuisines	rating	votes
308322	Hauz Khas Social	New Delhi	Hauz Khas Village	Continental, American, Asian	4.3	7931

---

Data Flow

Raw JSON
    │
    ▼
Read JSON
    │
    ▼
Detect Nested Columns
    │
    ▼
Flatten Struct Columns
    │
    ▼
Explode Array Columns
    │
    ▼
Repeat Until Flat
    │
    ▼
Flattened DataFrame
    │
    ▼
Delta Table / CSV / Parquet

---

Performance Notes

• Supports large nested JSON datasets
• Works efficiently with Spark distributed processing
• Compatible with Delta Lake
• Designed for enterprise ETL/ELT pipelines

---

Use Cases

• REST API Ingestion
• Event Hub
• Kafka
• MongoDB Export
• Cosmos DB
• Data Lake Bronze Layer
• Medallion Architecture
• Analytics Pipelines

---

Run

Clone the repository:

git clone https://github.com/hridoy1335/databricks_json_data_flatten.git

cd databricks_json_data_flatten

Install dependencies:

pip install -r requirements.txt

Run the project:

python main.py

Or execute the notebook directly in Databricks.

---

Future Improvements

• Schema evolution support
• Configuration-driven flattening
• Streaming support
• Delta Live Tables integration
• Unity Catalog integration

---

License

This project is licensed under the MIT License.

---

Author

Hridoy Khan

Data Engineer | Azure | Databricks | Apache Spark | Delta Lake

GitHub: https://github.com/hridoy1335

---

⭐ If you found this project useful, consider starring the repository.