Nirman.AI (निर्माण.AI)

Blueprint Intelligence Workspace for Indian Construction & EPC Teams Track: Autonomous Orchestration with Managed Agents

---

Nirman.AI is a drawing-first estimation workspace designed to eliminate the manual, slow, and error-prone process of construction quantity takeoff and cost estimation. By combining a persisted per-sheet element index, element-anchored high-resolution crops, Google Gemini's multimodal capabilities, and an autonomous multi-agent pipeline, Nirman.AI turns dense, complex engineering blueprints into a costed, source-grounded Bill of Quantities (BOQ) written directly to a Google Sheet.

🚀 The Core Problem

For Engineering-Procurement-Construction (EPC) contractors in India, estimating tenders is highly resource-intensive:

• Time-consuming: A manual BOQ takeoff from A2/A3 drawings takes upwards of 15 days per tender.
• Costly errors: Manual takeoff accuracy hovers around 85%. The missing 15% results in massive cost overruns once a bid is won.
• Throughput limit: Firms can only bid on ~40 tenders a year because of this takeoff bottleneck.
• Knowledge lock-in: The ability to accurately interpret complex engineering schedules lives in the minds of a few senior engineers.

Nirman.AI automates this entire cycle: Upload PDF → Navigate Canvas → Select an Element & Ask → Explore the Truth-3D model → Run 5-Agent Pipeline → Get Costed BOQ in Google Sheets.

---

🏗️ Architecture & How It Works

Nirman.AI features a modular React-based frontend and a FastAPI backend powered by SQLite, gspread, and the Google GenAI SDK.

┌────────────────┐   Upload PDF   ┌────────────────────────────────┐
│  React (Vite)  │ ─────────────► │ FastAPI Backend (Python 3.12)  │
│  PDF Canvas    │                │                                │
│  Element Pick  │  Ask + Context │ Ingestion:  PyMuPDF Render     │
│  Agent Status  │ ─────────────► │ Pipeline:   5-Agent DAG        │
└────────────────┘    /estimate   │ Persistence: SQLite            │
        ▲                         │ Deliverable: Google Sheets API │
        │   SSE Agent Stream      └────────────────────────────────┘
        └─────────────────────────────────────────┘

The 5-Agent Estimation Pipeline

A single LLM run fails at complex visual-geometric analysis, exact arithmetic, live market rate fetching, and structured formatting. Nirman.AI solves this by orchestrating a sequential 5-Agent DAG with live handoffs streamed to the user interface via Server-Sent Events (SSE):

#	Agent	Role	Model / Tools	Output Contract
A1	Drawing Reader	Reads each drawing element (vision) to extract dimensions, rebar schedules, and concrete grade.	gemini-3.5-flash (vision)	Structured raw drawing data
A2	Quantity Surveyor	Computes concrete volumes, steel weights, and formwork areas based on A1's output.	gemini-3.5-flash + Python Code Execution	TakeoffResult (Quantities & Formulas)
A3	Rate Analyst	Searches live Indian construction directories/marketplaces to fetch current INR (₹) rates.	gemini-3.5-flash + Google Search Grounding	RatedBOQ (Base unit rates in ₹)
A4	Bid Risk Analyzer	Analyzes escalation terms, project location, and adds safety buffers based on volatility.	gemini-3.5-flash (reasoning)	RiskedBOQ (Final buffered rates)
A5	Sheets Writer	Exports the final BOQ into the app, and (optionally) into a connected Google Sheet.	Deterministic gspread integration	In-app BOQ + Google Sheets tab

Element-anchored Q&A

At ingest, every sheet is broken into a structured element index (footings, walls, columns, rebar schedules, notes) — each with a bbox, kind, description, and (for schedules) transcribed rows. Instead of drawing boxes, you select an element: it highlights on the canvas (the viewer frames it from the stored bbox) and an "Ask AI" chat opens, scoped to that element but grounded in the whole sheet — its title block, notes, grades, and the other elements.

• POST /api/elements/{id}/ask (app/qna.py) → one gemini-3.5-flash call assembled from persisted data (element record + sheet understanding + a hi-res crop of the element's bbox). Both turns are saved to messages (anchored via element_id); GET /api/elements/{id}/messages restores the thread on reload.
• Ask "how much reinforcement steel," "what's the concrete volume," "what grade is this," "explain this detail." Answers are computed with formulas and stay honest — if a dimension isn't on the drawing it says what's missing instead of inventing one. Because the expensive understanding is done once at upload, each question is a cheap, well-grounded single call.

The Structure Mapper (project element index)

A drawing set describes one project across several sheets by role — a single element (e.g. the container) has its plan on one sheet, its section/levels on the GAD, its grade in the notes, and its reinforcement in a schedule on a fourth. Costing one sheet in isolation is therefore always incomplete. After per-sheet ingest, the Structure Mapper (app/structure.py) runs automatically:

1. Classifies each sheet by role — locator | general_arrangement | structural_detail | reinforcement | notes_legend.
2. Reads the quantifiable sheets (vision) for members + figured dimensions.
3. Fuses everything into a bill of elements — one entry per physical assembly, grade resolved from the notes, dimensions gathered from whichever sheet has them, each traced to its source, and flagged ready | missing_dims | out_of_scope.

The estimate then runs at the project level — it costs every ready assembly, so a layout/locator sheet contributes nothing (no confabulated BOQ), and the real structure (container M30, staging M25, footings, beams) is costed from the sheets that actually carry those dimensions.

Truth-3D — the drawing, reconstructed (measurable, source-traceable)

The Structure Mapper's element index is also rendered as an interactive 3D model of the whole structure — toggle "3D model" in the workspace (app/geometry.py, app/scene.py → GET /api/documents/{id}/geometry, Three.js viewer; design in docs/06-visualization.md):

• Two-layer trust contract: every solid's size comes only from a figured dimension the mapper read off a sheet (truth); the arrangement is inferred from the general-arrangement drawing and always labelled as such. Anything not dimensioned renders flagged assumed — never confabulated.
• Typology-aware builder: a detected overhead water tank gets a high-fidelity parametric model (bracing rings at their RLs, water fill animated to FSL with a computed-KL counter, helical stair); an architectural set renders as a storey-stack massing read off the elevation levels; any other structure (bridge GAD, culvert…) is assembled by a Gemini layout pass — with a deterministic exploded "parts catalogue" fallback so the screen is never blank.
• The trust rail: grade-coloured members (M15→M40 ramp + steel), per-system layer toggles, a live section cut through the structure, a cinematic reveal orbit, a 1.7 m human figure for scale, and click-any-part → provenance — grade, dimensions, confidence, and a jump straight to the source sheet with the element highlighted on the original drawing.

Validated on the 50 KL OHT set (parametric) and a Wardha minor-bridge GAD/RCC set (assembled scene).

Google Sheets export (user-connected)

A service account has no Drive quota, so it can't create files in a consumer account. Instead the BOQ lives in the app; to also push it to Sheets, the user connects their own sheet: an in-app modal shows the service-account email to share the sheet with (as Editor) and takes the pasted sheet link. The id is stored per-project in the DB, and each run writes a fresh tab into that sheet.

---

📂 Project Structure

construct.ai/              # Root project directory (Nirman.AI)
├── backend/               # FastAPI backend application
│   ├── app/
│   │   ├── main.py        # FastAPI endpoints & server routing
│   │   ├── db.py          # SQLite schema, connections, helpers, migrations
│   │   ├── ingest.py      # PDF render + per-sheet understanding index
│   │   ├── qna.py         # Phase 1: element-anchored Q&A (grounded in persisted context)
│   │   ├── structure.py   # Structure Mapper: fuse sheets into the project element index
│   │   ├── geometry.py    # Truth-3D: parametric OHT solids from the mapper's params/assemblies
│   │   ├── scene.py       # Truth-3D: typology-agnostic scene builder (buildings, bridges, fallback)
│   │   ├── pipeline.py    # 5-agent orchestration (per-sheet + project-level)
│   │   ├── sheets.py      # A5 gspread writer (user-connected sheet, tab per run)
│   │   ├── models.py      # Pydantic schemas (agent contracts + API shapes)
│   │   └── config.py      # Environment setup & application configurations
│   ├── storage/           # Local folder for uploaded PDFs and rendered page PNGs
│   ├── pyproject.toml     # uv configuration & Python dependencies
│   └── construct.db       # SQLite local database file (Phase 0 persistence)
│
├── frontend/              # Vite + React single page application
│   ├── src/               # React components, state, hooks, canvas renderer
│   ├── public/            # Static assets
│   ├── index.html         # HTML root page
│   ├── vite.config.ts     # Vite configuration
│   └── package.json       # React dependencies and scripts
│
└── docs/                  # Design, Product, and Specification materials
    ├── 01-problem-statement.md
    ├── 02-solution.md
    ├── 03-techstack.md
    ├── 04-implementation-plan.md
    ├── 05-structure-mapper.md   # project element index + project-level estimate design
    └── 06-visualization.md      # Truth-3D renderer design (truth layer vs comms layer)

---

⚡ Getting Started

Prerequisites

• Python 3.12+ (managed with uv)
• Node.js v18+ & npm
• A Google Gemini API Key (required for understanding + estimation)
• (Optional, for Google Sheets export) a Google Cloud service-account JSON at backend/sa.json, with the Google Sheets API enabled. No sheet id in .env is needed — the sheet is connected from inside the app (see Connecting a Google Sheet below).

---

Backend Setup

1. Navigate to the backend directory:

   cd backend

2. Configure Environment Variables: Copy the example and fill in your key (only the key is required):

   cp .env.example .env

   GEMINI_API_KEY="your-api-key-here"
   # optional render tuning:
   # RENDER_DPI=150
   # CROP_DPI=300

   With no key set, the spine still works (render + persist + read back); the understanding, Structure Mapper, and estimation layers activate once a key is present.

3. Install Dependencies and Run the Server:

   uv sync
   uv run uvicorn app.main:app --reload   # http://localhost:8000  (OpenAPI at /docs)

4. Connecting a Google Sheet (optional): Drop your service-account JSON at backend/sa.json and restart. Then in the app, open the Estimate tab → Export to Google Sheets: a modal shows the service-account email — create a Google Sheet, Share it with that email as Editor, and paste the sheet link. Each estimate then writes a fresh tab into your sheet. (Because a service account has no Drive quota, it writes into your sheet rather than creating its own.)

---

Frontend Setup

1. Navigate to the frontend directory:

   cd frontend

2. Install Dependencies:

   npm install

3. Start the Development Server:

   npm run dev

   The frontend will run at http://localhost:5173. Open this URL in your browser to view the canvas, upload drawings, and run estimations.

---

📈 Success Criteria Met

• Realblueprints: Tested on official Maharashtra PWD RCC drawings, GAD bridge drawings, and overhead water tank schedules.
• Numerical Integrity: Accurate calculation of concrete volumes and rebar weights verified via the Python Code Execution sandbox (no LLM math errors).
• Grounding: Rates fetched in real-time for materials (cement, reinforcement steel, Ready-Mix Concrete) in INR (₹) specific to Indian regions.
• Truth-3D reconstruction: The drawing set is rebuilt as an interactive, measurable 3D model — every solid sized from figured dimensions only, colour-coded by grade, with click-through provenance to the exact source sheet and element.
• State Persistence: SQLite guarantees that uploaded blueprints, annotated canvas points, user comments, and historic BOQ estimates persist reliably across application restarts.