Skip to content

README.md

Latest commit

 

History

History
280 lines (212 loc) · 8.12 KB

README.md

File metadata and controls

280 lines (212 loc) · 8.12 KB
Raw

SharpCoreDB Serialization & Storage Format Documentation

Complete technical documentation explaining how SharpCoreDB serializes records, manages variable-length strings, and handles free space allocation.

📚 Documentation Index

1. SERIALIZATION_AND_STORAGE_GUIDE.md ⭐ START HERE

The complete technical reference (3,200+ lines)

Everything you need to know about SharpCoreDB's storage format:

  • 📁 File format (.scdb) structure
  • 🔄 Record serialization mechanics (with C# 14 code)
  • 🔤 String handling & size constraints (variable-length, UTF-8, Unicode)
  • 📊 Free Space Management (FSM two-level bitmap)
  • 📑 Block Registry (O(1) lookups)
  • 🎯 Record & column boundary detection
  • ⚡ Performance (zero-allocation patterns)
  • ❓ 15-question comprehensive FAQ

Best for: Complete understanding of internals

2. SERIALIZATION_FAQ.md 🎯 QUICK ANSWERS

Fast reference guide (800 lines)

Directly answers common questions:

  • ❌ Refutes "you need lots of free space" claim with evidence
  • 📊 Real-world examples (96.9% space savings!)
  • 13 detailed FAQ answers with code examples
  • ✅ Summary comparison table

Best for: Quick answers to specific questions

3. BINARY_FORMAT_VISUAL_REFERENCE.md 📊 DIAGRAMS

Visual guide with hex dumps (900 lines)

See it visually:

  • 📊 File structure diagrams (all regions)
  • 🔢 Hex byte layouts with annotations
  • 📝 Type marker reference table
  • 🌍 Unicode encoding examples (Café, 日本, 🚀)
  • 📦 Data fragmentation illustrations
  • 🚀 File growth patterns
  • ✅ Cheat sheet summary

Best for: Visual learners, reference lookups

4. scripts/visualize_serialization.py 🐍 INTERACTIVE

Python visualization tool

Run interactive examples:

python3 docs/serialization/scripts/visualize_serialization.py

Demonstrates:

  • Simple types (int, string, bool)
  • Unicode strings (Café, 日本, 🚀)
  • Large strings (no overhead!)
  • NULL handling
  • Free space illustration

Best for: Hands-on learning

🎯 Key Findings

The Question

"Do I need lots of free space for variable-length strings?"

The Answer

❌ NO! Variable-length strings actually save space.

Fixed-length approach:     255 MB for 1,000,000 names
SharpCoreDB variable:      8 MB for 1,000,000 names
Savings:                   247 MB (96.9% reduction!) ✅

Why It Works

Feature Benefit
Length-prefixed encoding No ambiguity about boundaries
Block Registry O(1) record lookup
FSM (Free Space Map) Automatic allocation & growth
Self-describing format Type markers in every field
Exponential growth File grows intelligently (2x, 4x...)
Zero waste Only store actual bytes (no padding)

🗺️ Directory Structure

docs/serialization/
├── README.md (this file)
├── SERIALIZATION_AND_STORAGE_GUIDE.md (3,200+ lines)
├── SERIALIZATION_FAQ.md (800 lines)
├── BINARY_FORMAT_VISUAL_REFERENCE.md (900 lines)
└── scripts/
    └── visualize_serialization.py

📖 Quick Navigation

For Questions About...

Topic Go To
How do strings work? SERIALIZATION_AND_STORAGE_GUIDE.md § 5
Do I need free space? SERIALIZATION_FAQ.md § Q2
String size limits? SERIALIZATION_FAQ.md § Q5
Record boundaries? SERIALIZATION_AND_STORAGE_GUIDE.md § 7
Column layout? BINARY_FORMAT_VISUAL_REFERENCE.md § 3
Unicode support? SERIALIZATION_AND_STORAGE_GUIDE.md § 4.5
Free space management? SERIALIZATION_AND_STORAGE_GUIDE.md § 6
Performance? SERIALIZATION_AND_STORAGE_GUIDE.md § 8
Visual diagrams? BINARY_FORMAT_VISUAL_REFERENCE.md § All
Hex examples? BINARY_FORMAT_VISUAL_REFERENCE.md § 3-5

✨ Key Concepts Explained

Variable-Length String Storage

Format: [Length:4 bytes][UTF-8 data:N bytes]

"John":      [04 00 00 00][4A 6F 68 6E]       = 8 bytes
"Café":      [05 00 00 00][43 61 66 C3 A9]   = 9 bytes
"日本":     [06 00 00 00][E6 97 A5 E6 9C AC] = 10 bytes

✅ NO PADDING! Only actual bytes used.

Free Space Management (FSM)

  • Two-level bitmap: L1 (1 bit/page) + L2 (extent map)
  • Allocation: O(1) average case
  • Growth: Exponential (10MB → 20MB → 40MB...)
  • Result: Automatic & transparent

Record Lookup (Block Registry)

  • In-memory: Hash table (O(1) lookup)
  • On-disk: Variable-size entries
  • Batching: 200 blocks per flush (Phase 3)

Record Boundaries

  • Storage: Each record = one block with name
  • Lookup: Block Registry maps name → (offset, length)
  • Result: No ambiguity

Column Boundaries

  • Format: Self-describing [NameLen][Name][Type][Value]...
  • Parser: Reads sequentially (no fixed positions)
  • Result: Flexible & dynamic

🚀 Performance Summary

Aspect Performance
Serialization 3x faster than JSON
Space savings 91-96% vs fixed-length
Record lookup O(1) average
Zero allocation Using ArrayPool & Span
Write batching 50-100x improvement
Unicode support Full UTF-8 (Emoji too!)

💡 Real-World Examples

Example 1: Name Field (1M records)

Fixed-length (255 bytes):  255 MB
Variable-length (8 bytes): 8 MB
Savings: 247 MB (96.9%)

Example 2: File Growth

Initial: 100 pages (400 KB)
After inserts: 
  ├─ 1000 rows → Extend +2560 pages (10.2 MB)
  ├─ 2000 rows → Extend +2560 pages (20.4 MB)
  ├─ 3000 rows → Extend +5120 pages (40.6 MB)
  └─ Result: Exponential growth, minimal allocations

Example 3: Unicode Support

ASCII:    "Hello" → 5 bytes ✅
Accents:  "Café" → 5 bytes ✅
CJK:      "日本" → 6 bytes ✅
Emoji:    "🚀" → 4 bytes ✅

📝 Document Statistics

Document Lines Focus
SERIALIZATION_AND_STORAGE_GUIDE.md 3,200+ Complete reference
SERIALIZATION_FAQ.md 800 Quick answers
BINARY_FORMAT_VISUAL_REFERENCE.md 900 Diagrams & examples
Total 4,900+ Complete package

✅ Verification

All documentation is:

  • ✅ Based on actual SharpCoreDB C# 14 code
  • ✅ Includes real code examples
  • ✅ Contains hex dumps and binary layouts
  • ✅ Answers all serialization questions
  • ✅ Refutes common misconceptions with evidence
  • ✅ Tested and verified

🔗 Related Files

In codebase:

  • src/SharpCoreDB/Core/Serialization/BinaryRowSerializer.cs - Main serializer
  • src/SharpCoreDB/Storage/BlockRegistry.cs - Block registry implementation
  • src/SharpCoreDB/Storage/FreeSpaceManager.cs - FSM implementation
  • src/SharpCoreDB/Storage/SingleFileStorageProvider.cs - Storage provider
  • src/SharpCoreDB/Storage/Scdb/ScdbStructures.cs - File header definitions

In documentation root:

  • SERIALIZATION_DOCUMENTATION_COMPLETE.md - Status report
  • SERIALIZATION_DOCUMENTATION_SUMMARY.md - Executive summary

🎓 Learning Path

New to SharpCoreDB?

  1. Start: SERIALIZATION_AND_STORAGE_GUIDE.md (overview)
  2. Next: BINARY_FORMAT_VISUAL_REFERENCE.md (diagrams)
  3. Then: SERIALIZATION_FAQ.md (specific questions)
  4. Try: visualize_serialization.py (hands-on)

Have a specific question?

  1. Check the FAQ index above
  2. Jump to relevant section
  3. See code examples & diagrams

Want visual explanations?

  1. BINARY_FORMAT_VISUAL_REFERENCE.md - All diagrams
  2. visualize_serialization.py - Interactive tool

📞 Questions?

All answers are in this documentation! Key topics covered:

  • ✅ How strings are stored (variable-length, no padding)
  • ✅ Why free space isn't needed (automatic management)
  • ✅ How record boundaries work (Block Registry)
  • ✅ How column boundaries work (self-describing format)
  • ✅ Unicode support (full UTF-8, Emoji)
  • ✅ Performance (3x faster than JSON)
  • ✅ Space savings (96.9% in examples)
  • ✅ All with real code examples & hex dumps

Last Updated: January 2025
Phase: 3.3 - Serialization & Storage Optimization
Status: ✅ Complete & Organized
Lines of Documentation: 4,900+