Skip to content
mpreussePublic

About

Bulk loading of large data sets into Neo4j

Topics

Resources

Stars

24 stars

Watchers

2 watching

Forks

Repository files navigation

GraphIO

Tests PyPI License Neo4j Python Downloads

OGM and data loader for Neo4j with two main approaches:

  • OGM (Object Graph Mapper): Pydantic-based models with Neo4j integration for complex data models and applications
  • Datasets (NodeSet/RelationshipSet): Bulk data containers optimized for fast data loading and testing
  • Multi-Database Support: Full support for Neo4j Enterprise Edition multi-database feature

Documentation

Docs available at: https://graphio.readthedocs.io

Quick Start

Installation

Install graphio from PyPI:

pip install graphio

Install the latest version from GitHub:

pip install git+https://github.com/kaiserpreusse/graphio.git

Example

from graphio import NodeModel, Base
from neo4j import GraphDatabase

# Set up connection
driver = GraphDatabase.driver('neo4j://localhost:7687', auth=('neo4j', 'password'))
Base.set_driver(driver)
# Optional: Set target database (Enterprise Edition)
# Base.set_database('production')

# Define OGM model for structure and validation
class Person(NodeModel):
    _labels = ['Person']
    _merge_keys = ['email']
    name: str
    email: str

# Get bulk container directly from OGM model
people = Person.dataset()  # Automatically uses Person's labels and merge_keys

for person_data in large_dataset:
    # Create validated OGM instance and add directly
    person = Person(**person_data)  # Pydantic validation happens here
    people.add(person)  # Add validated instance to bulk dataset

people.create(driver)  # Bulk create with validation benefits

# Use OGM for application logic
alice = Person.match(Person.email == 'alice@example.com').first()

Development

Prerequisites

  • Python 3.10+
  • uv (recommended) or pip
  • Docker for running test databases

Setup

# Clone the repository
git clone https://github.com/kaiserpreusse/graphio.git
cd graphio

# Install dependencies
uv sync --extra dev

# Start Neo4j test databases
make localdb

Common Commands

# Run tests
make test

# Check code style  
make lint

# Format code
make format

# Fix linting issues and format
make fix

# Run all checks (lint + test)
make check

# Serve documentation locally
make docs

# See all available commands
make help

Testing

The test suite requires Docker containers running Neo4j. Start them with:

make localdb

Then run tests with:

make test
# or directly:
uv run pytest

All tests using the graph fixture will run against both Neo4j Community and Enterprise editions.

To run the suite against other servers, set GRAPHIO_TEST_TARGETS to a JSON array of targets. It replaces the default targets; name is used as the pytest parameter id. The suite wipes each target database before every test.

GRAPHIO_TEST_TARGETS='[
  {"name": "lts", "uri": "bolt://127.0.0.1:17687", "user": "neo4j", "password": "kwela-harness"},
  {"name": "cal", "uri": "bolt://127.0.0.1:27687", "user": "neo4j", "password": "kwela-harness"}
]' uv run pytest

Feedback

Please provide feedback, ideas and bug reports through GitHub issues.

About

Bulk loading of large data sets into Neo4j

Topics

Resources

Stars

24 stars

Watchers

2 watching

Forks

Releases

Packages

Used by

Contributors

Languages