Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
76 changes: 45 additions & 31 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,8 +47,8 @@ Try the live demo to explore GTFS data, view routes with colors, and see trip sc
- ✅ Full support for both GTFS-RT `time` and `delay` fields in stop time updates

### Smart Caching
- ✅ **Automatic caching** - Enabled by default, no configuration needed
- ✅ **Platform-optimized storage** - IndexedDB (browser) or FileSystem (Node.js)
- ✅ **Optional caching** - Copy cache implementations from `examples/cache/`
- ✅ **Platform-specific stores** - IndexedDBCacheStore (browser) or FileSystemCacheStore (Node.js)
- ✅ **Smart invalidation** - Based on file checksum, size, version, and library version
- ✅ **Automatic expiration** - Configurable (default: 7 days)
- ✅ **Cache management API** - Get stats, clean expired entries, clear cache
Expand Down Expand Up @@ -866,25 +866,39 @@ if (alert.cause === AlertCause.ACCIDENT) {

### Smart Caching

The library automatically caches processed GTFS databases to dramatically speed up subsequent loads. The first load processes the GTFS zip file (~5-10 seconds), but subsequent loads use the cached database (<1 second).
The library supports optional caching of processed GTFS databases to dramatically speed up subsequent loads. The first load processes the GTFS zip file (~5-10 seconds), but subsequent loads use the cached database (<1 second).

#### Automatic Caching (Default Behavior)
#### Setting Up Caching

Caching is enabled by default with no configuration needed:
Cache store implementations are available in `examples/cache/`. Copy the appropriate implementation to your project:

**Browser - IndexedDB:**
```typescript
// Copy examples/cache/IndexedDBCacheStore.ts to your project
import { GtfsSqlJs } from 'gtfs-sqljs';
import { IndexedDBCacheStore } from './IndexedDBCacheStore';

const cache = new IndexedDBCacheStore();

// First load: processes GTFS zip file and caches the result
const gtfs = await GtfsSqlJs.fromZip('gtfs.zip');
const gtfs = await GtfsSqlJs.fromZip('gtfs.zip', { cache });

// Second load: uses cached database (much faster!)
const gtfs2 = await GtfsSqlJs.fromZip('gtfs.zip');
const gtfs2 = await GtfsSqlJs.fromZip('gtfs.zip', { cache });
```

**Node.js - FileSystem:**
```typescript
// Copy examples/cache/FileSystemCacheStore.ts to your project
import { GtfsSqlJs } from 'gtfs-sqljs';
import { FileSystemCacheStore } from './FileSystemCacheStore';

const cache = new FileSystemCacheStore({ dir: './.cache/gtfs' });

const gtfs = await GtfsSqlJs.fromZip('gtfs.zip', { cache });
```

**Platform-specific storage:**
- **Browser**: Uses IndexedDB (supports large databases, async)
- **Node.js**: Uses FileSystem (default: `~/.cache/gtfs-sqljs/`)
**Note:** `FileSystemCacheStore` uses Node.js built-in modules (`fs`, `path`, `os`) and is **NOT compatible** with browser or React Native environments.

#### Cache Invalidation

Expand Down Expand Up @@ -916,35 +930,36 @@ const gtfs2 = await GtfsSqlJs.fromZip('gtfs.zip', {
- You want to force cache refresh
- Testing different processing configurations

#### Custom Cache Store
#### Cache Store Options

**Browser - Custom IndexedDB:**
**IndexedDBCacheStore options:**
```typescript
import { GtfsSqlJs, IndexedDBCacheStore } from 'gtfs-sqljs';
import { IndexedDBCacheStore } from './IndexedDBCacheStore';

const cache = new IndexedDBCacheStore({
dbName: 'my-app-gtfs-cache'
dbName: 'my-app-gtfs-cache' // Custom database name
});

const gtfs = await GtfsSqlJs.fromZip('gtfs.zip', { cache });
```

**Node.js - Custom Directory:**
**FileSystemCacheStore options:**
```typescript
import { GtfsSqlJs, FileSystemCacheStore } from 'gtfs-sqljs';
import { FileSystemCacheStore } from './FileSystemCacheStore';

const cache = new FileSystemCacheStore({
dir: './my-cache-dir'
dir: './my-cache-dir' // Custom cache directory
});

const gtfs = await GtfsSqlJs.fromZip('gtfs.zip', { cache });
```

#### Cache Management

Cache management methods require passing the cache store instance:

**Get cache statistics:**
```typescript
const stats = await GtfsSqlJs.getCacheStats();
import { IndexedDBCacheStore } from './IndexedDBCacheStore';

const cache = new IndexedDBCacheStore();
const stats = await GtfsSqlJs.getCacheStats(cache);

console.log(`Total entries: ${stats.totalEntries}`);
console.log(`Active entries: ${stats.activeEntries}`);
Expand All @@ -954,7 +969,7 @@ console.log(`Total size: ${stats.totalSizeMB} MB`);

**List cache entries:**
```typescript
const entries = await GtfsSqlJs.listCache();
const entries = await GtfsSqlJs.listCache(cache);

entries.forEach(entry => {
console.log(`Key: ${entry.key}`);
Expand All @@ -967,27 +982,26 @@ entries.forEach(entry => {
**Clean expired entries:**
```typescript
// Remove entries older than 7 days (default)
const deletedCount = await GtfsSqlJs.cleanExpiredCache();
const deletedCount = await GtfsSqlJs.cleanExpiredCache(cache);
console.log(`Deleted ${deletedCount} expired entries`);

// Custom expiration time (3 days)
const threeDays = 3 * 24 * 60 * 60 * 1000;
await GtfsSqlJs.cleanExpiredCache(undefined, threeDays);
await GtfsSqlJs.cleanExpiredCache(cache, threeDays);
```

**Clear all cache:**
```typescript
await GtfsSqlJs.clearCache();
await GtfsSqlJs.clearCache(cache);
```

#### Disable Caching
#### Without Caching

To disable caching, explicitly pass `null`:
By default, caching is disabled. Simply omit the `cache` option:

```typescript
const gtfs = await GtfsSqlJs.fromZip('gtfs.zip', {
cache: null // No caching
});
const gtfs = await GtfsSqlJs.fromZip('gtfs.zip');
// No caching - GTFS is processed fresh each time
```

#### Custom Cache Expiration
Expand Down
154 changes: 19 additions & 135 deletions dist/index.d.mts
Original file line number Diff line number Diff line change
Expand Up @@ -654,14 +654,11 @@ interface GtfsSqlJsOptions {
onProgress?: ProgressCallback;
/**
* Optional: Cache store for persisting processed GTFS databases
* Use IndexedDBCacheStore (browser) or FileSystemCacheStore (Node.js)
* or implement your own CacheStore
* Implement your own CacheStore or copy one from examples/cache/:
* - IndexedDBCacheStore (browser)
* - FileSystemCacheStore (Node.js only)
*
* If not provided, caching is enabled by default with:
* - IndexedDBCacheStore in browsers
* - FileSystemCacheStore in Node.js
*
* Set to `null` to disable caching
* If not provided, caching is disabled.
*/
cache?: CacheStore | null;
/**
Expand All @@ -682,6 +679,7 @@ declare class GtfsSqlJs {
private SQL;
private realtimeFeedUrls;
private stalenessThreshold;
private lastRealtimeFetchTimestamp;
/**
* Private constructor - use static factory methods instead
*/
Expand Down Expand Up @@ -922,6 +920,11 @@ declare class GtfsSqlJs {
* Get current staleness threshold
*/
getStalenessThreshold(): number;
/**
* Get timestamp of the last successful realtime data fetch and insertion
* @returns Unix timestamp in seconds, or null if no realtime data has been fetched
*/
getLastRealtimeFetchTimestamp(): number | null;
/**
* Fetch and load GTFS Realtime data from configured feed URLs or provided URLs
* @param urls - Optional array of feed URLs. If not provided, uses configured feed URLs
Expand Down Expand Up @@ -970,10 +973,10 @@ declare class GtfsSqlJs {
debugExportAllStopTimeUpdates(): StopTimeUpdate[];
/**
* Get cache statistics
* @param cacheStore - Cache store to query (optional, auto-detects if not provided)
* @param cacheStore - Cache store to query (required)
* @returns Cache statistics including size, entry count, and age information
*/
static getCacheStats(cacheStore?: CacheStore): Promise<{
static getCacheStats(cacheStore: CacheStore): Promise<{
totalEntries: number;
activeEntries: number;
expiredEntries: number;
Expand All @@ -984,28 +987,23 @@ declare class GtfsSqlJs {
}>;
/**
* Clean expired cache entries
* @param cacheStore - Cache store to clean (optional, auto-detects if not provided)
* @param cacheStore - Cache store to clean (required)
* @param expirationMs - Expiration time in milliseconds (default: 7 days)
* @returns Number of entries deleted
*/
static cleanExpiredCache(cacheStore?: CacheStore, expirationMs?: number): Promise<number>;
static cleanExpiredCache(cacheStore: CacheStore, expirationMs?: number): Promise<number>;
/**
* Clear all cache entries
* @param cacheStore - Cache store to clear (optional, auto-detects if not provided)
* @param cacheStore - Cache store to clear (required)
*/
static clearCache(cacheStore?: CacheStore): Promise<void>;
static clearCache(cacheStore: CacheStore): Promise<void>;
/**
* List all cache entries
* @param cacheStore - Cache store to query (optional, auto-detects if not provided)
* @param cacheStore - Cache store to query (required)
* @param includeExpired - Include expired entries (default: false)
* @returns Array of cache entries with metadata
*/
static listCache(cacheStore?: CacheStore, includeExpired?: boolean): Promise<CacheEntry[]>;
/**
* Get the default cache store for the current environment
* @returns Default cache store or null if unavailable
*/
private static getDefaultCacheStore;
static listCache(cacheStore: CacheStore, includeExpired?: boolean): Promise<CacheEntry[]>;
}

/**
Expand All @@ -1030,120 +1028,6 @@ interface IndexDefinition {
}
declare const GTFS_SCHEMA: TableSchema[];

/**
* IndexedDB-based cache store for browsers
*
* Stores GTFS databases in IndexedDB for fast access on subsequent loads.
* Suitable for large databases (100s of MB to several GB depending on browser limits).
*
* @example
* ```typescript
* import { GtfsSqlJs, IndexedDBCacheStore } from 'gtfs-sqljs';
*
* const cache = new IndexedDBCacheStore();
* const gtfs = await GtfsSqlJs.fromZip('gtfs.zip', {
* cache,
* cacheVersion: '1.0'
* });
* ```
*/
declare class IndexedDBCacheStore implements CacheStore {
private dbName;
private storeName;
private version;
constructor(options?: CacheStoreOptions);
/**
* Open IndexedDB connection
*/
private openDB;
/**
* Get a cached database with metadata
*/
get(key: string): Promise<CacheEntryWithData | null>;
/**
* Store a database in cache
*/
set(key: string, data: ArrayBuffer, metadata: CacheMetadata): Promise<void>;
/**
* Check if a cache entry exists
*/
has(key: string): Promise<boolean>;
/**
* Delete a specific cache entry
*/
delete(key: string): Promise<void>;
/**
* Clear all cache entries
*/
clear(): Promise<void>;
/**
* List all cached entries
*/
list(): Promise<CacheEntry[]>;
}

/**
* File system-based cache store for Node.js
*
* Stores GTFS databases as files on disk for fast access on subsequent loads.
* Suitable for any size database (limited only by disk space).
*
* @example
* ```typescript
* import { GtfsSqlJs, FileSystemCacheStore } from 'gtfs-sqljs';
*
* const cache = new FileSystemCacheStore({ dir: './.cache/gtfs' });
* const gtfs = await GtfsSqlJs.fromZip('gtfs.zip', {
* cache,
* cacheVersion: '1.0'
* });
* ```
*/
declare class FileSystemCacheStore implements CacheStore {
private cacheDir;
constructor(options?: CacheStoreOptions);
/**
* Get the cache directory path (lazy initialization)
*/
private getCacheDir;
/**
* Ensure cache directory exists
*/
private ensureCacheDir;
/**
* Get file path for a cache key
*/
private getFilePath;
/**
* Get metadata file path for a cache key
*/
private getMetadataPath;
/**
* Get a cached database with metadata
*/
get(key: string): Promise<CacheEntryWithData | null>;
/**
* Store a database in cache
*/
set(key: string, data: ArrayBuffer, metadata: CacheMetadata): Promise<void>;
/**
* Check if a cache entry exists
*/
has(key: string): Promise<boolean>;
/**
* Delete a specific cache entry
*/
delete(key: string): Promise<void>;
/**
* Clear all cache entries
*/
clear(): Promise<void>;
/**
* List all cached entries
*/
list(): Promise<CacheEntry[]>;
}

/**
* Compute SHA-256 checksum of data
* Uses Web Crypto API (available in both browser and Node.js 18+)
Expand Down Expand Up @@ -1202,4 +1086,4 @@ declare function getCacheStats(entries: CacheEntry[]): {
newestEntry: number | null;
};

export { type Agency, type AgencyFilters, type Alert, AlertCause, AlertEffect, type AlertFilters, type Attribution, type CacheEntry, type CacheEntryWithData, type CacheMetadata, type CacheStore, type CacheStoreOptions, type Calendar, type CalendarDate, type ColumnDefinition, CongestionLevel, DEFAULT_CACHE_EXPIRATION_MS, type EntitySelector, type FareAttribute, type FareRule, type FeedInfo, FileSystemCacheStore, type Frequency, GTFS_SCHEMA, type GeoJsonFeatureCollection, GtfsSqlJs, type GtfsSqlJsOptions, type IndexDefinition, IndexedDBCacheStore, type Level, OccupancyStatus, type Pathway, type Position, type RealtimeConfig, type Route, type RouteFilters, ScheduleRelationship, type Shape, type ShapeFilters, type Stop, type StopFilters, type StopTime, type StopTimeEvent, type StopTimeFilters, type StopTimeRealtime, type StopTimeUpdate, type StopTimeUpdateFilters, type StopTimeWithRealtime, type TableSchema, type TimeRange, type Transfer, type TranslatedString, type Trip, type TripFilters, type TripRealtime, type TripUpdate, type TripUpdateFilters, type TripWithRealtime, type VehicleDescriptor, type VehiclePosition, type VehiclePositionFilters, VehicleStopStatus, computeChecksum, computeZipChecksum, filterExpiredEntries, generateCacheKey, getCacheStats, isCacheExpired };
export { type Agency, type AgencyFilters, type Alert, AlertCause, AlertEffect, type AlertFilters, type Attribution, type CacheEntry, type CacheEntryWithData, type CacheMetadata, type CacheStore, type CacheStoreOptions, type Calendar, type CalendarDate, type ColumnDefinition, CongestionLevel, DEFAULT_CACHE_EXPIRATION_MS, type EntitySelector, type FareAttribute, type FareRule, type FeedInfo, type Frequency, GTFS_SCHEMA, type GeoJsonFeatureCollection, GtfsSqlJs, type GtfsSqlJsOptions, type IndexDefinition, type Level, OccupancyStatus, type Pathway, type Position, type RealtimeConfig, type Route, type RouteFilters, ScheduleRelationship, type Shape, type ShapeFilters, type Stop, type StopFilters, type StopTime, type StopTimeEvent, type StopTimeFilters, type StopTimeRealtime, type StopTimeUpdate, type StopTimeUpdateFilters, type StopTimeWithRealtime, type TableSchema, type TimeRange, type Transfer, type TranslatedString, type Trip, type TripFilters, type TripRealtime, type TripUpdate, type TripUpdateFilters, type TripWithRealtime, type VehicleDescriptor, type VehiclePosition, type VehiclePositionFilters, VehicleStopStatus, computeChecksum, computeZipChecksum, filterExpiredEntries, generateCacheKey, getCacheStats, isCacheExpired };
Loading