LLM-FACETS

LLM FActuality Cross-EvaluaTion System — An open-source, privacy-preserving evaluation framework with a browser-accessible interface and a plugin architecture, designed to help practitioners make informed, transparent decisions about LLM quality.

LLM-FACETS structures evaluation around three practitioner profiles — Technical Experts, Domain Experts, and Compliance Officers — inspired by the stakeholder categories identified in the EU AI Act (Art. 14) and the NIST AI Risk Management Framework as responsible for human oversight of AI systems.

🌟 Key Features

• Privacy-Preserving Architecture: Deterministic metrics run entirely within the self-hosted server — no evaluation data leaves your infrastructure. LLM-judge metrics contact external APIs explicitly, with full user control.
• BYOK (Bring Your Own Key): API keys are stored exclusively in the browser via IndexedDB with origin isolation. Keys are transmitted per-request and never persisted server-side.
• PII Anonymization Pipeline: Configurable anonymization (local NER via compromise.js, self-hosted Presidio, or model-based NER) redacts sensitive entities before any outbound API call.
• 10 LLM Providers: Unified support for OpenAI, Gemini, Anthropic, DeepSeek, Groq, Mistral, xAI, MiniMax, ZAI, and Devana.
• 18 Metric Variants: BLEU, ROUGE (6 variants), METEOR, BERTScore (P/R/F1), LogProbs, G-Eval, Jury, Evaluation Topologies, and RAG Triad (Faithfulness, Answer Relevance, Context Relevance).
• LogProbs Visualization: Token-by-token confidence analysis with 5-tier color coding for hallucination detection (native support for OpenAI, DeepSeek, xAI).
• Dataset Management: High-performance dataset import and querying via DuckDB and Parquet — all queries execute in-process with no external transmission.
• LLM-as-a-Judge: Qualitative evaluation with G-Eval, multi-judge Jury (multi-provider consensus with bias mitigation), and evaluation topologies (pointwise, pairwise with position-bias mitigation, debate).
• RAG Triad: End-to-end evaluation for RAG systems (Faithfulness, Answer Relevance, Context Relevance) with per-claim breakdowns and reasoning chains.
• Statistical Analysis Dashboard: Radar charts, box plots, correlation heatmaps, hallucination histograms, per-sample deep dive, multi-run comparison (up to 5 runs).
• Benchmark Mode: Batch evaluation of datasets across all metrics with exportable JSON/CSV audit trails.
• Cross-Validated Implementations: All deterministic metrics verified against canonical Python reference libraries (NLTK, HuggingFace evaluate) with absolute error < 10⁻⁵ across 267 test cases.
• Internationalization: English, French, German, and Italian.

📊 Available Metrics

Traditional NLP Metrics

• BLEU — N-gram precision matching (machine translation).
• ROUGE — 6 variants (ROUGE-1, ROUGE-2, ROUGE-L, ROUGE-W, ROUGE-S, ROUGE-SU) for recall-oriented evaluation (summarization).
• METEOR — Semantic alignment with stemming and synonym support.

Neural & Semantic Metrics

• BERTScore — Contextual embedding similarity powered by Transformers.js (all-MiniLM-L6-v2), running as a server-side singleton within the Next.js process. No text or embeddings are transmitted externally.
• Log Probabilities (LogProbs):
  • Token-level confidence scores with 5-tier color coding (Very High ≥ 0.8, High ≥ 0.6, Medium ≥ 0.4, Low ≥ 0.2, Very Low < 0.2).
  • Hover to reveal exact log-probability and top-k alternative tokens.
  • Native support for OpenAI, DeepSeek, and xAI.

LLM as a Judge

• G-Eval — Multi-criteria evaluation (Coherence, Fluency, etc.) with Chain-of-Thought and probabilistic scoring.
• Jury (Multi-Judges) — Consensus across multiple models with 5 aggregation strategies: mean, median, trimmed mean, weighted, majority vote.
• Evaluation Topologies:
  • Pointwise — Absolute scoring (O(N)).
  • Pairwise — A/B comparison and ranking.
  • Debate — Argumentative judge consensus.

RAG Evaluation (Retrieval-Augmented Generation)

• Faithfulness — Verifies each atomic statement in the answer against the source context.
• Answer Relevance — Measures the alignment between the answer and the original question.
• Context Relevance — Evaluates the signal-to-noise ratio in the retrieved context.

💾 Dataset Management

The application integrates a high-performance dataset manager:

• DuckDB & Parquet — In-process analytical database engine. All queries (schema introspection, random sampling, SQL filtering) execute within the Next.js server process — never transmitted externally.
• HuggingFace — Automatic download of standard datasets.
• SQuAD v2.0 — Reading-comprehension Question Answering benchmark.
• PsiloQA — Multilingual QA with complexity levels and hallucination labels.
• SelfAware — Knowledge-boundary and refusal evaluation (1,047 samples).
• HaluEval — Faithfulness and correctness evaluation with explicit hallucination labels (12,322 samples).
• Explorer — Web UI to visualize, sample, and query datasets (Settings → Manage Datasets).
• Custom Datasets — Upload CSV/Parquet files for proprietary evaluation corpora.

🚀 Installation & Configuration

Prerequisites

• Bun 1.0+ (recommended) or Node.js 18+

Installation

# Install dependencies
bun install

# Configure environment
cp .env.example .env.local

API Key Configuration

You can configure API keys in two ways:

1. Environment Variables: Edit .env.local for server-side defaults.
2. User Interface: Click the Settings icon in the app to add your personal API keys (stored securely in your browser via IndexedDB).

# OpenAI
OPENAI_API_KEY=sk-...

# Google Gemini
GEMINI_API_KEY=...

# Anthropic (Claude)
ANTHROPIC_API_KEY=sk-ant-...

# Groq
GROQ_API_KEY=...

# Mistral
MISTRAL_API_KEY=...

# Other Supported Providers
DEEPSEEK_API_KEY=...
XAI_API_KEY=...
MINIMAX_API_KEY=...
ZAI_API_KEY=...
DEVANA_API_KEY=...

Running the App

# Development server
bun run dev

# Build for production
bun run build
bun run start

Open http://localhost:3000 to access the application.

🗺️ Application Routes

Route	Description
/	Home — interactive metrics playground
/metrics	Metrics category overview
/metrics/bleu	BLEU interactive demo
/metrics/rouge	ROUGE interactive demo
/metrics/bertscore	BERTScore interactive demo
/metrics/logprobs	LogProbs visualization
/metrics/llm-judge	LLM-as-a-Judge (Faithfulness, Relevance, G-Eval, Jury)
/benchmark	Batch dataset evaluation

All routes are prefixed with a language segment (e.g. /en/, /fr/, /de/, /it/).

🔌 API Reference

Method	Route	Description
POST	/api/evaluate	Batch evaluation (BLEU, ROUGE, METEOR, BERTScore)
POST	/api/generate	LLM text generation
POST	/api/bleu	BLEU score
POST	/api/rouge	ROUGE variants
POST	/api/bertscore	BERTScore
POST	/api/logprobs	Token-level log probabilities
POST	/api/geval	G-Eval with Chain-of-Thought
POST	/api/faithfulness	Faithfulness evaluation
POST	/api/answer-relevance	Answer relevance
POST	/api/context-relevance	Context relevance
POST	/api/rag/evaluate	Unified RAG Triad
POST	/api/evaluation/pointwise	Pointwise scoring
POST	/api/evaluation/pairwise	Pairwise A/B comparison
POST	/api/evaluation/rank	Multi-option ranking
POST	/api/evaluation/jury	Multi-judge consensus
GET	/api/models	List all models by provider (cached 1 h)
DELETE	/api/models?provider=<name>	Clear provider model cache
GET	/api/providers/status	Provider health status
GET	/api/logprobs/providers	Providers supporting logprobs
GET	/api/datasets	List available datasets
GET	/api/datasets/status	Dataset download status
GET	/api/datasets/query	Query dataset samples

🔒 Data Sovereignty

LLM-FACETS enforces a clear data sovereignty boundary:

Metric Category	Execution Location	Outbound Data?
BLEU, ROUGE, METEOR	Next.js server-local (pure computation)	❌ None
BERTScore	Next.js server-local (Transformers.js singleton)	❌ None
LogProbs	External LLM API (provider-native)	✅ Query text
G-Eval, Eval Topologies	External LLM API	✅ Evaluation text
Jury	External LLM APIs (one call per judge)	✅ Evaluation text
Faithfulness, AR, CR	External LLM API (RAG chain)	✅ Evaluation text

For LLM-judge metrics that require external API calls, the anonymization pipeline can redact PII before transmission.

📊 Statistical Analysis Dashboard

Beyond per-sample evaluation, LLM-FACETS includes a multi-tab analysis dashboard:

• Overview: Radar chart (9 primary metrics), box plots (quartile distributions), hallucination risk histogram (bins below 0.7 flagged).
• Correlation: Pearson correlation heatmap, pairwise RAG scatter plots, jury agreement histogram.
• Deep Dive: Per-sample heatmap (30/page), BERTScore precision-vs-recall scatter, execution timeline (up to 5 metrics).
• Multi-Run Comparison: Comparative radar chart and grouped bar chart overlaying up to 5 benchmark runs.

All visualizations are generated client-side from IndexedDB-stored results with no server-side data transmission.

🏗️ Technical Architecture

• Framework: Next.js 16 (App Router)
• Language: TypeScript
• Machine Learning: Transformers.js (server-side singleton inference)
• Database: DuckDB (in-process, for datasets)
• Styling: Tailwind CSS 4
• Privacy: BYOK model, IndexedDB key storage, configurable PII anonymization
• API: Unified architecture for LLM providers, server-side model cache (TTL 1h)

📁 Project Structure

src/
├── app/                 # Next.js routes and API endpoints
├── components/          # React components (Visualizers, Demos)
├── hooks/               # Custom hooks (useModels, useDatasets)
├── lib/
│   ├── datasets/        # DuckDB handles and Dataset Services
│   ├── llm_api/         # Unified clients for LLM providers
│   └── metrics/         # Evaluation metrics implementation
├── types/               # TypeScript definitions
└── i18n/                # Multilingual configuration

👨‍💻 Developer Guide

Want to contribute? Here is how to extend the application.

1. Add a New LLM Provider

The architecture is modular and centralized. Follow these steps to add a provider (e.g., MyAI):

1. Create Provider Class: Create src/lib/llm_api/myai.ts by extending LLMProvider.
   • Implement call() for text generation and logprobs.
   • Implement getAvailableModels() to list models.
2. Register Provider:
   • Add the type to ProviderName (src/lib/llm_api/factory.ts).
   • Register the constructor in providerConstructors (src/lib/llm_api/factory.ts).
3. Update Models API: Add provider initialization in src/app/api/models/route.ts so it appears in the selector.
4. Configuration: Add the API key (MYAI_API_KEY) to .env.local.

2. Add a New Metric

1. Business Logic: Implement the calculation algorithm in src/lib/metrics/mymetric.ts.
2. API Endpoint: Create src/app/api/mymetric/route.ts to expose the calculation via a standardized API.
3. User Interface:
   • Create a component src/components/MyMetricDemo.tsx.
   • Use existing UI components (MetricCard, InputSection, etc.) for visual consistency.
   • Add the component to the main grid (src/app/page.tsx or src/components/home/EvaluationMetricsGrid.tsx).

✅ Verification & Performance

• 267 validation cases covering all deterministic metrics.
• Deterministic metrics (BLEU, ROUGE, METEOR, BERTScore) match Python reference implementations (NLTK, HuggingFace evaluate) with absolute error < 10⁻⁵.
• LLM-based metrics (RAG Triad, G-Eval, Jury) maintain σ < 0.05 across repeated runs (n=5, gpt-4o-mini).
• Performance benchmark: 13,369 samples processed in 24.7s total — text generation: 1.3 ms/sample, metric computation: 0.5 ms/sample/metric.

📄 Citation

If you use LLM-FACETS in your research, please cite:

@article{llmfacets2026,
  title={LLM-FACETS: A Privacy-Preserving Framework for Evaluating LLM Transparency and Accountability},
  journal={},
  year={}
}