Skip to content

About

Multi-provider abstraction for LLMOps including Comet Opik, Phoenix Arize, and Langfuse. OtelSemConv for Agents: https://plexusone.github.io/omniobserve/semconvagent.html

Topics

Resources

Contributing

Stars

1 star

Watchers

0 watching

Forks

Latest commit

 

History

214 Commits

Folders and files

Repository files navigation

OmniObserve

Go CI Go Lint Go SAST Docs Docs Visualization License

A unified Go library for observability. OmniObserve provides vendor-agnostic abstraction layers that enable you to instrument your applications once and seamlessly switch between different observability backends without code changes.

Two Provider Systems

Package Purpose Providers
llmops LLM/ML observability Opik, Langfuse, Phoenix, slog
observops App observability (metrics, traces, logs) OTLP, Datadog, New Relic, Dynatrace

Features

LLM Observability (llmops)

  • 🔗 Unified Interface: Single API for tracing, evaluation, prompts, and datasets across all providers
  • 🔄 Provider Agnostic: Switch between Opik, Langfuse, and Phoenix without changing your code
  • 🔍 Full Tracing: Trace LLM calls with spans, token usage, and cost tracking
  • 📊 Evaluation Support: Run metrics and add feedback scores to traces
  • 📦 Dataset Management: Create and manage evaluation datasets
  • 📝 Prompt Versioning: Store and version prompt templates (provider-dependent)

App Observability (observops)

  • 🚀 One-Call Setup: observops.Setup wires metrics, traces, and logs and returns native OpenTelemetry handles — full ecosystem interop, no wrapper tax
  • 📊 Full Telemetry: Metrics, distributed traces, and real OpenTelemetry logs
  • 🔀 Composable Exporters: Prometheus pull and OTLP push (gRPC or HTTP), plus stdout for debugging
  • 🌐 HTTP Instrumentation: One-line server middleware and client transport
  • 📝 slog Integration: Structured logs exported via OpenTelemetry, fanned out to the console
  • 📈 Vendor-Agnostic Drivers: Open("otlp"|"datadog"|"newrelic"|"dynatrace") for vendor-switching behind a single interface

Semantic Conventions

  • 🏷️ Agent Conventions: OpenTelemetry semantic conventions for agentic AI (gen_ai.agent.*)
  • 🛤️ Journey Conventions: User journey tracking for product analytics (session.*, page.*, ui.*)
  • 📐 Observability Specs: RED/USE metrics, class-based SLOs, and OpenSLO integration

Cross-Language Support

  • 🟦 TypeScript SDK: @omniobserve/core package mirrors Go types and semantic conventions
  • 🔀 Context Propagation: Automatic trace/span context propagation via context.Context
  • ⚙️ Functional Options: Clean, extensible configuration using the options pattern

Installation

go get github.com/plexusone/omniobserve

Quick Start

LLM Observability (llmops)

package main

import (
    "context"
    "log"

    "github.com/plexusone/omniobserve/llmops"
    _ "github.com/agentplexus/go-opik/llmops"  // Register Opik provider
)

func main() {
    provider, err := llmops.Open("opik",
        llmops.WithAPIKey("your-api-key"),
        llmops.WithProjectName("my-project"),
    )
    if err != nil {
        log.Fatal(err)
    }
    defer provider.Close()

    ctx := context.Background()

    ctx, trace, _ := provider.StartTrace(ctx, "chat-workflow",
        llmops.WithTraceInput(map[string]any{"query": "Hello, world!"}),
    )
    defer trace.End()

    _, span, _ := provider.StartSpan(ctx, "gpt-4-completion",
        llmops.WithSpanType(llmops.SpanTypeLLM),
        llmops.WithModel("gpt-4"),
    )

    span.SetUsage(llmops.TokenUsage{TotalTokens: 18})
    span.End()
}

App Observability (observops)

Use Setup for one-call bootstrap that returns native OpenTelemetry handles (recommended):

package main

import (
    "context"
    "log"
    "net/http"

    "github.com/plexusone/omniobserve/observops"
)

func main() {
    ctx := context.Background()

    tel, err := observops.Setup(ctx,
        observops.WithServiceName("my-service"),
        observops.WithPrometheus(),               // pull endpoint: tel.MetricsHandler
        observops.WithEndpoint("localhost:4317"), // OTLP push (gRPC)
        observops.WithInsecure(),
    )
    if err != nil {
        log.Fatal(err)
    }
    defer tel.Shutdown(ctx)

    // Native OpenTelemetry handles — full API, full ecosystem interop:
    reqs, _ := tel.Meter.Int64Counter("requests.total")
    reqs.Add(ctx, 1)

    ctx, span := tel.Tracer.Start(ctx, "handle-request")
    defer span.End()

    tel.Logger.InfoContext(ctx, "request processed", "user_id", "123")

    // Serve Prometheus metrics and instrument HTTP in one line each:
    mux := http.NewServeMux()
    mux.Handle("/metrics", tel.MetricsHandler)
    handler := tel.Middleware("api")(mux)
    _ = handler
}

Metrics, traces, and logs are enabled by default; disable any with WithMetrics, WithTraces, or WithLogs. Prometheus pull and OTLP push compose; WithStdout mirrors to stdout.

Prefer the vendor-neutral driver registry when you want to switch backends behind one interface (Open("otlp"|"datadog"|"newrelic"|"dynatrace")) — see observops/README.

Supported Providers

LLM Providers (llmops)

Provider Package Description
Opik go-opik/llmops Comet Opik - Open-source, full-featured
Langfuse omniobserve/llmops/langfuse Cloud & self-hosted, batch ingestion
Phoenix go-phoenix/llmops Arize Phoenix - OpenTelemetry-based
slog omniobserve/llmops/slog Local structured logging for development/debugging

App Observability Providers (observops)

Provider Package Description
OTLP omniobserve/observops/otlp OpenTelemetry Protocol - vendor-agnostic
Datadog omniobserve/observops/datadog Datadog APM via OTLP
New Relic omniobserve/observops/newrelic New Relic via OTLP
Dynatrace omniobserve/observops/dynatrace Dynatrace via OTLP

LLM Provider Capabilities

Feature Opik Langfuse Phoenix slog
Tracing ✅ ✅ ✅ ✅
Evaluation ✅ ✅ ✅ ❌
Prompts ✅ Partial ❌ ❌
Datasets ✅ ✅ Partial ❌
Experiments ✅ ✅ Partial ❌
Streaming ✅ ✅ Planned ❌
Distributed Tracing ✅ ❌ ✅ ❌
Cost Tracking ✅ ✅ ❌ ❌
OpenTelemetry ❌ ❌ ✅ ❌

observops Capabilities

Feature OTLP Datadog New Relic Dynatrace
Metrics ✅ ✅ ✅ ✅
Traces ✅ ✅ ✅ ✅
Logs ✅ ✅ ✅ ✅
slog Handler ✅ ✅ ✅ ✅

Architecture

omniobserve/
├── omniobserve.go       # Main package with re-exports
├── omniobserve/         # Unified entry point with HTTP middleware
├── llmops/              # LLM observability interfaces
│   ├── llmops.go        # Core interfaces (Provider, Tracer, Evaluator, etc.)
│   ├── trace.go         # Trace and Span interfaces
│   ├── types.go         # Data types (EvalInput, Dataset, Prompt, etc.)
│   ├── options.go       # Functional options
│   ├── provider.go      # Provider registration system
│   ├── errors.go        # Error definitions
│   ├── metrics/         # Evaluation metrics (hallucination, relevance, etc.)
│   ├── langfuse/        # Langfuse provider adapter
│   └── slog/            # slog provider adapter for local development
├── observops/           # App observability interfaces
│   ├── observops.go     # Core interfaces (Provider, Meter, Tracer)
│   ├── options.go       # Functional options
│   ├── otlp/            # OTLP provider (vendor-agnostic)
│   ├── datadog/         # Datadog provider
│   ├── newrelic/        # New Relic provider
│   └── dynatrace/       # Dynatrace provider
├── agentops/            # Agent operations monitoring
│   ├── ent/             # Ent ORM schemas
│   ├── middleware/      # HTTP middleware
│   └── postgres/        # PostgreSQL storage
├── semconv/             # OpenTelemetry semantic conventions
│   ├── agent/           # Agentic AI conventions (gen_ai.agent.*)
│   └── journey/         # User journey conventions (session.*, page.*, ui.*)
├── specs/               # Observability specifications
│   ├── classes/         # Class-based SLO management
│   ├── red/             # RED metrics (Rate, Errors, Duration)
│   ├── use/             # USE metrics (Utilization, Saturation, Errors)
│   ├── golden/          # 4 Golden Signals mapping
│   ├── openslo/         # OpenSLO SLI/SLO definitions
│   ├── otel/            # OTel semantic conventions
│   ├── recorder/        # Integration with observops.Provider
│   └── schema/          # JSON Schema generation
├── sloghandler/         # slog.Handler implementations
│   ├── dual.go          # Local + remote handler
│   ├── fanout.go        # Multi-handler fanout
│   └── trace.go         # Trace context injection
├── integrations/        # Integrations with external libraries
│   ├── omnillm/         # OmniLLM observability hook (separate module)
│   ├── sevaluation/     # Structured evaluation integration
│   └── observability/   # HTTP client/server middleware helpers
├── examples/            # Usage examples
│   └── evaluation/      # Metrics evaluation example
├── mlops/               # ML operations interfaces (experiments, model registry)
├── sdk/                 # Provider-specific SDKs
│   └── langfuse/        # Langfuse Go SDK
└── typescript/          # TypeScript SDK
    └── packages/core/   # @omniobserve/core - mirrors Go types and semconv

# LLM provider adapters in standalone SDKs:
# github.com/agentplexus/go-opik/llmops      # Opik provider
# github.com/agentplexus/go-phoenix/llmops   # Phoenix provider

Core Interfaces

Provider

The main interface that all observability backends implement:

type Provider interface {
    Tracer            // Trace/span operations
    Evaluator         // Evaluation and feedback
    PromptManager     // Prompt template management
    DatasetManager    // Test dataset management
    ProjectManager    // Project/workspace management
    AnnotationManager // Span/trace annotations
    io.Closer

    Name() string
}

Trace and Span

type Trace interface {
    ID() string
    Name() string
    StartSpan(ctx context.Context, name string, opts ...SpanOption) (context.Context, Span, error)
    SetInput(input any)
    SetOutput(output any)
    SetMetadata(metadata map[string]any)
    AddTag(key, value string)
    AddFeedbackScore(ctx context.Context, name string, score float64, opts ...FeedbackOption) error
    End(opts ...EndOption)
}

type Span interface {
    ID() string
    TraceID() string
    Name() string
    Type() SpanType
    StartSpan(ctx context.Context, name string, opts ...SpanOption) (context.Context, Span, error)
    SetInput(input any)
    SetOutput(output any)
    SetModel(model string)
    SetProvider(provider string)
    SetUsage(usage TokenUsage)
    End(opts ...EndOption)
}

Span Types

const (
    SpanTypeGeneral   SpanType = "general"
    SpanTypeLLM       SpanType = "llm"
    SpanTypeTool      SpanType = "tool"
    SpanTypeRetrieval SpanType = "retrieval"
    SpanTypeAgent     SpanType = "agent"
    SpanTypeChain     SpanType = "chain"
    SpanTypeGuardrail SpanType = "guardrail"
)

Usage Examples

Using Different Providers

// Opik
import _ "github.com/agentplexus/go-opik/llmops"
provider, _ := llmops.Open("opik", llmops.WithAPIKey("..."))

// Langfuse
import _ "github.com/plexusone/omniobserve/llmops/langfuse"
provider, _ := llmops.Open("langfuse",
    llmops.WithAPIKey("sk-lf-..."),
    llmops.WithEndpoint("https://cloud.langfuse.com"),
)

// Phoenix
import _ "github.com/agentplexus/go-phoenix/llmops"
provider, _ := llmops.Open("phoenix",
    llmops.WithEndpoint("http://localhost:6006"),
)

Nested Spans

ctx, trace, _ := provider.StartTrace(ctx, "rag-pipeline")
defer trace.End()

// Retrieval span
ctx, retrievalSpan, _ := provider.StartSpan(ctx, "vector-search",
    llmops.WithSpanType(llmops.SpanTypeRetrieval),
)
// ... perform retrieval ...
retrievalSpan.SetOutput(documents)
retrievalSpan.End()

// LLM span
ctx, llmSpan, _ := provider.StartSpan(ctx, "generate-response",
    llmops.WithSpanType(llmops.SpanTypeLLM),
    llmops.WithModel("gpt-4"),
)
// ... call LLM ...
llmSpan.SetUsage(llmops.TokenUsage{
    PromptTokens:     150,
    CompletionTokens: 50,
    TotalTokens:      200,
})
llmSpan.End()

Adding Feedback Scores

// Add a score to a span
span.AddFeedbackScore(ctx, "relevance", 0.95,
    llmops.WithFeedbackReason("Response directly addressed the query"),
    llmops.WithFeedbackCategory("quality"),
)

// Add a score to a trace
trace.AddFeedbackScore(ctx, "user_satisfaction", 0.8)

Working with Datasets

// Create a dataset
dataset, _ := provider.CreateDataset(ctx, "test-cases",
    llmops.WithDatasetDescription("Test cases for RAG evaluation"),
)

// Add items
provider.AddDatasetItems(ctx, "test-cases", []llmops.DatasetItem{
    {
        Input:    map[string]any{"query": "What is Go?"},
        Expected: map[string]any{"answer": "Go is a programming language..."},
    },
})

Working with Prompts (Opik)

// Create a versioned prompt
prompt, _ := provider.CreatePrompt(ctx, "chat-template",
    `You are a helpful assistant. User: {{.query}}`,
    llmops.WithPromptDescription("Main chat template"),
)

// Get a prompt
prompt, _ := provider.GetPrompt(ctx, "chat-template")

// Render with variables
rendered := prompt.Render(map[string]any{"query": "Hello!"})

Configuration Options

Client Options

llmops.WithAPIKey("...")           // API key for authentication
llmops.WithEndpoint("...")         // Custom endpoint URL
llmops.WithWorkspace("...")        // Workspace/organization name
llmops.WithProjectName("...")      // Default project name
llmops.WithHTTPClient(client)      // Custom HTTP client
llmops.WithTimeout(30 * time.Second)
llmops.WithDisabled(true)          // Disable tracing (no-op mode)
llmops.WithDebug(true)             // Enable debug logging

Trace Options

llmops.WithTraceProject("...")
llmops.WithTraceInput(input)
llmops.WithTraceOutput(output)
llmops.WithTraceMetadata(map[string]any{...})
llmops.WithTraceTags(map[string]string{...})
llmops.WithThreadID("...")

Span Options

llmops.WithSpanType(llmops.SpanTypeLLM)
llmops.WithSpanInput(input)
llmops.WithSpanOutput(output)
llmops.WithSpanMetadata(map[string]any{...})
llmops.WithModel("gpt-4")
llmops.WithProvider("openai")
llmops.WithTokenUsage(usage)
llmops.WithParentSpan(parentSpan)

Error Handling

The library provides typed errors for common conditions:

if errors.Is(err, llmops.ErrMissingAPIKey) {
    // Handle missing API key
}

if llmops.IsNotFound(err) {
    // Handle not found
}

if llmops.IsRateLimited(err) {
    // Handle rate limiting
}

Direct SDK Access

For provider-specific features, you can use the underlying SDKs directly:

import "github.com/plexusone/omniobserve/sdk/langfuse"  // Langfuse SDK
import "github.com/agentplexus/go-opik"                   // Opik SDK
import "github.com/agentplexus/go-phoenix"                // Phoenix SDK

OmniLLM Integration

OmniObserve provides an integration with OmniLLM, a multi-LLM abstraction layer. This allows you to automatically instrument all LLM calls made through OmniLLM with any OmniObserve provider.

go get github.com/plexusone/omniobserve/integrations/omnillm
package main

import (
    "github.com/plexusone/omnillm"
    omnillmhook "github.com/plexusone/omniobserve/integrations/omnillm"
    "github.com/plexusone/omniobserve/llmops"
    _ "github.com/agentplexus/go-opik/llmops"
)

func main() {
    // Initialize a OmniObserve provider
    provider, _ := llmops.Open("opik",
        llmops.WithAPIKey("your-api-key"),
        llmops.WithProjectName("my-project"),
    )
    defer provider.Close()

    // Create the observability hook
    hook := omnillmhook.NewHook(provider)

    // Attach to your OmniLLM client
    client := omnillm.NewClient(
        omnillm.WithObservabilityHook(hook),
    )

    // All LLM calls through this client are now automatically traced
}

The hook automatically captures:

  • Model and provider information
  • Input messages and output responses
  • Token usage (prompt, completion, total)
  • Streaming responses
  • Errors

The hook also automatically creates traces when none exists in context, ensuring all LLM calls are properly traced.

Requirements

  • Go 1.24.5 or later

Contributing

Contributions are welcome! Please feel free to submit issues and pull requests.

License

See LICENSE for details.

About

Multi-provider abstraction for LLMOps including Comet Opik, Phoenix Arize, and Langfuse. OtelSemConv for Agents: https://plexusone.github.io/omniobserve/semconvagent.html

Topics

Resources

Contributing

Stars

1 star

Watchers

0 watching

Forks

Releases

Contributors

Languages