Skip to content

CONFIG.md

Latest commit

 

History

History
124 lines (101 loc) · 5.73 KB

CONFIG.md

File metadata and controls

124 lines (101 loc) · 5.73 KB
Raw

Configuration reference

TPack has three flavors of YAML config:

  1. Dataset configs — configs/*.yaml. Drive tpack-eval (transform, train, evaluate). Define which span attributes are primary vs. dependent for a given dataset.
  2. Edge collector configs — fed to otelcol-tpack via --config=file:.... Configure the exporter (compressor).
  3. Backend collector configs — same binary, different config. Configure the receiver (decompressor).

Dataset configs (configs/*.yaml)

Loaded by tpack-eval (Go) and tpack_eval (Python) — both must agree on the field names.

name: tpack                       # Algorithm tag (kept for backward compat with output dir naming)
primary_attributes:               # Low-cardinality identity-defining attributes
  - service.name
  - span.kind
  - operation.name
  - status.code
  # ... add more as needed; the canonical otel-demo set has 22
dependent_attributes:             # High-cardinality passthrough attributes
  - http.url
  - net.peer.port
metadata_predictor: statistical   # "statistical" is the only shipped impl
offset_value: ratio               # "ratio" or "absolute"
offset_model: regression          # "regression" or "percentile"
Field Type Default Notes
name string tpack Used as the approach prefix in output directory names.
primary_attributes list[string] required Span attributes whose Cartesian product defines a span signature. Cardinality budget: ~22 attrs / ≤50 unique values each.
dependent_attributes list[string] [] Span attributes regenerated by the dependent-attribute predictor. Each adds a per-pair categorical distribution.
metadata_predictor string statistical Predictor implementation key.
offset_value string ratio How child gap/duration is parameterized: ratio (fraction of parent) or absolute (microseconds).
offset_model string regression OLS regression vs. percentile-based estimation.

Ablation YAMLs under configs/ablation/ follow the same schema; their name field uses the variant key (e.g. tpack_otel_demo_no_service_name) so the run script can produce output dirs like tpack_no_service_name_1/.

Edge collector config (compressor)

Fed via --config=file:./collector-edge.yaml to otelcol-tpack.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

exporters:
  tpack:
    flush_interval_seconds: 60       # required, > 0
    max_buffered_traces: 50000       # required, > 0
    model_server_port: 9090          # 0 disables gRPC streaming
    output_path: /var/lib/tpack       # optional; write each model to disk
    primary_attributes:              # optional override; defaults to canonical 4
      - service.name
      - operation.name
      - span.kind
      - status.code
    dependent_attributes:
      - http.status_code
    stratified_sampling: true        # split normal vs. error traces
    random_seed: 42

  otlp_grpc:
    endpoint: tempo:4317
    tls: { insecure: true }

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [tpack, otlp_grpc]   # tee to a low-rate head if you want exact traces too
Field Required Notes
flush_interval_seconds yes How often (seconds) to train + emit a model. Lower = lower visibility latency, higher CPU.
max_buffered_traces yes Force flush if bucket fills before interval. Defends against memory blowup under spikes.
model_server_port no If > 0, opens gRPC server-streaming on this port. If 0, you must set output_path.
output_path no Directory to write each model as tpack_model_<timestamp>.pb.
primary_attributes no Override the canonical primary-attribute set. Match the dataset config.
dependent_attributes no Defaults to empty. Each entry adds a per-pair categorical distribution.
stratified_sampling no If true, train separately on status.code = ERROR traces — preserves error-rate fidelity.
random_seed no For deterministic regeneration; set when reproducing experiments.

Validation rules (enforced by Validate() in exporter/tpackexporter/config.go): flush_interval_seconds > 0, max_buffered_traces > 0, model_server_port >= 0, model_server_port > 0 || output_path != "".

Backend collector config (decompressor)

receivers:
  tpack:
    source_type: grpc                                 # "grpc" or "filesystem"
    model_server_endpoint: edge-collector:9090         # grpc only
    # input_path: /var/lib/tpack/model.pb              # filesystem only
    # continuous_generation: false                     # filesystem only

exporters:
  otlp:
    endpoint: tempo:4317
    tls: { insecure: true }

service:
  pipelines:
    traces:
      receivers: [tpack]
      exporters: [otlp]
Field Required Notes
source_type yes grpc (live stream) or filesystem (replay a .pb model file).
model_server_endpoint grpc only host:port of the edge collector's gRPC server. Receiver auto-retries with exponential backoff (1s → 30s).
input_path filesystem only Absolute path inside the container to a serialized model.
continuous_generation no If true (filesystem mode), re-emits the same model in a loop. Useful for replay-driven dashboards.

Validation: source_type ∈ {grpc, filesystem}; grpc requires model_server_endpoint; filesystem requires input_path.

Builder config (cmd/otelcol-tpack/builder-config.yaml)

Used by the OpenTelemetry Collector Builder (OCB) to produce a custom collector binary. Most users don't touch this — make build invokes it. To add other components (e.g. a sampler before TPack), append them to processors: / extensions: and rerun make build.