Skip to content

[CONTP-1965] feat(ddi): Expand supported log configurations in DDI - #3331

Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 3 commits into
mainfrom
mathew.estafanous/ddi-logs-config-parity
Aug 7, 2026
Merged

[CONTP-1965] feat(ddi): Expand supported log configurations in DDI#3331
gh-worker-dd-mergequeue-cf854d[bot] merged 3 commits into
mainfrom
mathew.estafanous/ddi-logs-config-parity

Conversation

@Mathew-Estafanous

@Mathew-Estafanous Mathew-Estafanous commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

Expands DatadogInstrumentation log configuration with additional Agent-supported options for TCP/UDP listeners, syslog parsing, automatic multiline detection, file fingerprinting, and maximum message size.

Intentionally omitted configurations
Configuration Reason
tls Active security review
config_id, include_user_units, exclude_user_units, include_matches, exclude_matches, container_mode, default_application_name Node-scoped journald
query Node-scoped Windows
process_raw_message Structured node tailers
debug_attr_parsing Diagnostic-only
auto_multi_line_sample_size, auto_multi_line_match_threshold Legacy-only
auto_multi_line.timestamp_detector_match_threshold, auto_multi_line.pattern_table_match_threshold, custom sample match_threshold Avoid float CRD fields
experimental_adaptive_sampling, experimental_noisy_log_detection Experimental features
channel, channel_tags, channel_tags_mutex Runtime channel state

Motivation

spec.config.logs currently supports only a subset of Agent log configuration, preventing some Autodiscovery annotations from being migrated to DDI without losing behavior. This closes the practical workload-scoped parity gaps tracked in CONTP-1965.

Minimum Agent Versions

Are there minimum versions of the Datadog Agent and/or Cluster Agent required?

  • Agent: vX.Y.Z
  • Cluster Agent: vX.Y.Z

Describe your test plan

Example CR using all the new fields.

apiVersion: datadoghq.com/v1alpha1
kind: DatadogInstrumentation
metadata:
  name: fake-logs-normal
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: fake-app
  config:
    logs:
      - containerName: fake-app
        type: file
        path: /var/log/fake-app/*.log
        source: fake-app
        service: fake-app
        auto_multi_line_detection: true
        auto_multi_line_detection_custom_samples:
          - sample: "2026-08-06 ERROR request failed"
          - regex: "^panic:"
            label: start_group
        auto_multi_line:
          enable_json_detection: true
          enable_datetime_detection: true
          tokenizer_max_input_bytes: 256
          pattern_table_max_size: 128
          enable_json_aggregation: true
          tag_aggregated_json: true
          stack_trace_parsers:
            - go
        fingerprint_config:
          fingerprint_strategy: line_checksum
          count: 10
          count_to_skip: 0
          max_bytes: 4096
        max_message_size_bytes: 1000000

Checklist

  • PR has at least one valid label: bug, enhancement, refactoring, documentation, tooling, and/or dependencies
  • PR has a milestone or the qa/skip-qa label
  • All commits are signed (see: signing commits)

@linear-code

linear-code Bot commented Aug 5, 2026

Copy link
Copy Markdown

CONTP-96

@Mathew-Estafanous Mathew-Estafanous self-assigned this Aug 5, 2026
@Mathew-Estafanous Mathew-Estafanous added this to the v1.30.0 milestone Aug 5, 2026
@datadog-prod-us1-4

This comment has been minimized.

@Mathew-Estafanous Mathew-Estafanous added the enhancement New feature or request label Aug 6, 2026
@Mathew-Estafanous
Mathew-Estafanous marked this pull request as ready for review August 6, 2026 16:59
@Mathew-Estafanous
Mathew-Estafanous requested a review from a team August 6, 2026 16:59
@Mathew-Estafanous
Mathew-Estafanous requested review from a team as code owners August 6, 2026 16:59

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b00ea32fb0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread config/manager/kustomization.yaml Outdated
@Mathew-Estafanous
Mathew-Estafanous force-pushed the mathew.estafanous/ddi-logs-config-parity branch from b00ea32 to 3b6faa8 Compare August 6, 2026 17:02
@Mathew-Estafanous
Mathew-Estafanous requested review from a team and removed request for a team August 6, 2026 17:02
@zhuminyi

zhuminyi commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

/dd-review codex

@Mathew-Estafanous
Mathew-Estafanous force-pushed the mathew.estafanous/ddi-logs-config-parity branch from 3b6faa8 to fa0ad73 Compare August 7, 2026 00:05
@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot merged commit 1518bb5 into main Aug 7, 2026
67 checks passed
@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot deleted the mathew.estafanous/ddi-logs-config-parity branch August 7, 2026 01:31
gh-worker-dd-mergequeue-cf854d Bot pushed a commit to DataDog/datadog-agent that referenced this pull request Aug 7, 2026
…using DatadogInstrumentation (#54534)

<!--Please give us some feedback on your experience writing this PR ! https://app.datadoghq.com/forms/43db4c02-6837-400c-8083-692e141b1b88 !-->

### What does this PR do?

Points the agent to a newer datadog-operator version containing extended log configurations in the DatadogInstrumentation CRD (DataDog/datadog-operator#3331).

### Motivation

Log feature parity between DDI CRD and the annotation/file based log configuration

### Describe how you validated your changes

1. Deployed operator changes with CRD: DataDog/datadog-operator#3331
2. Deployed agent and instrumentation CR using new fields.

```yaml
apiVersion: datadoghq.com/v1alpha1
kind: DatadogInstrumentation
metadata:
  name: redis
  namespace: cache
spec:
  targetRef:
    apiVersion: apps/v1
    kind: StatefulSet
    name: redis
  config:
    logs:
      - containerName: redis
        source: redis
        service: redis
        tags:
          - service:redis
          - env:injector-dev
        auto_multi_line_detection: true
        auto_multi_line_detection_custom_samples:
          - regex: '^\d+:[MSC] \d{2} [A-Z][a-z]{2} \d{4}'
            label: redis_timestamp
          - sample: "1:M 06 Aug 2026 15:04:05.000 * Ready to accept connections"
            label: redis_startup
        auto_multi_line:
          enable_json_detection: true
          enable_datetime_detection: true
          tokenizer_max_input_bytes: 2048
          pattern_table_max_size: 100
          enable_json_aggregation: false
          tag_aggregated_json: false
          stack_trace_parsers: []
        fingerprint_config:
          fingerprint_strategy: line_checksum
          count: 2
          count_to_skip: 0
          max_bytes: 4096
        max_message_size_bytes: 262144
```

3. Confirmed log check scheduled with configurations

<img width="561" height="553" alt="image" src="https://github.com/user-attachments/assets/b3cd311f-6f01-4f81-a142-f44eb8ae8a1c" />

### Additional Notes 

Co-authored-by: mathew.estafanous <mathew.estafanous@datadoghq.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants