Add deep merge support for array fields in .cecli.conf.yml

[szmania]

Feature Request

Add support for array fields in configuration files with different merge behaviors:

• .cecli.conf.yml: Deep merge on array fields (merge entries into existing arrays)
• .cecli/conf.yml: Shallow merge (existing behavior - replaces entire arrays)

Current Behavior

Both .cecli.conf.yml and .cecli/conf.yml perform shallow merges on array fields, meaning that when an array is specified in a config file, it completely replaces the existing array values rather than merging with them.

Problem

Users cannot additively extend configuration arrays (like read, rules, skills_paths, mcp-servers-files, subagents_paths, tools_paths, etc.) without completely overriding the defaults. This makes it difficult to incrementally add configuration entries.

Example Use Case

If the default configuration includes:

read:
  - "src/**/*.py"
  - "tests/**/*.py"

A user who wants to add docs/**/*.md must currently specify ALL files:

read:
  - "src/**/*.py"
  - "tests/**/*.py"
  - "docs/**/*.md"  # Must remember to include defaults

With deep merge, they could simply add:

read:
  - "docs/**/*.md"  # Automatically merged with defaults

Proposed Solution

1. Configuration Hierarchy: Maintain existing hierarchy where .cecli.conf.yml (in project root) has higher precedence than .cecli/conf.yml (in .cecli directory).

2. Merge Behavior:

   • .cecli.conf.yml: Perform deep merge on array fields. New entries are appended/merged with existing values rather than replacing them.
   • .cecli/conf.yml: Maintain shallow merge (current behavior) for backward compatibility.

3. Array Field Detection: Identify which configuration fields are arrays that should support deep merging:

   • read
   • rules
   • skills_paths
   • mcp-servers-files
   • subagents_paths
   • tools_paths
   • Other array fields as needed

4. Implementation Strategy:

   • Modify the configuration loading/merging logic to detect the config file type.
   • For .cecli.conf.yml, use a deep merge algorithm for arrays (append unique entries, preserve order).
   • For .cecli/conf.yml, continue using shallow merge (replace).
   • Ensure backward compatibility: existing configs continue to work unchanged.

5. Deduplication: When deep merging arrays, deduplicate entries based on value to avoid duplicates.

Technical Considerations

• Order Preservation: Deep merge should append new entries to the end of existing arrays, maintaining the order of both existing and new items.
• Deduplication: Use value-based deduplication (not object identity) to prevent duplicate entries.
• Nested Arrays: Determine if nested arrays should also be deep-merged (likely yes, but need to define scope).
• Performance: Deep merge should be efficient even with large configuration arrays.
• Error Handling: Handle malformed YAML gracefully with clear error messages.

Acceptance Criteria

• Identify all array configuration fields that need deep merge support.
• Implement deep merge algorithm for arrays in .cecli.conf.yml.
• Maintain shallow merge behavior for .cecli/conf.yml.
• Ensure backward compatibility - existing configs work without changes.
• Add comprehensive tests for:
  • Deep merge of arrays
  • Shallow merge preservation
  • Deduplication
  • Nested array handling
  • Error cases (malformed YAML, invalid arrays)
• Update documentation to explain the new merge behavior and configuration options.
• Add examples showing how to use the new feature.

Jira Task

This issue is aligned with Jira task CLI-57.

Configuration File Precedence

Reminder of existing precedence (highest to lowest):

1. Command-line arguments
2. .cecli.conf.yml (project root) - Will support deep merge on arrays
3. .cecli/conf.yml (in .cecli directory) - Shallow merge (existing)
4. Default values