Wye: Intelligent Infrastructure Management System

Wye is a stateless, Git-driven infrastructure management system that separates persistent configuration from ephemeral, live-scanned observed state, instead of relying on a persistent global state file.

Configurable state is stored in Git and defines the desired system. Observed state is generated from live infrastructure and stored locally as ephemeral files, representing non-configurable attributes such as allocated IPs, provider IDs, and runtime metadata.

Observed state acts as a cache of real infrastructure. It may become stale over time, as full scans are not performed on every change. However, state for affected resources is refreshed during reconciliation, and a full or partial rescan can be triggered manually when needed.

This model eliminates the need for a persistent state backend: no shared state can become stale, corrupted, or contested. Infrastructure management becomes closer to working with Git — configurations are versioned, changes are auditable, drift is detectable on demand, and rollbacks are straightforward.

Nickel Overview

Wye configuration is written in Nickel, a typed, functional configuration language designed for correctness and composability. Key properties of Nickel that make it suitable for Wye include:

• Contracts (schemas) Every config file is validated against a provider-supplied contract using the | operator. Invalid configurations are rejected at evaluation time, before any infrastructure is touched.
• Imports The import keyword allows config files to include other files, including *.obs.json outputs from the reconciler. This is the mechanism by which inter-resource dependencies are expressed and autogenerated attributes (such as IDs and IPs) are threaded through the configuration graph.
• Laziness and purity Nickel is a pure functional language. Evaluation is deterministic and side-effect-free: the same inputs always produce the same desired-state record.
• Merge and overrides Nickel's record merge semantics allow shared definitions to be composed with resource-specific overrides, avoiding duplication.

State Model

Each resource in Wye has two distinct types of state:

• Configuration — defined declaratively in *.cfg.ncl files using Nickel and stored in a versioned Git repository. This represents the desired configuration of the resource.
• Observed state — stored in *.obs.json files, generated dynamically from live infrastructure during reconciliation or via manually triggered scans. This state is ephemeral, excluded from version control, and represents non-configurable attributes assigned by external systems (e.g., allocated IP addresses, provider-specific IDs, runtime metadata).

Observed state is not user-defined and cannot be configured. It reflects properties discovered from the actual infrastructure, which are required for correct operation and integration with external APIs. During reconciliation, Wye retrieves the observed state and provides it as input to the Nickel definitions of dependent resources, then deterministically computes and applies the necessary changes to converge the system.

Component Model

Wye is structured around three major layers:

• Configuration layer Nickel configuration files authored by operators, expressing desired state and inter-resource dependencies. Validated at evaluation time by provider schemas. Includes:
  • *.cfg.ncl — resource configurations
  • *.ncl — common/shared definitions
  • other files of any type — used for inclusion in Nickel expressions
• Provider layer Per-resource-type plugins that serve as adapters between Wye and the corresponding API or SDK. They provide the implementation to iterate over, observe, create, update, and delete resources of their kind.
• Reconciler Core engine: evaluates configurations, invokes providers to observe resources, manages the dependency graph, performs resource actions, and writes resulting *.obj.json outputs.

Providers

Wye manages resources through providers, which act as adapters between the system and the corresponding external API or SDK. Each provider knows how to:

• Observe the current state of its resources.
• Create, update, and delete resources to match the desired configuration.

Currently, Wye ships with support for:

• AWS EC2 — managing virtual machines on Amazon Web Services.
• Docker — managing containers and images locally or remotely.

The provider system is extensible: new providers can be added to manage other cloud services (e.g., Azure, S3), network devices, or other systems with accessible APIs or SDKs. This allows Wye to integrate with a wide range of infrastructure while keeping the core engine unchanged.

Repository Structure

A Wye repository follows a conventional layout, with some mandatory files:

• wye.ncl — mandatory root entry point. Declares all providers, global settings, and imports provider configuration files. Maps provider IDs to provider kinds.
• provider/ — conventional directory name for per-provider configuration files. Each file contains provider-specific configuration, and multiple provider configurations of the same kind are allowed.

Resource configuration files can be organized in any directory structure chosen by the operator.

The following files are associated with a resource:

• <name>.<provider-id>.cfg.ncl — desired-state configuration (configurable).
• <name>.<provider-id>.obs.json — observed state (ephemeral, written by the reconciler).
• <name>.<provider-id>.cfg.json — generated JSON representation of the configuration for unknown resources discovered during scanning. Wye cannot regenerate .cfg.ncl from the JSON representation because there is no unique source form. Indeed, a configuration may be composed from multiple imported files, include computed values, and rely on arbitrary composition patterns. Different .ncl structures can evaluate to the same resulting configuration, so the original source cannot be reconstructed unambiguously.
• <name>.<provider-id>.cfgdiff.json — generated difference between desired configuration and actual state, created during scanning. Existing .cfg.ncl files cannot be updated for the same reason.

The <name> part of resource files is the resource name, combined with <provider-id> to form a unique resource ID. The mapping from provider ID to provider kind is defined in wye.ncl.

Ignore Files

• .gitignore — controls what goes into Git. Typical content:

*.obs.json
untracked/

• .wyeignore - controls which parts of the repository are scanned during reconciliation. Typical content:

.git*
.wyeignore
untracked/

Supported Commands

Wye provides several commands to manage infrastructure and keep Git in sync with the current system state.

• stage Applies the specified worktree changes to the actual infrastructure and adds them to the Git index. The index always reflects the current system state.

  Example:

  wye stage <filename>...

  If a specified file is a configuration file and has no changes in the worktree, the resource state is still enforced to match the configuration.

• sync Updates the observed state file (*.obs.json) for the specified configuration file.

  Example:

  wye sync <configuration-filename>...

  If the -d flag is passed, Wye also checks for configuration drift. Any drift results in a corresponding *.cfgdiff.json file being generated.

• scan-sync Scans all resources for the specified providers (or all providers if none is specified) and updates the observed state files. For each configuration file, an updated observed state (*.obs.json) is written.

  Example:

  wye scan-sync <provider-id>...

  The -d flag can also be applied, with the same meaning as for sync. For unknown resources (resources that exist in the infrastructure but are not defined in the Git repository), Wye generates corresponding *.obs.json and *.cfgdiff.json files in the untracked/ directory. In this case, the *.cfgdiff.json file represents the difference between “no configuration” (the resource is missing from Git) and the full current configuration of the resource, effectively capturing the entire configurable state. These files are only created when the -d flag is specified.

Rollback Model

Because all desired state is stored in Git, rolling back to any previous configuration is equivalent to checking out an earlier repository revision. The reconciler then treats the difference between the current configurable state and the checked-out desired state as the change set to apply. Crucially, rollback is not a destructive global reset. The reconciler applies only the changes necessary to bring the current observed state into alignment with the target revision's desired state. Resources that have not changed between the current and target revision are left untouched.

Demo

👉 Repository: https://github.com/ababo/wye-demo

🎥 Video walkthrough: https://youtu.be/mt0Fm0gpwsE