Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/workflows/ci.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,8 @@ jobs:
- run: docker run --rm recense:test --version
- run: docker run --rm --entrypoint syft recense:test version
- run: docker run --rm --entrypoint grype recense:test version
- name: Exclude EKS backing storage
run: scripts/smoke-storage.sh
- name: End-to-end host and Docker collection
run: scripts/smoke-docker.sh
- uses: azure/setup-helm@v4.3.0
Expand Down
26 changes: 22 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ curl -fsSLO https://github.com/opf/recense/releases/latest/download/install.sh
sudo sh install.sh
```

The installer verifies the release archive checksum, installs binaries under `/opt/recense/bin`, and starts `recense.service`. Rerun with a release tag to upgrade or downgrade: `sudo sh install.sh v0.2.0`.
The installer verifies the release archive checksum, installs binaries under `/opt/recense/bin`, and starts `recense.service`. Rerun with a release tag to upgrade or downgrade: `sudo sh install.sh v0.2.1`.

```sh
sudo systemctl status recense
Expand All @@ -23,7 +23,7 @@ sudo tail -f /var/log/recense/events.jsonl

Set `RECENSE_ENVIRONMENT`, `RECENSE_CLUSTER`, and optional `RECENSE_HOST_ID` / `RECENSE_HOST_NAME` in `/etc/recense/environment`, then restart the service. Host IDs default to `/etc/machine-id`; cloned images must receive unique machine IDs. Set labels before collecting production data.

Recense sends a heartbeat every minute. Scans start after a random delay of up to 15 minutes, then repeat 24 hours after completion with another random delay. Set `--interval` and `--jitter` to adjust scheduling; `--once` skips the delay. Host and container scans run sequentially with one scanner worker. Each Syft/Grype invocation has a 45-minute timeout and a 128 MiB output limit. Scanner failure does not produce a clean result.
Recense sends a heartbeat every minute. Scans start after a random delay of up to 15 minutes, then repeat 24 hours after completion with another random delay. Set `--interval` and `--jitter` to adjust scheduling; `--once` skips the delay. Full package snapshots are emitted on the first successful inventory of each target, then at most once every seven days (`--inventory-interval=168h`). Every daily cycle still builds a fresh local inventory, checks all detected packages for CVEs, and emits individual findings and a severity summary. This reduces log traffic, not scanner work. Host and container scans run sequentially with one scanner worker. Each Syft/Grype invocation has a 45-minute timeout and a 128 MiB output limit. Scanner failure does not produce a clean result.

For a single run or custom scheduling:

Expand All @@ -40,7 +40,7 @@ For EC2-backed EKS or other Linux Kubernetes nodes with containerd and CRI v1:

```sh
helm upgrade --install recense \
https://github.com/opf/recense/releases/download/v0.2.0/recense-0.2.0.tgz \
https://github.com/opf/recense/releases/download/v0.2.1/recense-0.2.1.tgz \
--namespace recense --create-namespace \
--set clusterName=production-eks \
--set environment=production
Expand Down Expand Up @@ -99,7 +99,7 @@ Do not count every finding in a time window as a current vulnerability: repeated
## Coverage and limits

- Host scans read the host filesystem. Container scans read each running container's filesystem, including its writable layer. No image download or export is required. Containers sharing an image still need separate scans because their writable layers can differ.
- Scans exclude `/proc`, `/sys`, `/dev`, `/run`, `/tmp`, `/var/log`, `/var/cache`, `/var/lib/docker`, `/var/lib/containerd`, `/var/lib/kubelet`, and `/var/lib/recense`. Container scans also exclude mounted volumes. Software under excluded paths is not covered.
- Scans exclude `/proc`, `/sys`, `/dev`, `/run`, `/tmp`, `/var/log`, `/var/cache`, `/var/lib/docker`, `/var/lib/containerd`, `/var/lib/kubelet`, `/var/lib/recense`, and `/mnt/k8s-disks` (EKS local storage backing directories). Container scans also exclude mounted volumes. Software under excluded paths is not covered.
- Container roots must be visible in the host mount namespace. Recense verifies the root inode and process start time before and after inventory collection. Missing or changed roots fail the scan; they never fall back to another filesystem. A live filesystem is not an atomic snapshot.
- Container scan identity is the container ID. Image IDs are descriptive metadata. Inventoried binaries are installed versions, not proof that a process is running them.
- Ubuntu update checks use `python3-apt` and existing APT metadata. Recense does not refresh APT; keep the host's normal refresh timers enabled. Metadata refresh time is included when the Ubuntu success stamp exists. Missing `python3-apt` is reported as a failed check.
Expand All @@ -125,3 +125,21 @@ Go dependencies are pinned in `go.mod` / `go.sum`. Scanner versions are pinned i
Before a release, run CI and verify both VM fixture collection and container runtime discovery. Unit tests use fake scanners and local Docker/CRI API servers; they do not inspect the developer's host or containers. CI also runs real Syft/Grype scans against a disposable Docker host and writable container filesystems, and the Helm DaemonSet in a disposable kind cluster.

Apache-2.0. Bundled scanners and their licenses are listed in [THIRD_PARTY.md](THIRD_PARTY.md).

### Weekly inventory and daily CVE results

Package snapshots and CVE results have independent freshness. Use the latest successful `software.inventory_completed` scan for installed-package history. Use the latest successful `vulnerability.scan_completed` scan for current CVE results. A daily CVE scan may have no emitted package snapshot; its `inventory_snapshot_sent` field says whether one was emitted. An absent inventory marker on that scan does not mean the host has no packages.

A small schedule file at `/var/lib/recense/inventory-snapshots.json` records when each target's snapshot was emitted. Writes use atomic replacement. Missing or corrupt state triggers fresh snapshots; a write failure reports `agent.error` without stopping CVE checks. State loss may repeat snapshots. This file is not a delivery receipt: collector outages can still lose logs. One Recense process should own each state file. Use a fresh state file when changing the log destination.

The Helm cache volume preserves this schedule across container restarts, but pod replacement loses it. New container IDs receive an initial snapshot. Expired entries are removed when snapshots are recorded. No SBOMs or vulnerability results are cached for reuse between daily scans.

### SigNoz views

Filter logs by `service.name = recense`, then by event type:

- Package inventory: `software.inventory`, filtered by package name, host, or container and the latest successful inventory `scan_id`.
- CVE details: `vulnerability.finding`, filtered by `vulnerability_severity`, `package_name`, `vulnerability_id`, and `fix_available`.
- Daily totals: `vulnerability.scan_completed` with `status = success`; chart `critical_count`, `high_count`, `medium_count`, `low_count`, and `fix_available_count` per target.

Keep all detected packages in scope. Use dashboard filters to prioritize High/Critical findings and available fixes. Counts represent package/advisory matches, not unique CVEs. Choose the latest successful scan per target before aggregating; summing several days repeats findings. Show failed scans and scan age separately.
4 changes: 2 additions & 2 deletions charts/recense/Chart.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2,5 +2,5 @@ apiVersion: v2
name: recense
description: Host and container software inventory and vulnerability logs
type: application
version: 0.2.0
appVersion: "0.2.0"
version: 0.2.1
appVersion: "0.2.1"
1 change: 1 addition & 0 deletions charts/recense/templates/daemonset.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ spec:
- --runtime={{ .Values.runtime }}
- --socket=/runtime/runtime.sock
- --interval={{ .Values.interval }}
- --inventory-interval={{ .Values.inventoryInterval }}
- --jitter={{ .Values.jitter }}
- --scan-timeout={{ .Values.scanTimeout }}
env:
Expand Down
3 changes: 2 additions & 1 deletion charts/recense/values.yaml
Original file line number Diff line number Diff line change
@@ -1,11 +1,12 @@
image:
repository: ghcr.io/opf/recense
tag: "0.2.0"
tag: "0.2.1"
pullPolicy: IfNotPresent
imagePullSecrets: []
clusterName: ""
environment: ""
interval: 24h
inventoryInterval: 168h
jitter: 15m
scanTimeout: 45m
runtime: containerd
Expand Down
2 changes: 2 additions & 0 deletions cmd/recense/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,8 @@ func main() {
flag.StringVar(&c.Grype, "grype", "grype", "Grype executable")
flag.StringVar(&c.Output, "output", "", "append JSON Lines to this file instead of stdout")
flag.DurationVar(&c.Interval, "interval", 24*time.Hour, "time between completed collection cycles")
flag.DurationVar(&c.InventoryInterval, "inventory-interval", 7*24*time.Hour, "minimum time between full package snapshots per target")
flag.StringVar(&c.StateFile, "state-file", "/var/lib/recense/inventory-snapshots.json", "disposable inventory schedule file; loss triggers fresh snapshots")
flag.DurationVar(&c.Timeout, "scan-timeout", 45*time.Minute, "timeout per scanner invocation")
flag.DurationVar(&c.Jitter, "jitter", 15*time.Minute, "maximum random delay before startup and each cycle")
flag.BoolVar(&c.Once, "once", false, "collect once and exit; nonzero on collection failure")
Expand Down
13 changes: 11 additions & 2 deletions internal/agent/agent.go
Original file line number Diff line number Diff line change
Expand Up @@ -17,11 +17,14 @@ type Config struct {
Syft, Grype, Output, Version string
Interval, Timeout, Jitter time.Duration
Once bool
InventoryInterval time.Duration
StateFile string
snapshots *snapshotSchedule
}

func Run(ctx context.Context, c Config) error {
if c.Interval <= 0 || c.Timeout <= 0 || c.Jitter < 0 {
return errors.New("interval and scan-timeout must be positive; jitter must be nonnegative")
if c.Interval <= 0 || c.Timeout <= 0 || c.InventoryInterval <= 0 || c.Jitter < 0 {
return errors.New("interval, inventory-interval and scan-timeout must be positive; jitter must be nonnegative")
}
switch c.Runtime {
case "auto", "none", "docker", "containerd":
Expand All @@ -42,6 +45,12 @@ func Run(ctx context.Context, c Config) error {
"host_id": c.HostID, "host_name": host["host_name"], "agent_version": c.Version,
"environment": c.Environment, "k8s_cluster_name": c.Cluster,
}}
c.snapshots, err = loadSnapshotSchedule(c.StateFile, c.InventoryInterval)
if err != nil {
if emitErr := e.emit("agent.error", Event{"error": "inventory schedule could not be loaded; full snapshots will be sent", "stage": "inventory_schedule"}); emitErr != nil {
return emitErr
}
}
// Heartbeats continue during long scans, but do not imply that scans succeeded.
heartbeatCtx, cancel := context.WithCancel(ctx)
defer cancel()
Expand Down
34 changes: 26 additions & 8 deletions internal/agent/scan.go
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,7 @@ func scan(ctx context.Context, c Config, e *emitter, t target) (resultErr error)
}
env := []string{"GOMEMLIMIT=512MiB", "GOMAXPROCS=1"}
args := []string{"scan", t.Source, "-o", "syft-json", "-c", config, "--quiet", "--parallelism", "1", "--override-default-catalogers", "image", "--base-path", strings.TrimPrefix(t.Source, "dir:")}
excluded := append([]string{"proc", "sys", "dev", "run", "tmp", "var/log", "var/cache", "var/lib/docker", "var/lib/containerd", "var/lib/kubelet", "var/lib/recense"}, t.Excludes...)
excluded := append([]string{"proc", "sys", "dev", "run", "tmp", "var/log", "var/cache", "var/lib/docker", "var/lib/containerd", "var/lib/kubelet", "var/lib/recense", "mnt/k8s-disks"}, t.Excludes...)
for _, path := range excluded {
path = strings.TrimPrefix(filepath.ToSlash(path), "/")
path = strings.NewReplacer("\\", "\\\\", "*", "\\*", "?", "\\?", "[", "\\[", "]", "\\]", "{", "\\{", "}", "\\}").Replace(path)
Expand All @@ -116,15 +116,27 @@ func scan(ctx context.Context, c Config, e *emitter, t target) (resultErr error)
}
}

for _, p := range packages.Artifacts {
fields := packageFields(p)
if err := emit("software.inventory", fields); err != nil {
snapshotSent := false
key := snapshotKey(c.HostID, t)
if c.snapshots == nil || c.snapshots.due(key, time.Now()) {
for _, p := range packages.Artifacts {
if err := emit("software.inventory", packageFields(p)); err != nil {
return err
}
}
if err := emit("software.inventory_completed", Event{"status": "success", "package_count": len(packages.Artifacts)}); err != nil {
return err
}
snapshotSent = true
if c.snapshots != nil {
if err := c.snapshots.record(key, time.Now()); err != nil {
if emitErr := emit("agent.error", Event{"stage": "inventory_schedule", "error": "inventory schedule could not be saved; a restart may repeat snapshots"}); emitErr != nil {
return emitErr
}
}
}
}
if err := emit("software.inventory_completed", Event{"status": "success", "package_count": len(packages.Artifacts)}); err != nil {
return err
}

sbomPath := filepath.Join(work, "sbom.json")
if err := os.WriteFile(sbomPath, inventory, 0600); err != nil {
return err
Expand All @@ -146,20 +158,26 @@ func scan(ctx context.Context, c Config, e *emitter, t target) (resultErr error)
return fmt.Errorf("Grype report is missing matches")
}
counts := map[string]int{"critical": 0, "high": 0, "medium": 0, "low": 0, "negligible": 0, "unknown": 0}
fixAvailableCount := 0
for _, m := range report.Matches {
fields := packageFields(m.Artifact)
fields["vulnerability_id"] = m.Vulnerability.ID
fields["vulnerability_namespace"] = m.Vulnerability.Namespace
fields["vulnerability_severity"] = m.Vulnerability.Severity
fields["fix_state"] = m.Vulnerability.Fix.State
fields["fix_versions"] = m.Vulnerability.Fix.Versions
fields["fix_available"] = len(m.Vulnerability.Fix.Versions) > 0
if len(m.Vulnerability.Fix.Versions) > 0 {
fixAvailableCount++
}
fields["scanner_name"] = "grype"
if err := emit("vulnerability.finding", fields); err != nil {
return err
}
counts[strings.ToLower(m.Vulnerability.Severity)]++
}
fields := Event{"status": "success", "finding_count": len(report.Matches), "package_count": packageCount, "scanner_name": "grype", "scanner_version": report.Descriptor.Version, "duration_seconds": time.Since(started).Seconds()}
fields := Event{"status": "success", "finding_count": len(report.Matches), "package_count": packageCount, "inventory_snapshot_sent": snapshotSent, "scanner_name": "grype", "scanner_version": report.Descriptor.Version, "duration_seconds": time.Since(started).Seconds()}
fields["fix_available_count"] = fixAvailableCount
// Preserve database provenance only. Never emit descriptor.configuration:
// it can contain registry credentials and local environment information.
var db struct {
Expand Down
95 changes: 95 additions & 0 deletions internal/agent/snapshots.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
package agent

import (
"crypto/sha256"
"encoding/hex"
"encoding/json"
"fmt"
"io"
"os"
"path/filepath"
"time"
)

// This is a disposable emission schedule, not an inventory or delivery receipt.
// Missing or invalid state causes extra full snapshots, never skipped CVE scans.
type snapshotSchedule struct {
path string
interval time.Duration
sent map[string]time.Time
}

func loadSnapshotSchedule(path string, interval time.Duration) (*snapshotSchedule, error) {
s := &snapshotSchedule{path: path, interval: interval, sent: map[string]time.Time{}}
f, err := os.Open(path)
if os.IsNotExist(err) {
return s, nil
}
if err != nil {
return s, err
}
defer f.Close()
data, err := io.ReadAll(io.LimitReader(f, (4<<20)+1))
if err != nil {
return s, err
}
if len(data) > 4<<20 {
return s, fmt.Errorf("inventory schedule exceeds 4 MiB")
}
var sent map[string]time.Time
if err := json.Unmarshal(data, &sent); err != nil {
return s, err
}
if sent != nil {
s.sent = sent
}
return s, nil
}

func snapshotKey(host string, t target) string {
sum := sha256.Sum256([]byte(host + "\x00" + t.Kind + "\x00" + t.ID))
return hex.EncodeToString(sum[:])
}

func (s *snapshotSchedule) due(key string, now time.Time) bool {
last, ok := s.sent[key]
return !ok || last.After(now) || now.Sub(last) >= s.interval
}

func (s *snapshotSchedule) record(key string, now time.Time) error {
// Drop expired and future entries to bound state under container churn and
// recover conservatively from clock changes.
for k, last := range s.sent {
if last.After(now) || now.Sub(last) >= s.interval {
delete(s.sent, k)
}
}
s.sent[key] = now
data, err := json.Marshal(s.sent)
if err != nil {
return err
}
if len(data) > 4<<20 {
return fmt.Errorf("inventory schedule exceeds 4 MiB")
}
if err := os.MkdirAll(filepath.Dir(s.path), 0700); err != nil {
return err
}
f, err := os.CreateTemp(filepath.Dir(s.path), ".inventory-snapshots-*")
if err != nil {
return err
}
defer os.Remove(f.Name())
if _, err := f.Write(data); err != nil {
f.Close()
return err
}
if err := f.Sync(); err != nil {
f.Close()
return err
}
if err := f.Close(); err != nil {
return err
}
return os.Rename(f.Name(), s.path)
}
Loading
Loading