Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -123,6 +123,12 @@ so the scan takes about a second. Flags specific to this mode:
instead of a list of flags. See
[USAGE.md](USAGE.md#a-rules-file) and this repo's own
[`.codeshot.yaml`](.codeshot.yaml).
- `--sarif <file>` — the `--forbid` and `--rules` violations as SARIF 2.1.0,
for GitHub code scanning: each is an alert at the first line in the
depending file that calls into the other. Needs a rule to check, and
refuses a scan `--max-symbols` cut (a partial log would close live
alerts); written on every run, also when nothing breaks a rule. See
[USAGE.md](USAGE.md#code-scanning-alerts-with---sarif).
- `--source snapshot:<file>` — draw from a
[graph-assay](https://github.com/inth3shadows/graph-assay) snapshot
(`graph-assay/snapshot@1` JSON) instead of the codegraph index under `--path`.
Expand Down
2 changes: 2 additions & 0 deletions TECHNICAL.md

Large diffs are not rendered by default.

44 changes: 44 additions & 0 deletions USAGE.md
Original file line number Diff line number Diff line change
Expand Up @@ -153,6 +153,50 @@ overlap or a bad pattern fails at once, naming the entry. (Layers whose paths
are patterns are checked for overlap file by file instead.) This repo checks its own
[`.codeshot.yaml`](.codeshot.yaml) in CI.

#### Code-scanning alerts with `--sarif`

`--sarif <file>` also writes the violations as [SARIF
2.1.0](https://docs.oasis-open.org/sarif/sarif/v2.1.0/sarif-v2.1.0.html), the
format GitHub code scanning reads, so each one shows up as an alert on the line
of the pull request that added it:

```yaml
jobs:
architecture:
runs-on: ubuntu-latest
permissions:
contents: read
security-events: write # upload-sarif needs it
steps:
- uses: actions/checkout@v4
# ... install codeshot, codegraph and graphviz, build the index ...
- run: codeshot --architecture --path . --rules .codeshot.yaml --sarif codeshot.sarif --format svg --out arch.svg
- uses: github/codeql-action/upload-sarif@v3
if: always() # upload even when the rules failed the step
with:
sarif_file: codeshot.sarif
category: codeshot
```

Each result sits in the depending file, at the first line where it calls or
refers to the other file (the smallest line codegraph, the snapshot or the
SCIP index records; the calling function's first line when only that is
known), with the depended-on file as a related location. When no source knows
a line, the result points at line 1 of the file. codeshot sets the alert's
fingerprint (`primaryLocationLineHash`, which GitHub uses to track an alert)
from the rule and the two files, not the line, the rules file's name or a
rule's `reason`, so moving the call or rewording a reason doesn't open a new
alert. (upload-sarif logs a warning per result that the fingerprint differs
from the one it would compute from the line's text; it keeps codeshot's.) The file is written on every run, with an empty list when nothing
breaks a rule (which is what closes fixed alerts), and also with `--check`.
`--sarif` needs a rule to check (a `--forbid` flag, or a rules file with a
forbid rule or at least two layers): a run that checks none would upload an
empty list and close every open alert. For the same reason, a `--sarif` run
whose scan `--max-symbols` cut writes no file and exits 1, naming the
`--max-symbols` to rerun with: a partial list would close the alerts of
violations past the cut. Paths are the ones in the diagram, so run from the repository
root (`--path .`) for them to match what GitHub checked out.

### Drawing from another tool's graph

`--source snapshot:<file>` draws the architecture from a graph-assay snapshot
Expand Down
1,712 changes: 925 additions & 787 deletions docs/architecture.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
83 changes: 83 additions & 0 deletions internal/codegraph/index.go
Original file line number Diff line number Diff line change
Expand Up @@ -163,3 +163,86 @@ func nullable(s sql.NullString) string {
// ErrNoIdentity means the enumerated symbols lack qualifiedName (codegraph
// before 1.6.0), so they can't be matched to index rows.
var ErrNoIdentity = errors.New("symbols carry no qualifiedName")

// FilePair is a (from file, to file) dependency.
type FilePair struct{ From, To string }

// SiteLine is a dependency's first line in its From file. Exact: a call or
// reference line; otherwise the calling node's start line, a fallback.
type SiteLine struct {
Line int
Exact bool
}

// ReadSiteLines returns, for every pair of files with a callee edge between
// them (imports and file-kind targets aside, as --architecture draws), the
// smallest line in From of such an edge. A real edge line wins; only a pair
// whose edges carry no line falls back to the smallest calling node's start
// line. 0 counts as no line. An index whose edges table has no line column
// still gives the fallback. Read-only, with the same immutable retry as
// ReadIndexRows.
func ReadSiteLines(dbPath string) (map[FilePair]SiteLine, error) {
read := func(uri string) (map[FilePair]SiteLine, error) {
lines, err := readSiteLines(uri, "e.line")
if IsSchemaError(err) && strings.Contains(err.Error(), "line") {
lines, err = readSiteLines(uri, "NULL") // V6: no edges.line
}
return lines, err
}
lines, err := read(indexURI(dbPath, false))
if err != nil && !IsSchemaError(err) && walIsEmpty(dbPath) {
if retry, rerr := read(indexURI(dbPath, true)); rerr == nil {
return retry, nil
}
}
return lines, err
}

func readSiteLines(uri, edgeLine string) (map[FilePair]SiteLine, error) {
db, err := sql.Open("sqlite", uri)
if err != nil {
return nil, err
}
defer func() { _ = db.Close() }()
var kinds []string
for _, k := range CalleeEdgeKinds {
if k != "imports" {
kinds = append(kinds, k)
}
}
placeholders := strings.TrimSuffix(strings.Repeat("?, ", len(kinds)), ", ")
args := make([]any, len(kinds))
for i, k := range kinds {
args[i] = k
}
rows, err := db.Query(`SELECT s.file_path, t.file_path,
MIN(CASE WHEN `+edgeLine+` > 0 THEN `+edgeLine+` END), MIN(CASE WHEN s.start_line > 0 THEN s.start_line END)
FROM edges e
JOIN nodes s ON s.id = e.source
JOIN nodes t ON t.id = e.target
WHERE e.kind IN (`+placeholders+`) AND t.kind != 'file' AND s.file_path != t.file_path
GROUP BY s.file_path, t.file_path`, args...)
if err != nil {
return nil, err
}
defer func() { _ = rows.Close() }()
out := map[FilePair]SiteLine{}
for rows.Next() {
var from, to sql.NullString
var line, start sql.NullInt64
if err := rows.Scan(&from, &to, &line, &start); err != nil {
return nil, err
}
if !from.Valid || !to.Valid {
continue
}
k := FilePair{From: from.String, To: to.String}
switch {
case line.Valid:
out[k] = SiteLine{Line: int(line.Int64), Exact: true}
case start.Valid:
out[k] = SiteLine{Line: int(start.Int64)}
}
}
return out, rows.Err()
}
102 changes: 102 additions & 0 deletions internal/codegraph/index_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -99,3 +99,105 @@ func TestReadIndexRowsClassifiesSchemaErrors(t *testing.T) {
t.Errorf("corrupt file: err = %v, IsSchemaError = %v; want a non-schema error", err, IsSchemaError(err))
}
}

// ReadSiteLines: the smallest edge line per file pair, the caller's start
// line when the edge has none; imports, file targets and same-file edges
// are left out.
func TestReadSiteLinesTakesTheFirstCallSitePerFilePair(t *testing.T) {
dbPath := filepath.Join(t.TempDir(), ".codegraph", "codegraph.db")
if err := os.MkdirAll(filepath.Dir(dbPath), 0o755); err != nil {
t.Fatal(err)
}
db, err := sql.Open("sqlite", dbPath)
if err != nil {
t.Fatal(err)
}
for _, q := range []string{
`CREATE TABLE nodes (id TEXT PRIMARY KEY, kind TEXT, name TEXT, qualified_name TEXT NOT NULL, file_path TEXT, start_line INTEGER)`,
`CREATE TABLE edges (source TEXT, target TEXT, kind TEXT, line INTEGER)`,
`INSERT INTO nodes VALUES ('a1','function','A1','A1','a.go',3), ('a2','function','A2','A2','a.go',40),
('b','function','B','B','b.go',1), ('c','function','C','C','c.go',1), ('f','file','d.go','d.go','d.go',1)`,
`INSERT INTO edges VALUES ('a2','b','calls',44), ('a1','b','calls',12), ('a1','b','imports',1),
('a2','c','calls',NULL), ('a1','f','calls',2), ('a1','a2','calls',5)`,
} {
if _, err := db.Exec(q); err != nil {
t.Fatalf("%s: %v", q, err)
}
}
_ = db.Close()
got, err := ReadSiteLines(dbPath)
if err != nil {
t.Fatal(err)
}
want := map[FilePair]int{{From: "a.go", To: "b.go"}: 12, {From: "a.go", To: "c.go"}: 40}
if len(got) != len(want) || got[FilePair{"a.go", "b.go"}] != (SiteLine{12, true}) || got[FilePair{"a.go", "c.go"}] != (SiteLine{40, false}) {
t.Errorf("got %v, want %v", got, want)
}
}

// F3: a real call line beats the caller's start line, whatever the order,
// and a 0 is no line.
func TestReadSiteLinesPrefersARealCallLineAndIgnoresZero(t *testing.T) {
dbPath := filepath.Join(t.TempDir(), ".codegraph", "codegraph.db")
if err := os.MkdirAll(filepath.Dir(dbPath), 0o755); err != nil {
t.Fatal(err)
}
db, err := sql.Open("sqlite", dbPath)
if err != nil {
t.Fatal(err)
}
for _, q := range []string{
`CREATE TABLE nodes (id TEXT PRIMARY KEY, kind TEXT, name TEXT, qualified_name TEXT NOT NULL, file_path TEXT, start_line INTEGER)`,
`CREATE TABLE edges (source TEXT, target TEXT, kind TEXT, line INTEGER)`,
`INSERT INTO nodes VALUES ('a1','function','A1','A1','a.go',40), ('a0','file','a.go','a.go','a.go',0),
('b','function','B','B','b.go',1), ('c','function','C','C','c.go',1), ('d','function','D','D','d.go',1)`,
`INSERT INTO edges VALUES ('a1','b','calls',55), ('a1','b','references',NULL),
('a0','c','calls',0), ('a1','c','calls',12),
('a1','d','references',NULL)`,
} {
if _, err := db.Exec(q); err != nil {
t.Fatalf("%s: %v", q, err)
}
}
_ = db.Close()
got, err := ReadSiteLines(dbPath)
if err != nil {
t.Fatal(err)
}
for pair, want := range map[FilePair]SiteLine{{"a.go", "b.go"}: {55, true}, {"a.go", "c.go"}: {12, true}, {"a.go", "d.go"}: {40, false}} {
if got[pair] != want {
t.Errorf("%v: got %v, want %v", pair, got[pair], want)
}
}
}

// V6: an index with node start lines but no edge line column still gives
// the start-line fallback.
func TestReadSiteLinesFallsBackWhenEdgesHaveNoLineColumn(t *testing.T) {
dbPath := filepath.Join(t.TempDir(), ".codegraph", "codegraph.db")
if err := os.MkdirAll(filepath.Dir(dbPath), 0o755); err != nil {
t.Fatal(err)
}
db, err := sql.Open("sqlite", dbPath)
if err != nil {
t.Fatal(err)
}
for _, q := range []string{
`CREATE TABLE nodes (id TEXT PRIMARY KEY, kind TEXT, name TEXT, qualified_name TEXT NOT NULL, file_path TEXT, start_line INTEGER)`,
`CREATE TABLE edges (source TEXT, target TEXT, kind TEXT)`,
`INSERT INTO nodes VALUES ('a','function','A','A','a.go',7), ('b','function','B','B','b.go',1)`,
`INSERT INTO edges VALUES ('a','b','calls')`,
} {
if _, err := db.Exec(q); err != nil {
t.Fatalf("%s: %v", q, err)
}
}
_ = db.Close()
got, err := ReadSiteLines(dbPath)
if err != nil {
t.Fatal(err)
}
if got[FilePair{"a.go", "b.go"}] != (SiteLine{7, false}) {
t.Errorf("got %v, want a.go -> b.go at start line 7", got)
}
}
26 changes: 21 additions & 5 deletions internal/codeshot/arch.go
Original file line number Diff line number Diff line change
Expand Up @@ -218,6 +218,8 @@ type Violation struct {
Weight int
SeenBy string // multi-source: "codegraph 3, scip-go 5"
Breaks string // rules file: what the edge breaks; empty for a --forbid flag
Kind string // "layers" for a layer violation; "" for a forbid rule or flag
Line int // --sarif: first call-site line in From; 0 when unknown or not asked
}

// ParseForbidRule parses FROM->TO; ok=false when malformed.
Expand Down Expand Up @@ -504,6 +506,7 @@ type ArchParams struct {
Forbid []ForbidRule
Rules *Rules // --rules; nil when not given
Tooltips bool // svg: per-source edge weights (multi-source only)
SiteLines bool // --sarif: look up each violation's first call-site line
}

// ArchSource is where --architecture's graph comes from: every symbol in
Expand Down Expand Up @@ -538,11 +541,15 @@ func (s CodegraphSource) EmptyGraphWarning() string { return EmptyArchitectureWa

// RunArchitectureMode builds the file-level graph and checks --forbid rules.
func RunArchitectureMode(src ArchSource, p ArchParams, stderr io.Writer) (string, []Violation, error) {
rawFileEdges, err := sourceFileEdges(src, p.MaxSymbols, "", stderr)
rawFileEdges, err := sourceFileEdges(src, p.MaxSymbols, "", p.SiteLines, stderr)
if err != nil {
return "", nil, err
}
return finishArchitecture(rawFileEdges, p, nil, func() { warn(stderr, src.EmptyGraphWarning()) }, stderr)
dot, vs, err := finishArchitecture(rawFileEdges, p, nil, func() { warn(stderr, src.EmptyGraphWarning()) }, stderr)
if err == nil && p.SiteLines {
fillSiteLines(vs, []LabeledSource{{Src: src}}, stderr)
}
return dot, vs, err
}

// LabeledSource is one entry of a multi-source --source list: the source and
Expand All @@ -561,7 +568,7 @@ func RunMultiArchitectureMode(srcs []LabeledSource, p ArchParams, stderr io.Writ
perSource := make([][]FileEdge, len(srcs))
for i, s := range srcs {
labels[i] = s.Label
edges, err := sourceFileEdges(s.Src, p.MaxSymbols, s.Label, stderr)
edges, err := sourceFileEdges(s.Src, p.MaxSymbols, s.Label, p.SiteLines, stderr)
if err != nil {
return "", nil, labelError(s.Label, err)
}
Expand All @@ -580,17 +587,26 @@ func RunMultiArchitectureMode(srcs []LabeledSource, p ArchParams, stderr io.Writ
warn(stderr, labelWarning(s.Label, s.Src.EmptyGraphWarning()))
}
}
return finishArchitecture(union, p, labels, emptyGraph, stderr)
dot, vs, err := finishArchitecture(union, p, labels, emptyGraph, stderr)
if err == nil && p.SiteLines {
fillSiteLines(vs, srcs, stderr)
}
return dot, vs, err
}

// sourceFileEdges enumerates one source's symbols, applies --max-symbols and
// aggregates its callee edges into file edges. A non-empty label prefixes the
// budget warning (multi-source runs).
func sourceFileEdges(src ArchSource, maxSymbols int, label string, stderr io.Writer) ([]FileEdge, error) {
func sourceFileEdges(src ArchSource, maxSymbols int, label string, refuseCut bool, stderr io.Writer) ([]FileEdge, error) {
symbols, err := src.Symbols()
if err != nil {
return nil, err
}
if len(symbols) > maxSymbols && refuseCut {
// V2: a partial log would close the alerts of violations past the
// cut, which are still in the code.
return nil, &Exit{Code: 1, Stderr: labelWarning(label, fmt.Sprintf("codeshot: --sarif: the scan stopped at --max-symbols %d of %d, and a partial log would close the alerts of violations past the cut; rerun with --max-symbols %d or more. No SARIF file was written.", maxSymbols, len(symbols), len(symbols)))}
}
if len(symbols) > maxSymbols {
symbols = symbols[:maxSymbols]
msg := SymbolBudgetWarning(maxSymbols)
Expand Down
33 changes: 30 additions & 3 deletions internal/codeshot/cli.go
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ const (
)

const usage = `Usage: codeshot <symbol> [--source codegraph|snapshot:<file>|scip[:<file>]] [--path <repoPath>] [--out <file.png>] [--limit <n>] [--max-render <n>] [--format <fmt>] [--depth <n>] [--max-depth-nodes <n>] [--embed <file.md> [--check]]
or: codeshot --architecture [--source codegraph|snapshot:<file>|scip[:<file>][,...]] [--path <repoPath>] [--out <file.png>] [--max-render <n>] [--max-symbols <n>] [--group-depth <n>] [--forbid <from->to>]... [--rules <file>] [--format <fmt>] [--embed <file.md> [--check]]
or: codeshot --architecture [--source codegraph|snapshot:<file>|scip[:<file>][,...]] [--path <repoPath>] [--out <file.png>] [--max-render <n>] [--max-symbols <n>] [--group-depth <n>] [--forbid <from->to>]... [--rules <file>] [--sarif <file>] [--format <fmt>] [--embed <file.md> [--check]]
or: codeshot --diff [--diff-ref <range>] [--source codegraph|snapshot:<file>|scip[:<file>]] [--path <repoPath>] [--out <file.png>] [--limit <n>] [--max-render <n>] [--max-symbols <n>] [--format <fmt>] [--embed <file.md> [--check]]`

type optSpec struct {
Expand All @@ -51,7 +51,7 @@ var options = map[string]optSpec{
"version": {boolean: true}, "path": {}, "out": {}, "limit": {}, "max-render": {},
"format": {}, "depth": {}, "max-depth-nodes": {}, "architecture": {boolean: true},
"max-symbols": {}, "group-depth": {}, "diff": {boolean: true}, "diff-ref": {},
"embed": {}, "check": {boolean: true}, "forbid": {multiple: true}, "source": {}, "rules": {},
"embed": {}, "check": {boolean: true}, "forbid": {multiple: true}, "source": {}, "rules": {}, "sarif": {},
}

// parsed holds flag values; a string flag absent from str was not given.
Expand Down Expand Up @@ -394,6 +394,26 @@ func run(args []string, stdout, stderr io.Writer) (int, error) {
return 0, &Exit{Code: 1, Stderr: "codeshot: " + err.Error()}
}
}
// --sarif: the rule violations as SARIF 2.1.0, for code-scanning alerts.
sarifFile, hasSARIF := p.str["sarif"]
if hasSARIF {
if sarifFile == "" {
return 0, &Exit{Code: 1, Stderr: "codeshot: --sarif must not be empty"}
}
if !arch {
return 0, &Exit{Code: 1, Stderr: "codeshot: --sarif only applies with --architecture (it reports --forbid and --rules violations)"}
}
// An empty log closes every open alert, so a run that checks no
// rule must not write one (F6).
if len(rules) == 0 && rulesFile == nil {
return 0, &Exit{Code: 1, Stderr: "codeshot: --sarif needs --forbid or --rules: with no rule checked it would upload an empty log and close every open alert"}
}
// V3: a rules file with no forbid rule and fewer than two layers
// can't flag anything either.
if len(rules) == 0 && len(rulesFile.Forbid) == 0 && len(rulesFile.Layers) < 2 {
return 0, &Exit{Code: 1, Stderr: fmt.Sprintf("codeshot: --sarif needs a rule to check: '%s' has no forbid rule and fewer than two layers, so the log would be empty and close every open alert", rulesFile.File)}
}
}
embedFile, hasEmbed := p.str["embed"]
if hasEmbed && embedFile == "" {
return 0, &Exit{Code: 1, Stderr: "codeshot: --embed must not be empty"}
Expand Down Expand Up @@ -574,7 +594,7 @@ func run(args []string, stdout, stderr io.Writer) (int, error) {
case scipIdx != nil:
src = ScipSource{Index: scipIdx}
}
ap := ArchParams{MaxSymbols: maxSymbols, MaxRender: maxRender, GroupDepth: groupDepth, Forbid: rules, Rules: rulesFile, Tooltips: tooltips}
ap := ArchParams{MaxSymbols: maxSymbols, MaxRender: maxRender, GroupDepth: groupDepth, Forbid: rules, Rules: rulesFile, Tooltips: tooltips, SiteLines: hasSARIF}
var dot string
var violations []Violation
if multi != nil {
Expand Down Expand Up @@ -611,6 +631,13 @@ func run(args []string, stdout, stderr io.Writer) (int, error) {
if err != nil {
return 0, err
}
// Written before the diagram, so a failing --check still leaves CI
// its findings, and with zero violations, so fixed alerts close.
if hasSARIF {
if err := WriteSARIF(sarifFile, violations, versionString()); err != nil {
return 0, err
}
}
out.MarkerID = archMarkerID
var notes []string
if groupDepth != 0 {
Expand Down
Loading
Loading