From 3bb0169eb8b407cf0d88ccb7abe528bd30a3a0cf Mon Sep 17 00:00:00 2001 From: Josh Vaughen Date: Thu, 22 Oct 2026 13:55:04 -0700 Subject: [PATCH 1/6] refactor(rsync): name the long awk and Python programs Each awk or Python program of more than two lines moves out of its command into a named var in a new "programs" part of vars:, with the comment that explains it: LIST_FILES, LIST_DIRS, CUT_BATCHES, TLPDB_SUMS, TLPDB_NAMES, LABEL_PY, DELETE_JSON, SAME_PAGES, PAGE_ENTRIES, PAGE_HTML. The style of the directory pages moves into PAGE_CSS, which YAML joins with no space. No behavior change: - The rendered commands have the same tokens as v2.9.1, except PAGE_CSS, which has the same 1,779 bytes as the old string concatenation. Thus, PAGE_FORMAT stays 3. - offline passes. run-pages matches want/ byte for byte. - The parity harness of lib-rewrite: 115 of 115 scenarios identical against v2.9.1 (tree, exit code and output). --- CLAUDE.md | 25 +- README.md | 5 + engines/rsync.yml | 580 +++++++++++++++++++++----------------- examples/rsync/render.txt | 185 +++++------- 4 files changed, 411 insertions(+), 384 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 2f9034d..048402d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -19,8 +19,10 @@ Read that section before you write. - Do not read a call var in a global var. Task renders a global var that reads a different var one time. It uses the values of the root and of the command line, and not a call - var. Thus, `GPGCHECK` gets the fingerprint as an awk `-v` in each command that uses it. In - that command, the call var `TL_KEY` is available. + var. Thus, each program in the `programs` part of `vars:` in `engines/rsync.yml` gets its + inputs from awk `-v`, `ENVIRON` or an argument, in the command that uses it. For example, + each command that uses `GPGCHECK` gives it the fingerprint as an awk `-v`. In that + command, the call var `TL_KEY` is available. - Some vars have a different name and read a var that a mirror can set, for example `BATCHES_MAX` from `MAX_BATCHES`. Such a var also gets the value on the command line (verified). Thus, write each default one time, inline. @@ -75,9 +77,9 @@ Read that section before you write. no key without a slash. Thus, if the only changed directory of a run is the root, that awk sends no line to `xargs`. - A `>-` folded block keeps a newline if a line that continues the line before it starts - with more spaces than the lines around it. The awk programs of `pages` use this. Put a - backslash at the end of a shell line that must continue. `task check` shows how the block - renders. + with more spaces than the lines around it. The two-line awk programs in the commands of + `split` and `pages` use this. Put a backslash at the end of a shell line that must + continue. `task check` shows how the block renders. - On a macOS disk, two directory names that are different only in uppercase and lowercase letters are one directory. Thus, the pages that a laptop writes from a listing of upstream are not complete if upstream has two such directories. For example, CTAN has @@ -113,12 +115,13 @@ Read that section before you write. encoded first is encoded two times (`a%20b` goes out as `a%2520b`), and the copy then reads a different key (verified with `aws --debug`, aws-cli 2.36.24). - Increase `PAGE_FORMAT` in `engines/rsync.yml` at each change to the HTML or the style of - a directory page. If you do not, each mirror keeps the pages of the directories that do - not change, and they show the old format. The stamp in `.state/indexed.txt.xz` includes - `PAGE_FORMAT`. A new value makes each mirror write all of its pages again, one time. -- The page awk of `pages` writes each page directly to its two files. Do not make the page - with a string that gets a row at a time. That method copies the full string for each - row. The `archive/` directory of TeX Live has approximately 34k rows. + a directory page (`PAGE_HTML` and `PAGE_CSS`). If you do not, each mirror keeps the pages + of the directories that do not change, and they show the old format. The stamp in + `.state/indexed.txt.xz` includes `PAGE_FORMAT`. A new value makes each mirror write all + of its pages again, one time. +- `PAGE_HTML`, the page awk of `pages`, writes each page directly to its two files. Do not + make the page with a string that gets a row at a time. That method copies the full string + for each row. The `archive/` directory of TeX Live has approximately 34k rows. - In a check that reads the output of a task with `grep`, find text that only the output has. go-task shows each command before it runs it, and that line also has the text of each `echo` in the command. For example, `offline` finds `not 08` in the output of diff --git a/README.md b/README.md index f870117..70f6a5a 100644 --- a/README.md +++ b/README.md @@ -1387,6 +1387,11 @@ these rules: - A new hook is an empty task, with a `desc` that gives the mirror that fills it. The verb that runs the hook gives it the vars that it uses. - A new var that a mirror can set is an inline `{{.X | default N}}` in the command that reads it. Write the default one time. Add a row to the table in [The vars](#the-vars). +- Put each awk or Python program of more than two lines in the `programs` part of `vars:`. + Write a comment above it that gives its function and its inputs. The command that runs + the program gives it each input with awk `-v`, `ENVIRON` or an argument. A program has + no template of a var that a caller can give, because task renders a global var without + the call vars. - A change to a verb is a change to `examples//render.txt`. The diff of that file in the pull request is the review. diff --git a/engines/rsync.yml b/engines/rsync.yml index 00d883f..3c514b4 100644 --- a/engines/rsync.yml +++ b/engines/rsync.yml @@ -47,11 +47,7 @@ # - FRESH_HOURS: 24. Digits, with no 0 first. When the clock in FRESH_KEY is more # than FRESH_HOURS before the time of the check, smoke stops the run with a failure. # -# The label verb here selects the Content-Type of each object. The AWS CLI does not select -# it, for two causes: -# - The CLI keeps the first value of the result of guess_type in Python. Thus, the CLI -# sends foo.diff.gz as a diff. -# - The results of the CLI change with the image. +# The label verb selects the Content-Type of each object. The AWS CLI does not select it. # # Hooks: prepare, verify, index, smoke-mirror and report-engine are the verbs that a # mirror can replace. To replace one, a mirror puts it in `excludes:` on this include and @@ -87,8 +83,8 @@ vars: # The page state as the directory pages showed it the last time, after a stamp line. INDEXED: .state/indexed.txt.xz # The format of the directory pages. Increase it at each change to the HTML or the style - # of a page. The stamp in INDEXED includes it. Thus, when a mirror gets a release with a - # new format, its next run makes all of its pages again. + # of a page (PAGE_HTML and PAGE_CSS). The stamp in INDEXED includes it. Thus, when a + # mirror gets a release with a new format, its next run makes all of its pages again. PAGE_FORMAT: '3' # Batches for each run. The remaining batches chain the next run. BATCHES_MAX: '{{.MAX_BATCHES | default 4}}' @@ -96,31 +92,15 @@ vars: RETRY_S: '{{.RETRY_BASE | default 15}}' # Hours. After 28 h, the monitors of GNU and CTAN show a mirror as expired. FRESH_H: '{{.FRESH_HOURS | default 24}}' - # The type of a compressed file, then the name endings that select it. These are the - # types of ftp.gnu.org. The ending is more important than the remaining part of the - # name: foo.diff.gz is gzip. A browser that gets text/x-diff shows the compressed bytes - # as text. label uses this table for each upload. The endings are case-sensitive: .Z is - # compress. - COMPRESSED: |- - application/x-gzip .gz .tgz .taz .tz .svgz - application/x-bzip2 .bz2 .tbz2 .tbz - application/x-xz .xz .txz - application/x-compress .Z - application/zstd .zst .tzst - application/x-lzip .lz - application/x-lzma .lzma - # The clock of upstream in a timestamp file, as Unix time or as "YYYY-MM-DD HH:MM" for - # date -d. mirror-updated-timestamp.txt (gnu) and 00_TIME.txt (Savannah) start with Unix - # time. The timestamp file of CTAN holds 2026-10-08-05-02, after a comment and the name - # of the master. - STAMP: |- - awk '!/^#/ { for (i = 1; i <= NF; i++) { - if ($i ~ /^[0-9]+$/ && length($i) >= 9) { print $i; exit } - if ($i ~ /^[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]-[0-9][0-9]-[0-9][0-9]$/) { - split($i, d, "-"); print d[1] "-" d[2] "-" d[3] " " d[4] ":" d[5]; exit } } }' RSYNC: rsync --timeout=300 --contimeout=60 --no-h CURL: curl -fsS --connect-timeout 15 --max-time 60 --retry 6 --retry-connrefused --retry-max-time 600 AWS_FLAGS: --no-progress --cli-connect-timeout 60 --cli-read-timeout 300 + + # ---- programs ---- + # The awk and Python programs of the verbs. A program gets each input from awk -v, + # ENVIRON or an argument. It has no template of a var that a caller can give, because + # task renders a global var one time, without the call vars (lib CLAUDE.md, Hazards). + # The keyring is a usual file from the same mirror as the signature. Thus, only the # pinned fingerprint identifies the correct key. GOODSIG is also necessary, because gpgv # gives VALIDSIG and exit 0 for an expired key or a revoked key. This is an awk program @@ -145,7 +125,7 @@ vars: awk 'tolower($1) == "content-type:" { sub(/\r$/, ""); sub(/^[^:]*:[ \t]*/, ""); print }' # Percent-encoding for awk. enc(s) keeps the unreserved set and "/", and it encodes all # other characters. Upstream paths contain spaces and ( ) [ ] { } & # > ~ $. This text - # goes into ENCODE, and into the awk program of the pages verb. + # goes into ENCODE and PAGE_HTML. URLENC: |- BEGIN { for (i = 0; i < 256; i++) ord[sprintf("%c", i)] = i } function enc(s, out, j, c) { out = ""; for (j = 1; j <= length(s); j++) { c = substr(s, j, 1) @@ -160,6 +140,286 @@ vars: ANCESTORS: |- awk -F'\t' '!("" in s) { s[""] = 1; print "" } { n = split($1, c, "/"); d = ""; for (i = 1; i < n; i++) { d = d c[i] "/"; if (!(d in s)) { s[d] = 1; print d } } }' + # The clock of upstream in a timestamp file, as Unix time or as "YYYY-MM-DD HH:MM" for + # date -d. mirror-updated-timestamp.txt (gnu) and 00_TIME.txt (Savannah) start with Unix + # time. The timestamp file of CTAN holds 2026-10-08-05-02, after a comment and the name + # of the master. + STAMP: |- + awk '!/^#/ { for (i = 1; i <= NF; i++) { + if ($i ~ /^[0-9]+$/ && length($i) >= 9) { print $i; exit } + if ($i ~ /^[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]-[0-9][0-9]-[0-9][0-9]$/) { + split($i, d, "-"); print d[1] "-" d[2] "-" d[3] " " d[4] ":" d[5]; exit } } }' + # normalise: a line "path TAB size TAB mtime" for each regular file of a listing (rsync + # --list-only). The size column can contain commas (rsync 3.x), or it can be empty for an + # empty file (openrsync). Thus, the program finds the size from the date, which always has + # the same width, and not from a field number. + # + # The program keeps each regular file in the listing of upstream. tlnet keeps each + # container with a revision-stamped name, and a symlink with the stable name is adjacent + # to it. -L resolves the two, and the mirror sends the two. A mirror that wants only the + # stable names makes the listing smaller with FILTER. tlnet does this. + LIST_FILES: |- + /^-/ && match($0, / [0-9][0-9][0-9][0-9]\/[0-9][0-9]\/[0-9][0-9] [0-9][0-9]:[0-9][0-9]:[0-9][0-9] /) { + path = substr($0, RSTART + RLENGTH); mtime = substr($0, RSTART + 1, RLENGTH - 2) + size = substr($0, 11, RSTART - 11); gsub(/[ ,]/, "", size) + print path "\t" size + 0 "\t" mtime } + # normalise: a line "path/" for each directory of a listing, with a slash at the end. The + # root (".") is not in the output. pages makes a page for each directory, also for a + # directory with no file. + LIST_DIRS: |- + /^d/ && match($0, / [0-9][0-9][0-9][0-9]\/[0-9][0-9]\/[0-9][0-9] [0-9][0-9]:[0-9][0-9]:[0-9][0-9] /) { + path = substr($0, RSTART + RLENGTH); if (path != ".") print path "/" } + # split: divide the delta (changed.txt) into the batch files DIR/batch-NNNN.txt. CAP is the + # number of bytes for each batch. The program adds the sizes in the listing. It uses only + # the listing to divide the delta, not the files. A file that is larger than BATCH_GB is a + # batch with no other file, and the batch that the program fills does not change. + # + # The decision batch is the last batch. It holds tlpkg/ of TL (the tlpdb) and the files at + # the bucket root, which include the FRESH_KEY file. A client reads these files before it + # selects the files to download. Because this batch is the last batch, these files cannot + # identify a key that is not in the bucket at that time. + # + # A path that has the previous path as its first part (x, x.sha512, x.sha512.asc) does not + # start a new batch. Also, diff puts x in the delta when x.sha512 is in the delta. Thus, + # verify always finds the file that its signature identifies. + CUT_BATCHES: |- + index($1, TL "/tlpkg/") == 1 || $1 !~ /\// { dec[++nd] = $0; next } + $2 > CAP { f = name(++n); print > f; close(f); next } + { if (sz + $2 > CAP && index($1, prev) != 1) flush(); buf[++nb] = $0; sz += $2; prev = $1 } + END { flush(); if (nd) { f = name(++n); for (i = 1; i <= nd; i++) print dec[i] > f; close(f) } + printf "%d lines in %d batches\n", NR, n } + function name(i) { return sprintf("%s/batch-%04d.txt", DIR, i) } + function flush( i, f) { if (!nb) return; f = name(++n) + for (i = 1; i <= nb; i++) print buf[i] > f; close(f); nb = sz = 0; split("", buf) } + # verify: a "sum path" line for shasum -c for each TL container of a batch. The inputs are + # the batch, then the tlpdb. The program uses the two names that upstream has: + # - the stable name, which the tlpdb gives + # - the revision-stamped name, which the tlpdb gives through the revision. + # The two names have the same bytes, and thus the same checksum. The END clause is a second + # check: it rejects a container that is not in the tlpdb. + TLPDB_SUMS: |- + NR == FNR { if (index($1, TL "/archive/") == 1) want[$1] = 1; next } + /^name /{split($0, a, " "); n = a[2]} /^revision /{split($0, a, " "); r = a[2]} + /^containerchecksum /{split($0, a, " "); emit(a[2], n)} + /^doccontainerchecksum /{split($0, a, " "); emit(a[2], n ".doc")} + /^srccontainerchecksum /{split($0, a, " "); emit(a[2], n ".source")} + function emit(sum, base, p, q) { p = TL "/archive/" base ".tar.xz"; q = TL "/archive/" base ".r" r ".tar.xz" + if (p in want) { print sum " " p; delete want[p] } + if (q in want) { print sum " " q; delete want[q] } } + END { for (p in want) { print "no checksum in the tlpdb for " p > "/dev/stderr"; bad = 1 } exit bad } + # verify: the path of each container that a tlpdb gives, at its stable name. + TLPDB_NAMES: |- + /^name /{n=$2} + /^containerchecksum /{print TL "/archive/" n ".tar.xz"} + /^doccontainerchecksum /{print TL "/archive/" n ".doc.tar.xz"} + /^srccontainerchecksum /{print TL "/archive/" n ".source.tar.xz"} + # The type of a compressed file, then the name endings that select it. These are the + # types of ftp.gnu.org. The ending is more important than the remaining part of the + # name: foo.diff.gz is gzip. A browser that gets text/x-diff shows the compressed bytes + # as text. label uses this table for each upload. The endings are case-sensitive: .Z is + # compress. + COMPRESSED: |- + application/x-gzip .gz .tgz .taz .tz .svgz + application/x-bzip2 .bz2 .tbz2 .tbz + application/x-xz .xz .txz + application/x-compress .Z + application/zstd .zst .tzst + application/x-lzip .lz + application/x-lzma .lzma + # label: a line "path TAB type" for each file under a directory. The arguments are the + # directory and COMPRESSED. The rules, in sequence: + # 1. A COMPRESSED ending gives the type directly. + # 2. An ending that Python reads as compressed and that COMPRESSED does not have (.br) + # gets the type of a signature, or application/octet-stream. It does not get text, + # because brotli has no signature to find. + # 3. If not, the mime.types of the image gives the type. But a type that a browser shows + # (text, XML, JSON, JavaScript, PDF) must agree with the bytes. If the bytes are + # compressed, contain a NUL, or are a PDF without %PDF-, they give the type. + # 4. A name that has no type in mime.types gets a type from its bytes. This type is a + # known signature, text/plain without a NUL, or application/octet-stream with a NUL. + # 5. A text/* type gets "; charset=utf-8" when the full file is UTF-8 (ASCII is UTF-8). A + # browser then shows its characters correctly. The bytes of other text do not tell + # their encoding. Thus, other text gets no charset. + # A control byte is not a NUL: some .sty files of CTAN contain ^Z, and they stay text. + # Without "./", guess_type reads a name with a colon as a URL scheme. + LABEL_PY: |- + import codecs, mimetypes, os, sys + + root, table = sys.argv[1], sys.argv[2] + endings = {s: f[0] for f in map(str.split, table.splitlines()) if f for s in f[1:]} + signatures = ((b"\x1f\x8b", "application/x-gzip"), (b"BZh", "application/x-bzip2"), + (b"\xfd7zXZ\x00", "application/x-xz"), (b"\x1f\x9d", "application/x-compress"), + (b"\x28\xb5\x2f\xfd", "application/zstd"), (b"LZIP", "application/x-lzip"), + (b"PK\x03\x04", "application/zip"), (b"%PDF-", "application/pdf")) + drawn = ("text/", "application/xml", "application/json", "application/javascript", "application/pdf") + + def from_bytes(head): + for sig, t in signatures: + if head.startswith(sig): + return t + text = b"\0" not in head or head.startswith((b"\xfe\xff", b"\xff\xfe")) + return "text/plain" if text else "application/octet-stream" + + def label(name, head): + for ending, t in endings.items(): + if name.endswith(ending): + return t + t, encoding = mimetypes.guess_type("./" + name) + seen = from_bytes(head) + if encoding: + return "application/octet-stream" if seen == "text/plain" else seen + if t is None: + return seen + if t.startswith(drawn) or t.endswith(("+xml", "+json")): + agrees = b"%PDF-" in head if t == "application/pdf" else seen == "text/plain" + return t if agrees else seen + return t + + # One MiB at a time: some text files are large (FILES.byname of CTAN is 27 MB). + def utf8(path): + d = codecs.getincrementaldecoder("utf-8")() + try: + with open(path, "rb") as h: + for part in iter(lambda: h.read(1 << 20), b""): + d.decode(part) + d.decode(b"", final=True) + except UnicodeDecodeError: + return False + return True + + out = sys.stdout.buffer + for d, _, files in os.walk(root): + for f in files: + p = os.path.join(d, f) + with open(p, "rb") as h: + head = h.read(1024) + t = label(f, head) + if t.startswith("text/") and utf8(p): + t += "; charset=utf-8" + out.write(os.fsencode(os.path.relpath(p, root)) + b"\t" + t.encode() + b"\n") + # remove: the JSON body of one DeleteObjects call, for the keys of a file, one key on each + # line. The program escapes each backslash and each double quotation mark of a key. + DELETE_JSON: |- + BEGIN { printf "{\"Objects\":[" } { gsub(/\\/, "\\\\"); gsub(/"/, "\\\"") + printf "%s{\"Key\":\"%s\"}", (NR > 1 ? "," : ""), $0 } END { print "],\"Quiet\":true}" } + # index-same: a directory is the same if each of its keys is the same. The program writes + # the same directories to Y (index-same.txt), the other directories to M (index-make.new), + # and the files of the same directories to its output. The program finds each input with + # FILENAME, because an empty file has no first line. Its other vars: + # - E: the ETag of each key in the bucket + # - H: the MD5 of each staged key (page-md5.txt) + # - I: INDEX + # - S and R: STAGING and SLASH, each with a slash at the end. + SAME_PAGES: |- + FILENAME == E { gsub(/"/, "", $2); e[$1] = $2; next } + FILENAME == H { if ($2 == "s") d = $3 "/"; else if (substr($3, length($3) - length(I) + 1) == I) d = substr($3, 1, length($3) - length(I)); else next + seen[d] = 1; if (e[$3] != $1) diff[d] = 1; next } + !($0 in seen) || ($0 in diff) { print > M; next } + { print > Y; print S $0 I; if ($0 != "") { b = substr($0, 1, length($0) - 1); print R split(b, c, "/") "/" b } } + # pages: the entries "dir TAB d|e|f TAB name TAB size TAB mtime" of each directory of the + # first input (index-make.txt), from the page state (the second input): the files in the + # bucket and the directories of upstream. A line "dir/" of the page state gives an e entry + # for that directory. The e entry opens the page and adds no row. Thus, a directory with no + # file also gets a page. + PAGE_ENTRIES: |- + NR == FNR { want[$0] = 1; next } + { n = split($1, c, "/"); d = "" + for (i = 1; i < n; i++) { if ((d in want) && !((d c[i]) in seen)) { seen[d c[i]] = 1; print d, "d", c[i], "", "" }; d = d c[i] "/" } + if (d in want) print d, (c[n] == "" ? "e" : "f"), c[n], $2, $3 } + # pages: the style of each directory page. Each line ends with a backslash. Thus, YAML + # joins the lines with no space, and the style is one line in each page. + PAGE_CSS: ":root{color-scheme:light dark;--bg:#f6f5f2;--fg:#1c1c28;--mut:#6e6e7e;--line:#e2e0da;\ + --hov:#edebe5;--a:#3b45d9}\ + @media (prefers-color-scheme:dark){:root{--bg:#121218;--fg:#f0f0f5;--mut:#8b8b9c;\ + --line:#262630;--hov:#1a1a22;--a:#8b93ff}}\ + body{margin:0 auto;max-width:60rem;padding:2rem 1rem 3rem;background:var(--bg);\ + color:var(--fg);font:15px/1.45 system-ui,-apple-system,sans-serif}\ + h1{font-size:1.35rem;font-weight:650;margin:0;overflow-wrap:anywhere}\ + p{color:var(--mut);margin:.2rem 0 1.25rem;font-size:.9rem}\ + table{width:100%;border-collapse:collapse;\ + font:13.5px/1.4 ui-monospace,SFMono-Regular,Menlo,Consolas,monospace}\ + th{font:600 .7rem/1 system-ui,sans-serif;letter-spacing:.06em;\ + text-transform:uppercase;color:var(--mut);text-align:left;padding:0 .5rem .6rem;\ + white-space:nowrap}\ + td{padding:.4rem .5rem;border-top:1px solid var(--line);vertical-align:baseline}\ + td:first-child{overflow-wrap:anywhere;width:100%}\ + td+td{white-space:nowrap;color:var(--mut);font-variant-numeric:tabular-nums}\ + th:last-child,td:last-child{text-align:right}tr:hover td{background:var(--hov)}\ + a{color:var(--a);text-decoration:none}a:hover{text-decoration:underline}\ + .s a{color:var(--mut)}\ + .k{font-size:.72rem;margin-left:.6rem;padding:.05rem .4rem;\ + border:1px solid var(--line);border-radius:.3rem;color:var(--mut)}\ + .k:hover{color:var(--a);border-color:var(--a);text-decoration:none}\ + footer{margin-top:2rem;padding-top:1rem;border-top:1px solid var(--line);\ + color:var(--mut);font-size:.85rem;overflow-wrap:anywhere}\ + footer a{color:inherit;text-decoration:underline}\ + @media (max-width:36rem){thead{display:none}\ + tr{display:grid;grid-template-columns:1fr auto;border-top:1px solid var(--line)}\ + td{border:0;padding:.1rem .5rem}td:first-child{grid-column:1/-1;padding-top:.45rem}\ + td+td{padding-bottom:.45rem;font-size:.8rem}}" + # pages: the page of each directory, from its entries, sorted. The program keeps the rows of + # a directory in arrays. At shut, it writes the page to its two keys with the same calls. + # Thus, the two keys have the same bytes. With the base href, one document is correct for + # /dir/ and for /dir. Without it, a relative href in the second resolves against the + # parent. The program percent-encodes each name in href and HTML-escapes it in text. + # + # The footer is PAGE_FOOT (F), with %s for the encoded path of the directory. When + # PAGE_FOOT is empty, the program makes the footer from UPSTREAM (U) and CONTACT (C). + # CONTACT gets its default from HOST (H). + # + # rank gives the position of each name in INDEX_FILES (N). real is the index file of the + # open directory with the lowest rank. Without a real, shut writes the page key, and with + # a real, it writes a line to X (index-real.txt). + # + # G gives the suffixes of the signature files and the checksum files. The match ignores + # uppercase and lowercase. A file whose name is a different file of the directory plus one + # or more of these suffixes goes on the row of that file. + # + # Its other vars: S and R, STAGING and SLASH, each with a slash at the end, I (INDEX) and + # css (PAGE_CSS). + PAGE_HTML: |- + {{.URLENC}} + BEGIN { n = split(N, nm, " "); for (i = n; i >= 1; i--) rank[nm[i]] = i + G = "\\.(sig|sign|asc|gpg|dsig|proof|md5|md5sum|sha|sha1|sha224|sha256|sha256sum|sha512|sum|checksums)$" + if (C == "" && (j = index(H, ".")) > 1) C = substr(H, 1, j - 1) "@" substr(H, j + 1) } + function esc(s) { gsub(/&/, "\\&", s); gsub(//, "\\>", s); gsub(/"/, "\\"", s); return s } + function grp(n, s, o) { s = sprintf("%.0f", n); o = "" + while (length(s) > 3) { o = "," substr(s, length(s) - 2) o; s = substr(s, 1, length(s) - 3) } return s o } + function iec(n, u, i) { if (n < 1024) return sprintf("%.0f B", n) + split("KiB MiB GiB TiB", u, " "); n /= 1024; i = 1; while (n >= 1024 && i < 4) { n /= 1024; i++ } + return sprintf(n < 10 ? "%.1f %s" : "%.0f %s", n, u[i]) } + function ymd(t) { gsub(/\//, "-", t); return substr(t, 1, 16) } + function foot( t) { if (F != "") return "
" sprintf(F, enc(cur)) "
" + t = (U == "" ? "" : "Mirror of " esc(U) enc(cur) ".") + if (C != "") t = t (t == "" ? "" : " ") "Report a problem to " esc(C) "." + return (t == "" ? "" : "
" t "
") } + function open(d) { cur = d; live = 1; real = ""; nd = 0; nfl = 0; tot = 0; split("", has) } + function w(s) { if (real == "") printf("%s", s) > fa; if (cur != "") printf("%s", s) > fb } + function shut( i, r, x, c, e, k, b) { split("", fold); split("", links) + for (i = 1; i <= nfl; i++) { r = fname[i]; x = "" + while (match(tolower(r), G)) { r = substr(r, 1, RSTART - 1); if (r in has) x = r } + if (x != "") { fold[i] = 1 + links[x] = links[x] "" esc(substr(fname[i], length(x) + 1)) "" } } + c = (nd ? grp(nd) (nd == 1 ? " directory" : " directories") : "") + if (nfl) c = c (c == "" ? "" : ", ") grp(nfl) (nfl == 1 ? " file" : " files") ", " iec(tot) + if (c == "") c = "This directory is empty." + fa = S cur I; if (real != "") print cur real "\t" cur I > X + if (cur != "") { b = substr(cur, 1, length(cur) - 1); k = split(b, e, "/"); fb = R k "/" b } + w("\nIndex of /" esc(cur) "\n") + w("\n\n

Index of /" esc(cur) "

" c "

\n") + w("\n") + if (cur != "") w("\n") + for (i = 1; i <= nd; i++) w("\n") + for (i = 1; i <= nfl; i++) if (!(i in fold)) + w("\n") + w("
NameModified (UTC)Size
../
" esc(dname[i]) "/
" esc(fname[i]) "" links[fname[i]] "" ymd(ftime[i]) "" iec(fsize[i]) "
" foot() "\n") + if (real == "") close(fa); if (cur != "") close(fb) } + !live || $1 != cur { if (live) shut(); open($1) } + $2 == "e" { next } + $2 == "d" { dname[++nd] = $3; next } + ($3 in rank) && (real == "" || rank[$3] < rank[real]) { real = $3 } + { fname[++nfl] = $3; fsize[nfl] = $4; ftime[nfl] = $5; has[$3] = 1; tot += $4 } + END { if (live) shut() } tasks: pipeline: @@ -220,30 +480,8 @@ tasks: desc: RUN/listing.txt (rsync --list-only) to RUN/upstream.txt, every regular file as "path TAB size TAB mtime", and RUN/upstream-dirs.txt, every directory but the root as "path/"; both byte-sorted set: [pipefail] cmds: - # The size column can contain commas (rsync 3.x), or it can be empty for an empty - # file (openrsync). Thus, the program finds the size from the date, which always has - # the same width, and not from a field number. - # - # The program keeps each regular file in the listing of upstream. tlnet keeps each - # container with a revision-stamped name, and a symlink with the stable name is - # adjacent to it. -L resolves the two, and the mirror sends the two. A mirror that - # wants only the stable names makes the listing smaller with FILTER. tlnet does - # this. - - >- - awk ' - /^-/ && match($0, / [0-9][0-9][0-9][0-9]\/[0-9][0-9]\/[0-9][0-9] [0-9][0-9]:[0-9][0-9]:[0-9][0-9] /) { - path = substr($0, RSTART + RLENGTH); mtime = substr($0, RSTART + 1, RLENGTH - 2) - size = substr($0, 11, RSTART - 11); gsub(/[ ,]/, "", size) - print path "\t" size + 0 "\t" mtime }' {{.RUN}}/listing.txt \ - | LC_ALL=C sort > {{.RUN}}/upstream.txt - # This command writes the directories of upstream, each with a slash at the end. The - # root (".") is not in the file. pages makes a page for each directory, also for a - # directory with no file. - - >- - awk ' - /^d/ && match($0, / [0-9][0-9][0-9][0-9]\/[0-9][0-9]\/[0-9][0-9] [0-9][0-9]:[0-9][0-9]:[0-9][0-9] /) { - path = substr($0, RSTART + RLENGTH); if (path != ".") print path "/" }' {{.RUN}}/listing.txt \ - | LC_ALL=C sort > {{.RUN}}/upstream-dirs.txt + - awk '{{.LIST_FILES}}' {{.RUN}}/listing.txt | LC_ALL=C sort > {{.RUN}}/upstream.txt + - awk '{{.LIST_DIRS}}' {{.RUN}}/listing.txt | LC_ALL=C sort > {{.RUN}}/upstream-dirs.txt # A truncated listing must not become a list of deletions. LIST_FLOOR is a line # count, approximately 90% of the usual size of the listing. A correct listing # always has more lines. When LIST_FLOOR is not set, there is no guard. @@ -341,33 +579,12 @@ tasks: '$2 / 1024 > kb - 1048576 { print "no room for " $1 " (" $2 " bytes)"; bad = 1 } END { exit bad }' \ {{.RUN}}/changed.txt - rm -f {{.RUN}}/batch-*.txt - # The program adds the sizes in the listing. It uses only the listing to divide the - # delta, not the files. A file that is larger than BATCH_GB is a batch with no other - # file, and the batch that the program fills does not change. - # - # The decision batch is the last batch. It holds tlpkg/ of TL (the tlpdb) and the - # files at the bucket root, which include the FRESH_KEY file. A client reads these - # files before it selects the files to download. Because this batch is the last - # batch, these files cannot identify a key that is not in the bucket at that time. - # - # A path that has the previous path as its first part (x, x.sha512, x.sha512.asc) - # does not start a new batch. Also, diff puts x in the delta when x.sha512 is in the - # delta. Thus, verify always finds the file that its signature identifies. - # # BATCH_GB goes into the command with no quotes, because it is in $(( )), where a # quoted operand is a syntax error. The first command of split stops the run before # this line on each value that is not a whole number. - >- - awk -F'\t' -v OFS='\t' -v CAP=$(({{.BATCH_GB | default 4}} * 1000000000)) -v DIR={{.RUN}} -v TL="{{.TL}}" ' - index($1, TL "/tlpkg/") == 1 || $1 !~ /\// { dec[++nd] = $0; next } - $2 > CAP { f = name(++n); print > f; close(f); next } - { if (sz + $2 > CAP && index($1, prev) != 1) flush(); buf[++nb] = $0; sz += $2; prev = $1 } - END { flush(); if (nd) { f = name(++n); for (i = 1; i <= nd; i++) print dec[i] > f; close(f) } - printf "%d lines in %d batches\n", NR, n } - function name(i) { return sprintf("%s/batch-%04d.txt", DIR, i) } - function flush( i, f) { if (!nb) return; f = name(++n) - for (i = 1; i <= nb; i++) print buf[i] > f; close(f); nb = sz = 0; split("", buf) }' \ - {{.RUN}}/changed.txt + awk -F'\t' -v OFS='\t' -v CAP=$(({{.BATCH_GB | default 4}} * 1000000000)) -v DIR={{.RUN}} -v TL="{{.TL}}" + '{{.CUT_BATCHES}}' {{.RUN}}/changed.txt # ---- the batches ---- prepare: @@ -447,12 +664,7 @@ tasks: gpgv --status-fd 1 --keyring {{.RUN}}/tl/tlpkg/gpg/pubring.gpg "$f.asc" "$f" | awk -v k={{.TL_KEY}} '{{.GPGCHECK}}') || exit 1; done # Each TL container in this batch must agree with the checksum that the tlpdb gives - # for it. prepare examined that tlpdb. The check uses the two names that upstream - # has: - # - the stable name, which the tlpdb gives - # - the revision-stamped name, which the tlpdb gives through the revision. - # The two names have the same bytes, and thus the same checksum. The END clause is a - # second check: it rejects a container that is not in the tlpdb. + # for it (TLPDB_SUMS). prepare examined that tlpdb. # # The awk program writes to a file, not to a pipe. Thus, the status of the command # is the status of awk. verify operates without pipefail, because the grep in the @@ -470,16 +682,7 @@ tasks: # between the two. - >- cd {{.STAGING}} && { test "$(grep -c '^{{.TL}}/archive/' {{.B}})" = 0 || { - awk -F'\t' -v TL="{{.TL}}" 'NR == FNR { if (index($1, TL "/archive/") == 1) want[$1] = 1; next } - /^name /{split($0, a, " "); n = a[2]} /^revision /{split($0, a, " "); r = a[2]} - /^containerchecksum /{split($0, a, " "); emit(a[2], n)} - /^doccontainerchecksum /{split($0, a, " "); emit(a[2], n ".doc")} - /^srccontainerchecksum /{split($0, a, " "); emit(a[2], n ".source")} - function emit(sum, base, p, q) { p = TL "/archive/" base ".tar.xz"; q = TL "/archive/" base ".r" r ".tar.xz" - if (p in want) { print sum " " p; delete want[p] } - if (q in want) { print sum " " q; delete want[q] } } - END { for (p in want) { print "no checksum in the tlpdb for " p > "/dev/stderr"; bad = 1 } exit bad }' \ - {{.B}} {{.RUN}}/tl/tlpkg/texlive.tlpdb > {{.RUN}}/tl/sums.txt && + awk -F'\t' -v TL="{{.TL}}" '{{.TLPDB_SUMS}}' {{.B}} {{.RUN}}/tl/tlpkg/texlive.tlpdb > {{.RUN}}/tl/sums.txt && shasum -a 512 -c --quiet {{.RUN}}/tl/sums.txt; }; } # The decision batch. For this batch, these conditions must be correct: # - The tlpdb in the batch is the tlpdb that prepare examined at the start of the @@ -493,10 +696,7 @@ tasks: xz -dc {{.TL}}/tlpkg/texlive.tlpdb.xz | cmp - {{.TL}}/tlpkg/texlive.tlpdb && { cut -f1 {{.RUN}}/applied.txt; cut -f1 {{.B}}; } | LC_ALL=C sort -u | LC_ALL=C comm -23 - {{.RUN}}/deleted.txt > {{.RUN}}/tl/after.txt && - awk -v TL="{{.TL}}" '/^name /{n=$2} - /^containerchecksum /{print TL "/archive/" n ".tar.xz"} - /^doccontainerchecksum /{print TL "/archive/" n ".doc.tar.xz"} - /^srccontainerchecksum /{print TL "/archive/" n ".source.tar.xz"}' {{.TL}}/tlpkg/texlive.tlpdb \ + awk -v TL="{{.TL}}" '{{.TLPDB_NAMES}}' {{.TL}}/tlpkg/texlive.tlpdb | LC_ALL=C sort | LC_ALL=C comm -23 - {{.RUN}}/tl/after.txt > {{.RUN}}/tl/missing.txt && test ! -s {{.RUN}}/tl/missing.txt; }; } @@ -536,79 +736,14 @@ tasks: vars: DIR: '{{if .DIR}}{{.DIR}}{{else}}{{.STAGING}}{{end}}' cmds: - # The rules, in sequence: - # 1. A COMPRESSED ending gives the type directly. - # 2. An ending that Python reads as compressed and that COMPRESSED does not have - # (.br) gets the type of a signature, or application/octet-stream. It does not - # get text, because brotli has no signature to find. - # 3. If not, the mime.types of the image gives the type. But a type that a browser - # shows (text, XML, JSON, JavaScript, PDF) must agree with the bytes. If the - # bytes are compressed, contain a NUL, or are a PDF without %PDF-, they give - # the type. - # 4. A name that has no type in mime.types gets a type from its bytes. This type - # is a known signature, text/plain without a NUL, or application/octet-stream - # with a NUL. - # 5. A text/* type gets "; charset=utf-8" when the full file is UTF-8 (ASCII is - # UTF-8). A browser then shows its characters correctly. The bytes of other - # text do not tell their encoding. Thus, other text gets no charset. - # A control byte is not a NUL: some .sty files of CTAN contain ^Z, and they stay - # text. Without "./", guess_type reads a name with a colon as a URL scheme. + # The AWS CLI does not select the Content-Type, for two causes: + # - The CLI keeps the first value of the result of guess_type in Python. Thus, the + # CLI sends foo.diff.gz as a diff. + # - The results of the CLI change with the image. + # LABEL_PY gives the rules. - | python3 -I - {{shellQuote .DIR}} {{shellQuote .COMPRESSED}} > {{.RUN}}/labels.raw <<'PY' - import codecs, mimetypes, os, sys - - root, table = sys.argv[1], sys.argv[2] - endings = {s: f[0] for f in map(str.split, table.splitlines()) if f for s in f[1:]} - signatures = ((b"\x1f\x8b", "application/x-gzip"), (b"BZh", "application/x-bzip2"), - (b"\xfd7zXZ\x00", "application/x-xz"), (b"\x1f\x9d", "application/x-compress"), - (b"\x28\xb5\x2f\xfd", "application/zstd"), (b"LZIP", "application/x-lzip"), - (b"PK\x03\x04", "application/zip"), (b"%PDF-", "application/pdf")) - drawn = ("text/", "application/xml", "application/json", "application/javascript", "application/pdf") - - def from_bytes(head): - for sig, t in signatures: - if head.startswith(sig): - return t - text = b"\0" not in head or head.startswith((b"\xfe\xff", b"\xff\xfe")) - return "text/plain" if text else "application/octet-stream" - - def label(name, head): - for ending, t in endings.items(): - if name.endswith(ending): - return t - t, encoding = mimetypes.guess_type("./" + name) - seen = from_bytes(head) - if encoding: - return "application/octet-stream" if seen == "text/plain" else seen - if t is None: - return seen - if t.startswith(drawn) or t.endswith(("+xml", "+json")): - agrees = b"%PDF-" in head if t == "application/pdf" else seen == "text/plain" - return t if agrees else seen - return t - - # One MiB at a time: some text files are large (FILES.byname of CTAN is 27 MB). - def utf8(path): - d = codecs.getincrementaldecoder("utf-8")() - try: - with open(path, "rb") as h: - for part in iter(lambda: h.read(1 << 20), b""): - d.decode(part) - d.decode(b"", final=True) - except UnicodeDecodeError: - return False - return True - - out = sys.stdout.buffer - for d, _, files in os.walk(root): - for f in files: - p = os.path.join(d, f) - with open(p, "rb") as h: - head = h.read(1024) - t = label(f, head) - if t.startswith("text/") and utf8(p): - t += "; charset=utf-8" - out.write(os.fsencode(os.path.relpath(p, root)) + b"\t" + t.encode() + b"\n") + {{.LABEL_PY}} PY - LC_ALL=C sort {{.RUN}}/labels.raw > {{.RUN}}/labels.txt && rm {{.RUN}}/labels.raw @@ -696,9 +831,7 @@ tasks: # errors. Thus, the command uses grep to find them. - >- for f in {{.RUN}}/del-*; do - awk 'BEGIN { printf "{\"Objects\":[" } { gsub(/\\/, "\\\\"); gsub(/"/, "\\\"") - printf "%s{\"Key\":\"%s\"}", (NR > 1 ? "," : ""), $0 } END { print "],\"Quiet\":true}" }' "$f" \ - > "$f.json" && aws s3api delete-objects --bucket {{.BUCKET}} --delete "file://$f.json" | tee -a {{.RUN}}/publish.txt + awk '{{.DELETE_JSON}}' "$f" > "$f.json" && aws s3api delete-objects --bucket {{.BUCKET}} --delete "file://$f.json" | tee -a {{.RUN}}/publish.txt | { test "$(grep -c Errors)" = 0; }; done delete: @@ -825,18 +958,11 @@ tasks: { (cd {{.STAGING}} && find . -type f -exec md5sum {} +) | sed 's# \./#\tp\t#'; for d in {{.SLASH}}/*/; do test -d "$d" || continue; (cd "$d" && find . -type f -exec md5sum {} +) | sed 's# \./#\ts\t#'; done; } > {{.RUN}}/page-md5.txt - # A directory is the same if each of its keys is the same. The awk writes the same - # directories to index-same.txt, the other directories to index-make.new, and the - # files of the same directories to its output. The awk finds each input with - # FILENAME, because an empty file has no first line. + # SAME_PAGES writes the files of the same directories. xargs removes them. - >- awk -F'\t' -v E="{{.RUN}}/bucket-etags.txt" -v H="{{.RUN}}/page-md5.txt" -v I="{{.INDEX}}" -v S="{{.STAGING}}/" -v R="{{.SLASH}}/" - -v Y="{{.RUN}}/index-same.txt" -v M="{{.RUN}}/index-make.new" ' - FILENAME == E { gsub(/"/, "", $2); e[$1] = $2; next } - FILENAME == H { if ($2 == "s") d = $3 "/"; else if (substr($3, length($3) - length(I) + 1) == I) d = substr($3, 1, length($3) - length(I)); else next - seen[d] = 1; if (e[$3] != $1) diff[d] = 1; next } - !($0 in seen) || ($0 in diff) { print > M; next } - { print > Y; print S $0 I; if ($0 != "") { b = substr($0, 1, length($0) - 1); print R split(b, c, "/") "/" b } }' {{.RUN}}/bucket-etags.txt {{.RUN}}/page-md5.txt {{.RUN}}/index-make.txt \ + -v Y="{{.RUN}}/index-same.txt" -v M="{{.RUN}}/index-make.new" '{{.SAME_PAGES}}' + {{.RUN}}/bucket-etags.txt {{.RUN}}/page-md5.txt {{.RUN}}/index-make.txt | tr '\n' '\0' | xargs -0r rm -f - touch {{.RUN}}/index-same.txt {{.RUN}}/index-make.new && mv {{.RUN}}/index-make.new {{.RUN}}/index-make.txt @@ -923,93 +1049,17 @@ tasks: awk -v r="{{.SLASH}}/" '$0 != "" { n = split($0, c, "/"); d = "" for (i = 1; i < n - 1; i++) d = d c[i] "/"; print r (n - 1) "/" d }' {{.RUN}}/index-make.txt \ | LC_ALL=C sort -u | tr '\n' '\0' | xargs -0r mkdir -p - # The verb gets the entries, "dir TAB d|e|f TAB name TAB size TAB mtime", from the - # page state: the files in the bucket and the directories of upstream. A line "dir/" - # of the page state gives an e entry for that directory. The e entry opens the page - # and adds no row. Thus, a directory with no file also gets a page. After the sort, + # PAGE_ENTRIES gives the entries of each directory of index-make.txt. After the sort, # the lines of a directory are together, with its subdirectories first and the names - # in version sequence (sort -V: x-2.9 before x-2.10). Thus, the verb writes each page - # one time and closes it. The verb percent-encodes each name in href and HTML-escapes - # it in text. - # - # The verb keeps the rows of a directory in arrays. At shut, it writes the page to its - # two keys with the same calls. Thus, the two keys have the same bytes. With the base - # href, one document is correct for /dir/ and for /dir. Without it, a relative href in - # the second resolves against the parent. The footer is PAGE_FOOT, with %s for the - # encoded path of the directory. When PAGE_FOOT is empty, the verb makes the footer - # from UPSTREAM and CONTACT. CONTACT gets its default from HOST. - # - # rank gives the position of each name in INDEX_FILES. real is the index file of the - # open directory with the lowest rank. Without a real, shut writes the page key. - # - # G gives the suffixes of the signature files and the checksum files. The match - # ignores uppercase and lowercase. A file whose name is a different file of the - # directory plus one or more of these suffixes goes on the row of that file. + # in version sequence (sort -V: x-2.9 before x-2.10). Thus, PAGE_HTML writes each page + # one time and closes it. - >- test ! -s {{.RUN}}/index-make.txt || - awk -F'\t' -v OFS='\t' 'NR == FNR { want[$0] = 1; next } - { n = split($1, c, "/"); d = "" - for (i = 1; i < n; i++) { if ((d in want) && !((d c[i]) in seen)) { seen[d c[i]] = 1; print d, "d", c[i], "", "" }; d = d c[i] "/" } - if (d in want) print d, (c[n] == "" ? "e" : "f"), c[n], $2, $3 }' {{.RUN}}/index-make.txt {{.RUN}}/page-state.txt \ - | LC_ALL=C sort -t "$(printf '\t')" -k1,1 -k2,2 -k3,3V | awk -F'\t' -v S="{{.STAGING}}/" -v R="{{.SLASH}}/" -v I="{{.INDEX}}" -v F={{shellQuote (.PAGE_FOOT | default "")}} \ - -v N={{shellQuote (.INDEX_FILES | default "")}} -v X="{{.RUN}}/index-real.txt" -v U={{shellQuote (.UPSTREAM | default "")}} \ - -v C={{shellQuote (.CONTACT | default "")}} -v H="{{.HOST}}" '{{.URLENC}} - BEGIN { n = split(N, nm, " "); for (i = n; i >= 1; i--) rank[nm[i]] = i - G = "\\.(sig|sign|asc|gpg|dsig|proof|md5|md5sum|sha|sha1|sha224|sha256|sha256sum|sha512|sum|checksums)$" - if (C == "" && (j = index(H, ".")) > 1) C = substr(H, 1, j - 1) "@" substr(H, j + 1) - css = ":root{color-scheme:light dark;--bg:#f6f5f2;--fg:#1c1c28;--mut:#6e6e7e;--line:#e2e0da;--hov:#edebe5;--a:#3b45d9}" - css = css "@media (prefers-color-scheme:dark){:root{--bg:#121218;--fg:#f0f0f5;--mut:#8b8b9c;--line:#262630;--hov:#1a1a22;--a:#8b93ff}}" - css = css "body{margin:0 auto;max-width:60rem;padding:2rem 1rem 3rem;background:var(--bg);color:var(--fg);font:15px/1.45 system-ui,-apple-system,sans-serif}" - css = css "h1{font-size:1.35rem;font-weight:650;margin:0;overflow-wrap:anywhere}p{color:var(--mut);margin:.2rem 0 1.25rem;font-size:.9rem}" - css = css "table{width:100%;border-collapse:collapse;font:13.5px/1.4 ui-monospace,SFMono-Regular,Menlo,Consolas,monospace}" - css = css "th{font:600 .7rem/1 system-ui,sans-serif;letter-spacing:.06em;text-transform:uppercase;color:var(--mut);text-align:left;padding:0 .5rem .6rem;white-space:nowrap}" - css = css "td{padding:.4rem .5rem;border-top:1px solid var(--line);vertical-align:baseline}td:first-child{overflow-wrap:anywhere;width:100%}" - css = css "td+td{white-space:nowrap;color:var(--mut);font-variant-numeric:tabular-nums}th:last-child,td:last-child{text-align:right}" - css = css "tr:hover td{background:var(--hov)}a{color:var(--a);text-decoration:none}a:hover{text-decoration:underline}.s a{color:var(--mut)}" - css = css ".k{font-size:.72rem;margin-left:.6rem;padding:.05rem .4rem;border:1px solid var(--line);border-radius:.3rem;color:var(--mut)}" - css = css ".k:hover{color:var(--a);border-color:var(--a);text-decoration:none}" - css = css "footer{margin-top:2rem;padding-top:1rem;border-top:1px solid var(--line);color:var(--mut);font-size:.85rem;overflow-wrap:anywhere}" - css = css "footer a{color:inherit;text-decoration:underline}@media (max-width:36rem){thead{display:none}" - css = css "tr{display:grid;grid-template-columns:1fr auto;border-top:1px solid var(--line)}td{border:0;padding:.1rem .5rem}" - css = css "td:first-child{grid-column:1/-1;padding-top:.45rem}td+td{padding-bottom:.45rem;font-size:.8rem}}" } - function esc(s) { gsub(/&/, "\\&", s); gsub(//, "\\>", s); gsub(/"/, "\\"", s); return s } - function grp(n, s, o) { s = sprintf("%.0f", n); o = "" - while (length(s) > 3) { o = "," substr(s, length(s) - 2) o; s = substr(s, 1, length(s) - 3) } return s o } - function iec(n, u, i) { if (n < 1024) return sprintf("%.0f B", n) - split("KiB MiB GiB TiB", u, " "); n /= 1024; i = 1; while (n >= 1024 && i < 4) { n /= 1024; i++ } - return sprintf(n < 10 ? "%.1f %s" : "%.0f %s", n, u[i]) } - function ymd(t) { gsub(/\//, "-", t); return substr(t, 1, 16) } - function foot( t) { if (F != "") return "
" sprintf(F, enc(cur)) "
" - t = (U == "" ? "" : "Mirror of " esc(U) enc(cur) ".") - if (C != "") t = t (t == "" ? "" : " ") "Report a problem to " esc(C) "." - return (t == "" ? "" : "
" t "
") } - function open(d) { cur = d; live = 1; real = ""; nd = 0; nfl = 0; tot = 0; split("", has) } - function w(s) { if (real == "") printf("%s", s) > fa; if (cur != "") printf("%s", s) > fb } - function shut( i, r, x, c, e, k, b) { split("", fold); split("", links) - for (i = 1; i <= nfl; i++) { r = fname[i]; x = "" - while (match(tolower(r), G)) { r = substr(r, 1, RSTART - 1); if (r in has) x = r } - if (x != "") { fold[i] = 1 - links[x] = links[x] "" esc(substr(fname[i], length(x) + 1)) "" } } - c = (nd ? grp(nd) (nd == 1 ? " directory" : " directories") : "") - if (nfl) c = c (c == "" ? "" : ", ") grp(nfl) (nfl == 1 ? " file" : " files") ", " iec(tot) - if (c == "") c = "This directory is empty." - fa = S cur I; if (real != "") print cur real "\t" cur I > X - if (cur != "") { b = substr(cur, 1, length(cur) - 1); k = split(b, e, "/"); fb = R k "/" b } - w("\nIndex of /" esc(cur) "\n") - w("\n\n

Index of /" esc(cur) "

" c "

\n") - w("\n") - if (cur != "") w("\n") - for (i = 1; i <= nd; i++) w("\n") - for (i = 1; i <= nfl; i++) if (!(i in fold)) - w("\n") - w("
NameModified (UTC)Size
../
" esc(dname[i]) "/
" esc(fname[i]) "" links[fname[i]] "" ymd(ftime[i]) "" iec(fsize[i]) "
" foot() "\n") - if (real == "") close(fa); if (cur != "") close(fb) } - !live || $1 != cur { if (live) shut(); open($1) } - $2 == "e" { next } - $2 == "d" { dname[++nd] = $3; next } - ($3 in rank) && (real == "" || rank[$3] < rank[real]) { real = $3 } - { fname[++nfl] = $3; fsize[nfl] = $4; ftime[nfl] = $5; has[$3] = 1; tot += $4 } - END { if (live) shut() }' + awk -F'\t' -v OFS='\t' '{{.PAGE_ENTRIES}}' {{.RUN}}/index-make.txt {{.RUN}}/page-state.txt + | LC_ALL=C sort -t "$(printf '\t')" -k1,1 -k2,2 -k3,3V | awk -F'\t' -v S="{{.STAGING}}/" -v R="{{.SLASH}}/" -v I="{{.INDEX}}" + -v F={{shellQuote (.PAGE_FOOT | default "")}} -v N={{shellQuote (.INDEX_FILES | default "")}} -v X="{{.RUN}}/index-real.txt" + -v U={{shellQuote (.UPSTREAM | default "")}} -v C={{shellQuote (.CONTACT | default "")}} -v H="{{.HOST}}" + -v css={{shellQuote .PAGE_CSS}} '{{.PAGE_HTML}}' smoke: desc: Read a key of every type this run published back through the domain, size against the listing and type against the label; with TL set the tlpdb sha512 against the verified copy; with INDEX set one redrawn page under both keys; with CANARY set the canary, as a Perl client, against the bucket's copy; no Content-Encoding on a .tar.gz; warn of paths that never land; fresh; then the mirror's own checks, smoke-mirror diff --git a/examples/rsync/render.txt b/examples/rsync/render.txt index 6938945..65f43cd 100644 --- a/examples/rsync/render.txt +++ b/examples/rsync/render.txt @@ -29,16 +29,12 @@ task: [retry] for i in 1 2 3 4 5; do s=$((15 * 2 ** i + RANDOM % (2 * 15 + 1))); echo "rsync exit $rc, retry $i in ${s}s" >&2; sleep $s done; exit $rc -task: [normalise] awk ' - /^-/ && match($0, / [0-9][0-9][0-9][0-9]\/[0-9][0-9]\/[0-9][0-9] [0-9][0-9]:[0-9][0-9]:[0-9][0-9] /) { - path = substr($0, RSTART + RLENGTH); mtime = substr($0, RSTART + 1, RLENGTH - 2) - size = substr($0, 11, RSTART - 11); gsub(/[ ,]/, "", size) - print path "\t" size + 0 "\t" mtime }' /work/examples/rsync/.run/listing.txt \ -| LC_ALL=C sort > /work/examples/rsync/.run/upstream.txt -task: [normalise] awk ' - /^d/ && match($0, / [0-9][0-9][0-9][0-9]\/[0-9][0-9]\/[0-9][0-9] [0-9][0-9]:[0-9][0-9]:[0-9][0-9] /) { - path = substr($0, RSTART + RLENGTH); if (path != ".") print path "/" }' /work/examples/rsync/.run/listing.txt \ -| LC_ALL=C sort > /work/examples/rsync/.run/upstream-dirs.txt +task: [normalise] awk '/^-/ && match($0, / [0-9][0-9][0-9][0-9]\/[0-9][0-9]\/[0-9][0-9] [0-9][0-9]:[0-9][0-9]:[0-9][0-9] /) { + path = substr($0, RSTART + RLENGTH); mtime = substr($0, RSTART + 1, RLENGTH - 2) + size = substr($0, 11, RSTART - 11); gsub(/[ ,]/, "", size) + print path "\t" size + 0 "\t" mtime }' /work/examples/rsync/.run/listing.txt | LC_ALL=C sort > /work/examples/rsync/.run/upstream.txt +task: [normalise] awk '/^d/ && match($0, / [0-9][0-9][0-9][0-9]\/[0-9][0-9]\/[0-9][0-9] [0-9][0-9]:[0-9][0-9]:[0-9][0-9] /) { + path = substr($0, RSTART + RLENGTH); if (path != ".") print path "/" }' /work/examples/rsync/.run/listing.txt | LC_ALL=C sort > /work/examples/rsync/.run/upstream-dirs.txt task: [normalise] test "$(wc -l < /work/examples/rsync/.run/upstream.txt)" -gt 1 task: [state] mkdir -p /work/examples/rsync/.run task: [pull] rc=0; aws s3 ls s3://example/.state/applied.txt.xz > /dev/null || rc=$? @@ -66,16 +62,14 @@ task: [split] awk -F'\t' -v kb="$(df -Pk /work/examples/rsync/staging | awk 'NR '$2 / 1024 > kb - 1048576 { print "no room for " $1 " (" $2 " bytes)"; bad = 1 } END { exit bad }' \ /work/examples/rsync/.run/changed.txt task: [split] rm -f /work/examples/rsync/.run/batch-*.txt -task: [split] awk -F'\t' -v OFS='\t' -v CAP=$((4 * 1000000000)) -v DIR=/work/examples/rsync/.run -v TL="" ' - index($1, TL "/tlpkg/") == 1 || $1 !~ /\// { dec[++nd] = $0; next } - $2 > CAP { f = name(++n); print > f; close(f); next } - { if (sz + $2 > CAP && index($1, prev) != 1) flush(); buf[++nb] = $0; sz += $2; prev = $1 } - END { flush(); if (nd) { f = name(++n); for (i = 1; i <= nd; i++) print dec[i] > f; close(f) } - printf "%d lines in %d batches\n", NR, n } - function name(i) { return sprintf("%s/batch-%04d.txt", DIR, i) } - function flush( i, f) { if (!nb) return; f = name(++n) - for (i = 1; i <= nb; i++) print buf[i] > f; close(f); nb = sz = 0; split("", buf) }' \ -/work/examples/rsync/.run/changed.txt +task: [split] awk -F'\t' -v OFS='\t' -v CAP=$((4 * 1000000000)) -v DIR=/work/examples/rsync/.run -v TL="" 'index($1, TL "/tlpkg/") == 1 || $1 !~ /\// { dec[++nd] = $0; next } +$2 > CAP { f = name(++n); print > f; close(f); next } +{ if (sz + $2 > CAP && index($1, prev) != 1) flush(); buf[++nb] = $0; sz += $2; prev = $1 } +END { flush(); if (nd) { f = name(++n); for (i = 1; i <= nd; i++) print dec[i] > f; close(f) } + printf "%d lines in %d batches\n", NR, n } +function name(i) { return sprintf("%s/batch-%04d.txt", DIR, i) } +function flush( i, f) { if (!nb) return; f = name(++n) + for (i = 1; i <= nb; i++) print buf[i] > f; close(f); nb = sz = 0; split("", buf) }' /work/examples/rsync/.run/changed.txt task: [prepare] mkdir -p /work/examples/rsync/.run/tl task: [prepare] printf '%s\n' tlpkg/texlive.tlpdb.xz tlpkg/texlive.tlpdb.sha512 tlpkg/texlive.tlpdb.sha512.asc tlpkg/gpg/pubring.gpg > /work/examples/rsync/.run/tl/files.txt task: [retry] for i in 1 2 3 4 5; do @@ -98,20 +92,18 @@ done; exit $rc task: [verify] cd /work/examples/rsync/staging && cut -f1 | grep -E '^/(tlpkg/texlive\.tlpdb|[^/]+)\.sha512$' | while IFS= read -r f; do (cd "$(dirname "$f")" && f=$(basename "$f") && shasum -a 512 -c "$f" && gpgv --status-fd 1 --keyring /work/examples/rsync/.run/tl/tlpkg/gpg/pubring.gpg "$f.asc" "$f" | awk -v k= '/^\[GNUPG:\] GOODSIG /{g=1} /^\[GNUPG:\] VALIDSIG / && $NF == k {v=1} END{exit !(g&&v)}') || exit 1; done task: [verify] cd /work/examples/rsync/staging && { test "$(grep -c '^/archive/' )" = 0 || { awk -F'\t' -v TL="" 'NR == FNR { if (index($1, TL "/archive/") == 1) want[$1] = 1; next } - /^name /{split($0, a, " "); n = a[2]} /^revision /{split($0, a, " "); r = a[2]} - /^containerchecksum /{split($0, a, " "); emit(a[2], n)} - /^doccontainerchecksum /{split($0, a, " "); emit(a[2], n ".doc")} - /^srccontainerchecksum /{split($0, a, " "); emit(a[2], n ".source")} - function emit(sum, base, p, q) { p = TL "/archive/" base ".tar.xz"; q = TL "/archive/" base ".r" r ".tar.xz" - if (p in want) { print sum " " p; delete want[p] } - if (q in want) { print sum " " q; delete want[q] } } - END { for (p in want) { print "no checksum in the tlpdb for " p > "/dev/stderr"; bad = 1 } exit bad }' \ - /work/examples/rsync/.run/tl/tlpkg/texlive.tlpdb > /work/examples/rsync/.run/tl/sums.txt && shasum -a 512 -c --quiet /work/examples/rsync/.run/tl/sums.txt; }; } +/^name /{split($0, a, " "); n = a[2]} /^revision /{split($0, a, " "); r = a[2]} +/^containerchecksum /{split($0, a, " "); emit(a[2], n)} +/^doccontainerchecksum /{split($0, a, " "); emit(a[2], n ".doc")} +/^srccontainerchecksum /{split($0, a, " "); emit(a[2], n ".source")} +function emit(sum, base, p, q) { p = TL "/archive/" base ".tar.xz"; q = TL "/archive/" base ".r" r ".tar.xz" + if (p in want) { print sum " " p; delete want[p] } + if (q in want) { print sum " " q; delete want[q] } } +END { for (p in want) { print "no checksum in the tlpdb for " p > "/dev/stderr"; bad = 1 } exit bad }' /work/examples/rsync/.run/tl/tlpkg/texlive.tlpdb > /work/examples/rsync/.run/tl/sums.txt && shasum -a 512 -c --quiet /work/examples/rsync/.run/tl/sums.txt; }; } task: [verify] cd /work/examples/rsync/staging && { test ! -f /tlpkg/texlive.tlpdb || { cmp /tlpkg/texlive.tlpdb /work/examples/rsync/.run/tl/tlpkg/texlive.tlpdb && xz -dc /tlpkg/texlive.tlpdb.xz | cmp - /tlpkg/texlive.tlpdb && { cut -f1 /work/examples/rsync/.run/applied.txt; cut -f1 ; } | LC_ALL=C sort -u | LC_ALL=C comm -23 - /work/examples/rsync/.run/deleted.txt > /work/examples/rsync/.run/tl/after.txt && awk -v TL="" '/^name /{n=$2} - /^containerchecksum /{print TL "/archive/" n ".tar.xz"} - /^doccontainerchecksum /{print TL "/archive/" n ".doc.tar.xz"} - /^srccontainerchecksum /{print TL "/archive/" n ".source.tar.xz"}' /tlpkg/texlive.tlpdb \ -| LC_ALL=C sort | LC_ALL=C comm -23 - /work/examples/rsync/.run/tl/after.txt > /work/examples/rsync/.run/tl/missing.txt && test ! -s /work/examples/rsync/.run/tl/missing.txt; }; } +/^containerchecksum /{print TL "/archive/" n ".tar.xz"} +/^doccontainerchecksum /{print TL "/archive/" n ".doc.tar.xz"} +/^srccontainerchecksum /{print TL "/archive/" n ".source.tar.xz"}' /tlpkg/texlive.tlpdb | LC_ALL=C sort | LC_ALL=C comm -23 - /work/examples/rsync/.run/tl/after.txt > /work/examples/rsync/.run/tl/missing.txt && test ! -s /work/examples/rsync/.run/tl/missing.txt; }; } task: [label] python3 -I - /work/examples/rsync/staging $'application/x-gzip .gz .tgz .taz .tz .svgz\napplication/x-bzip2 .bz2 .tbz2 .tbz\napplication/x-xz .xz .txz\napplication/x-compress .Z\napplication/zstd .zst .tzst\napplication/x-lzip .lz\napplication/x-lzma .lzma' > /work/examples/rsync/.run/labels.raw <<'PY' import codecs, mimetypes, os, sys @@ -244,8 +236,7 @@ task: [checkpoint] find /work/examples/rsync/staging -mindepth 1 -delete task: [batches] test "$(ls /work/examples/rsync/.run/batch-*.txt 2>/dev/null | wc -l)" -le 4 || touch /work/examples/rsync/.run/chain task: [remove] rm -f /work/examples/rsync/.run/del-*; split -l 1000 -a 4 /work/examples/rsync/.run/deleted.txt /work/examples/rsync/.run/del- task: [remove] for f in /work/examples/rsync/.run/del-*; do awk 'BEGIN { printf "{\"Objects\":[" } { gsub(/\\/, "\\\\"); gsub(/"/, "\\\"") - printf "%s{\"Key\":\"%s\"}", (NR > 1 ? "," : ""), $0 } END { print "],\"Quiet\":true}" }' "$f" \ -> "$f.json" && aws s3api delete-objects --bucket example --delete "file://$f.json" | tee -a /work/examples/rsync/.run/publish.txt | { test "$(grep -c Errors)" = 0; }; done + printf "%s{\"Key\":\"%s\"}", (NR > 1 ? "," : ""), $0 } END { print "],\"Quiet\":true}" }' "$f" > "$f.json" && aws s3api delete-objects --bucket example --delete "file://$f.json" | tee -a /work/examples/rsync/.run/publish.txt | { test "$(grep -c Errors)" = 0; }; done task: [merge] (cd /work/examples/rsync/staging && find . -type f | sed 's|^\./||' | LC_ALL=C sort) > /work/examples/rsync/.run/landed.txt task: [merge] awk -F'\t' 'FILENAME == ARGV[1] { drop[$1] = 1; next } !($1 in drop)' /work/examples/rsync/.run/deleted.txt /work/examples/rsync/.run/applied.txt | cat - <(LC_ALL=C join -t "$(printf '\t')" /work/examples/rsync/.run/landed.txt /work/examples/rsync/.run/deleted.txt) | LC_ALL=C sort > /work/examples/rsync/.run/applied.new task: [push] xz -T0 -c /work/examples/rsync/.run/applied.new > /work/examples/rsync/.run/applied.new.xz @@ -264,8 +255,7 @@ task: [reconcile] cat /work/examples/rsync/.run/applied.txt /work/examples/rsync task: [reconcile] cut -f1 /work/examples/rsync/.run/bucket.txt | grep -vE '^\.state/' | awk -v i="/example.katoptra.org.directory.index.html" '{ s = "/" $0 } substr(s, length(s) - length(i) + 1) != i' | LC_ALL=C comm -23 - /work/examples/rsync/.run/index-keys.txt | LC_ALL=C comm -23 - /work/examples/rsync/.run/paths.txt > /work/examples/rsync/.run/orphans.txt task: [remove] rm -f /work/examples/rsync/.run/del-*; split -l 1000 -a 4 /work/examples/rsync/.run/orphans.txt /work/examples/rsync/.run/del- task: [remove] for f in /work/examples/rsync/.run/del-*; do awk 'BEGIN { printf "{\"Objects\":[" } { gsub(/\\/, "\\\\"); gsub(/"/, "\\\"") - printf "%s{\"Key\":\"%s\"}", (NR > 1 ? "," : ""), $0 } END { print "],\"Quiet\":true}" }' "$f" \ -> "$f.json" && aws s3api delete-objects --bucket example --delete "file://$f.json" | tee -a /work/examples/rsync/.run/publish.txt | { test "$(grep -c Errors)" = 0; }; done + printf "%s{\"Key\":\"%s\"}", (NR > 1 ? "," : ""), $0 } END { print "],\"Quiet\":true}" }' "$f" > "$f.json" && aws s3api delete-objects --bucket example --delete "file://$f.json" | tee -a /work/examples/rsync/.run/publish.txt | { test "$(grep -c Errors)" = 0; }; done task: [merge] (cd /work/examples/rsync/staging && find . -type f | sed 's|^\./||' | LC_ALL=C sort) > /work/examples/rsync/.run/landed.txt task: [merge] awk -F'\t' 'FILENAME == ARGV[1] { drop[$1] = 1; next } !($1 in drop)' /work/examples/rsync/.run/orphans.txt /work/examples/rsync/.run/applied.txt | cat - <(LC_ALL=C join -t "$(printf '\t')" /work/examples/rsync/.run/landed.txt /work/examples/rsync/.run/orphans.txt) | LC_ALL=C sort > /work/examples/rsync/.run/applied.new task: [push] xz -T0 -c /work/examples/rsync/.run/applied.new > /work/examples/rsync/.run/applied.new.xz @@ -300,87 +290,66 @@ task: [pages] test ! -s /work/examples/rsync/.run/index-make.txt || awk -v r="/w for (i = 1; i < n - 1; i++) d = d c[i] "/"; print r (n - 1) "/" d }' /work/examples/rsync/.run/index-make.txt \ | LC_ALL=C sort -u | tr '\n' '\0' | xargs -0r mkdir -p task: [pages] test ! -s /work/examples/rsync/.run/index-make.txt || awk -F'\t' -v OFS='\t' 'NR == FNR { want[$0] = 1; next } - { n = split($1, c, "/"); d = "" - for (i = 1; i < n; i++) { if ((d in want) && !((d c[i]) in seen)) { seen[d c[i]] = 1; print d, "d", c[i], "", "" }; d = d c[i] "/" } - if (d in want) print d, (c[n] == "" ? "e" : "f"), c[n], $2, $3 }' /work/examples/rsync/.run/index-make.txt /work/examples/rsync/.run/page-state.txt \ -| LC_ALL=C sort -t "$(printf '\t')" -k1,1 -k2,2 -k3,3V | awk -F'\t' -v S="/work/examples/rsync/staging/" -v R="/work/examples/rsync/.run/slash/" -v I="example.katoptra.org.directory.index.html" -v F='' \ - -v N='index.html index.htm' -v X="/work/examples/rsync/.run/index-real.txt" -v U='' \ - -v C='' -v H="example.katoptra.org" 'BEGIN { for (i = 0; i < 256; i++) ord[sprintf("%c", i)] = i } +{ n = split($1, c, "/"); d = "" + for (i = 1; i < n; i++) { if ((d in want) && !((d c[i]) in seen)) { seen[d c[i]] = 1; print d, "d", c[i], "", "" }; d = d c[i] "/" } + if (d in want) print d, (c[n] == "" ? "e" : "f"), c[n], $2, $3 }' /work/examples/rsync/.run/index-make.txt /work/examples/rsync/.run/page-state.txt | LC_ALL=C sort -t "$(printf '\t')" -k1,1 -k2,2 -k3,3V | awk -F'\t' -v S="/work/examples/rsync/staging/" -v R="/work/examples/rsync/.run/slash/" -v I="example.katoptra.org.directory.index.html" -v F='' -v N='index.html index.htm' -v X="/work/examples/rsync/.run/index-real.txt" -v U='' -v C='' -v H="example.katoptra.org" -v css=':root{color-scheme:light dark;--bg:#f6f5f2;--fg:#1c1c28;--mut:#6e6e7e;--line:#e2e0da;--hov:#edebe5;--a:#3b45d9}@media (prefers-color-scheme:dark){:root{--bg:#121218;--fg:#f0f0f5;--mut:#8b8b9c;--line:#262630;--hov:#1a1a22;--a:#8b93ff}}body{margin:0 auto;max-width:60rem;padding:2rem 1rem 3rem;background:var(--bg);color:var(--fg);font:15px/1.45 system-ui,-apple-system,sans-serif}h1{font-size:1.35rem;font-weight:650;margin:0;overflow-wrap:anywhere}p{color:var(--mut);margin:.2rem 0 1.25rem;font-size:.9rem}table{width:100%;border-collapse:collapse;font:13.5px/1.4 ui-monospace,SFMono-Regular,Menlo,Consolas,monospace}th{font:600 .7rem/1 system-ui,sans-serif;letter-spacing:.06em;text-transform:uppercase;color:var(--mut);text-align:left;padding:0 .5rem .6rem;white-space:nowrap}td{padding:.4rem .5rem;border-top:1px solid var(--line);vertical-align:baseline}td:first-child{overflow-wrap:anywhere;width:100%}td+td{white-space:nowrap;color:var(--mut);font-variant-numeric:tabular-nums}th:last-child,td:last-child{text-align:right}tr:hover td{background:var(--hov)}a{color:var(--a);text-decoration:none}a:hover{text-decoration:underline}.s a{color:var(--mut)}.k{font-size:.72rem;margin-left:.6rem;padding:.05rem .4rem;border:1px solid var(--line);border-radius:.3rem;color:var(--mut)}.k:hover{color:var(--a);border-color:var(--a);text-decoration:none}footer{margin-top:2rem;padding-top:1rem;border-top:1px solid var(--line);color:var(--mut);font-size:.85rem;overflow-wrap:anywhere}footer a{color:inherit;text-decoration:underline}@media (max-width:36rem){thead{display:none}tr{display:grid;grid-template-columns:1fr auto;border-top:1px solid var(--line)}td{border:0;padding:.1rem .5rem}td:first-child{grid-column:1/-1;padding-top:.45rem}td+td{padding-bottom:.45rem;font-size:.8rem}}' 'BEGIN { for (i = 0; i < 256; i++) ord[sprintf("%c", i)] = i } function enc(s, out, j, c) { out = ""; for (j = 1; j <= length(s); j++) { c = substr(s, j, 1) out = out ((c ~ /[A-Za-z0-9\/._~-]/) ? c : sprintf("%%%02X", ord[c])) } return out } - BEGIN { n = split(N, nm, " "); for (i = n; i >= 1; i--) rank[nm[i]] = i - G = "\\.(sig|sign|asc|gpg|dsig|proof|md5|md5sum|sha|sha1|sha224|sha256|sha256sum|sha512|sum|checksums)$" - if (C == "" && (j = index(H, ".")) > 1) C = substr(H, 1, j - 1) "@" substr(H, j + 1) - css = ":root{color-scheme:light dark;--bg:#f6f5f2;--fg:#1c1c28;--mut:#6e6e7e;--line:#e2e0da;--hov:#edebe5;--a:#3b45d9}" - css = css "@media (prefers-color-scheme:dark){:root{--bg:#121218;--fg:#f0f0f5;--mut:#8b8b9c;--line:#262630;--hov:#1a1a22;--a:#8b93ff}}" - css = css "body{margin:0 auto;max-width:60rem;padding:2rem 1rem 3rem;background:var(--bg);color:var(--fg);font:15px/1.45 system-ui,-apple-system,sans-serif}" - css = css "h1{font-size:1.35rem;font-weight:650;margin:0;overflow-wrap:anywhere}p{color:var(--mut);margin:.2rem 0 1.25rem;font-size:.9rem}" - css = css "table{width:100%;border-collapse:collapse;font:13.5px/1.4 ui-monospace,SFMono-Regular,Menlo,Consolas,monospace}" - css = css "th{font:600 .7rem/1 system-ui,sans-serif;letter-spacing:.06em;text-transform:uppercase;color:var(--mut);text-align:left;padding:0 .5rem .6rem;white-space:nowrap}" - css = css "td{padding:.4rem .5rem;border-top:1px solid var(--line);vertical-align:baseline}td:first-child{overflow-wrap:anywhere;width:100%}" - css = css "td+td{white-space:nowrap;color:var(--mut);font-variant-numeric:tabular-nums}th:last-child,td:last-child{text-align:right}" - css = css "tr:hover td{background:var(--hov)}a{color:var(--a);text-decoration:none}a:hover{text-decoration:underline}.s a{color:var(--mut)}" - css = css ".k{font-size:.72rem;margin-left:.6rem;padding:.05rem .4rem;border:1px solid var(--line);border-radius:.3rem;color:var(--mut)}" - css = css ".k:hover{color:var(--a);border-color:var(--a);text-decoration:none}" - css = css "footer{margin-top:2rem;padding-top:1rem;border-top:1px solid var(--line);color:var(--mut);font-size:.85rem;overflow-wrap:anywhere}" - css = css "footer a{color:inherit;text-decoration:underline}@media (max-width:36rem){thead{display:none}" - css = css "tr{display:grid;grid-template-columns:1fr auto;border-top:1px solid var(--line)}td{border:0;padding:.1rem .5rem}" - css = css "td:first-child{grid-column:1/-1;padding-top:.45rem}td+td{padding-bottom:.45rem;font-size:.8rem}}" } - function esc(s) { gsub(/&/, "\\&", s); gsub(//, "\\>", s); gsub(/"/, "\\"", s); return s } - function grp(n, s, o) { s = sprintf("%.0f", n); o = "" - while (length(s) > 3) { o = "," substr(s, length(s) - 2) o; s = substr(s, 1, length(s) - 3) } return s o } - function iec(n, u, i) { if (n < 1024) return sprintf("%.0f B", n) - split("KiB MiB GiB TiB", u, " "); n /= 1024; i = 1; while (n >= 1024 && i < 4) { n /= 1024; i++ } - return sprintf(n < 10 ? "%.1f %s" : "%.0f %s", n, u[i]) } - function ymd(t) { gsub(/\//, "-", t); return substr(t, 1, 16) } - function foot( t) { if (F != "") return "
" sprintf(F, enc(cur)) "
" - t = (U == "" ? "" : "Mirror of " esc(U) enc(cur) ".") - if (C != "") t = t (t == "" ? "" : " ") "Report a problem to " esc(C) "." - return (t == "" ? "" : "
" t "
") } - function open(d) { cur = d; live = 1; real = ""; nd = 0; nfl = 0; tot = 0; split("", has) } - function w(s) { if (real == "") printf("%s", s) > fa; if (cur != "") printf("%s", s) > fb } - function shut( i, r, x, c, e, k, b) { split("", fold); split("", links) - for (i = 1; i <= nfl; i++) { r = fname[i]; x = "" - while (match(tolower(r), G)) { r = substr(r, 1, RSTART - 1); if (r in has) x = r } - if (x != "") { fold[i] = 1 - links[x] = links[x] "" esc(substr(fname[i], length(x) + 1)) "" } } - c = (nd ? grp(nd) (nd == 1 ? " directory" : " directories") : "") - if (nfl) c = c (c == "" ? "" : ", ") grp(nfl) (nfl == 1 ? " file" : " files") ", " iec(tot) - if (c == "") c = "This directory is empty." - fa = S cur I; if (real != "") print cur real "\t" cur I > X - if (cur != "") { b = substr(cur, 1, length(cur) - 1); k = split(b, e, "/"); fb = R k "/" b } - w("\nIndex of /" esc(cur) "\n") - w("\n\n

Index of /" esc(cur) "

" c "

\n") - w("\n") - if (cur != "") w("\n") - for (i = 1; i <= nd; i++) w("\n") - for (i = 1; i <= nfl; i++) if (!(i in fold)) - w("\n") - w("
NameModified (UTC)Size
../
" esc(dname[i]) "/
" esc(fname[i]) "" links[fname[i]] "" ymd(ftime[i]) "" iec(fsize[i]) "
" foot() "\n") - if (real == "") close(fa); if (cur != "") close(fb) } - !live || $1 != cur { if (live) shut(); open($1) } - $2 == "e" { next } - $2 == "d" { dname[++nd] = $3; next } - ($3 in rank) && (real == "" || rank[$3] < rank[real]) { real = $3 } - { fname[++nfl] = $3; fsize[nfl] = $4; ftime[nfl] = $5; has[$3] = 1; tot += $4 } - END { if (live) shut() }' +BEGIN { n = split(N, nm, " "); for (i = n; i >= 1; i--) rank[nm[i]] = i + G = "\\.(sig|sign|asc|gpg|dsig|proof|md5|md5sum|sha|sha1|sha224|sha256|sha256sum|sha512|sum|checksums)$" + if (C == "" && (j = index(H, ".")) > 1) C = substr(H, 1, j - 1) "@" substr(H, j + 1) } +function esc(s) { gsub(/&/, "\\&", s); gsub(//, "\\>", s); gsub(/"/, "\\"", s); return s } +function grp(n, s, o) { s = sprintf("%.0f", n); o = "" + while (length(s) > 3) { o = "," substr(s, length(s) - 2) o; s = substr(s, 1, length(s) - 3) } return s o } +function iec(n, u, i) { if (n < 1024) return sprintf("%.0f B", n) + split("KiB MiB GiB TiB", u, " "); n /= 1024; i = 1; while (n >= 1024 && i < 4) { n /= 1024; i++ } + return sprintf(n < 10 ? "%.1f %s" : "%.0f %s", n, u[i]) } +function ymd(t) { gsub(/\//, "-", t); return substr(t, 1, 16) } +function foot( t) { if (F != "") return "
" sprintf(F, enc(cur)) "
" + t = (U == "" ? "" : "Mirror of " esc(U) enc(cur) ".") + if (C != "") t = t (t == "" ? "" : " ") "Report a problem to " esc(C) "." + return (t == "" ? "" : "
" t "
") } +function open(d) { cur = d; live = 1; real = ""; nd = 0; nfl = 0; tot = 0; split("", has) } +function w(s) { if (real == "") printf("%s", s) > fa; if (cur != "") printf("%s", s) > fb } +function shut( i, r, x, c, e, k, b) { split("", fold); split("", links) + for (i = 1; i <= nfl; i++) { r = fname[i]; x = "" + while (match(tolower(r), G)) { r = substr(r, 1, RSTART - 1); if (r in has) x = r } + if (x != "") { fold[i] = 1 + links[x] = links[x] "" esc(substr(fname[i], length(x) + 1)) "" } } + c = (nd ? grp(nd) (nd == 1 ? " directory" : " directories") : "") + if (nfl) c = c (c == "" ? "" : ", ") grp(nfl) (nfl == 1 ? " file" : " files") ", " iec(tot) + if (c == "") c = "This directory is empty." + fa = S cur I; if (real != "") print cur real "\t" cur I > X + if (cur != "") { b = substr(cur, 1, length(cur) - 1); k = split(b, e, "/"); fb = R k "/" b } + w("\nIndex of /" esc(cur) "\n") + w("\n\n

Index of /" esc(cur) "

" c "

\n") + w("\n") + if (cur != "") w("\n") + for (i = 1; i <= nd; i++) w("\n") + for (i = 1; i <= nfl; i++) if (!(i in fold)) + w("\n") + w("
NameModified (UTC)Size
../
" esc(dname[i]) "/
" esc(fname[i]) "" links[fname[i]] "" ymd(ftime[i]) "" iec(fsize[i]) "
" foot() "\n") + if (real == "") close(fa); if (cur != "") close(fb) } +!live || $1 != cur { if (live) shut(); open($1) } +$2 == "e" { next } +$2 == "d" { dname[++nd] = $3; next } +($3 in rank) && (real == "" || rank[$3] < rank[real]) { real = $3 } +{ fname[++nfl] = $3; fsize[nfl] = $4; ftime[nfl] = $5; has[$3] = 1; tot += $4 } +END { if (live) shut() }' task: [index-same] aws s3api list-objects-v2 --bucket example --query 'Contents[].[Key,ETag]' --output text > /work/examples/rsync/.run/bucket-etags.txt || { echo "index-same: the listing failed; index sends each page"; : > /work/examples/rsync/.run/bucket-etags.txt; } task: [index-same] { (cd /work/examples/rsync/staging && find . -type f -exec md5sum {} +) | sed 's# \./#\tp\t#'; for d in /work/examples/rsync/.run/slash/*/; do test -d "$d" || continue; (cd "$d" && find . -type f -exec md5sum {} +) | sed 's# \./#\ts\t#'; done; } > /work/examples/rsync/.run/page-md5.txt -task: [index-same] awk -F'\t' -v E="/work/examples/rsync/.run/bucket-etags.txt" -v H="/work/examples/rsync/.run/page-md5.txt" -v I="example.katoptra.org.directory.index.html" -v S="/work/examples/rsync/staging/" -v R="/work/examples/rsync/.run/slash/" -v Y="/work/examples/rsync/.run/index-same.txt" -v M="/work/examples/rsync/.run/index-make.new" ' - FILENAME == E { gsub(/"/, "", $2); e[$1] = $2; next } - FILENAME == H { if ($2 == "s") d = $3 "/"; else if (substr($3, length($3) - length(I) + 1) == I) d = substr($3, 1, length($3) - length(I)); else next - seen[d] = 1; if (e[$3] != $1) diff[d] = 1; next } - !($0 in seen) || ($0 in diff) { print > M; next } - { print > Y; print S $0 I; if ($0 != "") { b = substr($0, 1, length($0) - 1); print R split(b, c, "/") "/" b } }' /work/examples/rsync/.run/bucket-etags.txt /work/examples/rsync/.run/page-md5.txt /work/examples/rsync/.run/index-make.txt \ -| tr '\n' '\0' | xargs -0r rm -f +task: [index-same] awk -F'\t' -v E="/work/examples/rsync/.run/bucket-etags.txt" -v H="/work/examples/rsync/.run/page-md5.txt" -v I="example.katoptra.org.directory.index.html" -v S="/work/examples/rsync/staging/" -v R="/work/examples/rsync/.run/slash/" -v Y="/work/examples/rsync/.run/index-same.txt" -v M="/work/examples/rsync/.run/index-make.new" 'FILENAME == E { gsub(/"/, "", $2); e[$1] = $2; next } +FILENAME == H { if ($2 == "s") d = $3 "/"; else if (substr($3, length($3) - length(I) + 1) == I) d = substr($3, 1, length($3) - length(I)); else next + seen[d] = 1; if (e[$3] != $1) diff[d] = 1; next } +!($0 in seen) || ($0 in diff) { print > M; next } +{ print > Y; print S $0 I; if ($0 != "") { b = substr($0, 1, length($0) - 1); print R split(b, c, "/") "/" b } }' /work/examples/rsync/.run/bucket-etags.txt /work/examples/rsync/.run/page-md5.txt /work/examples/rsync/.run/index-make.txt | tr '\n' '\0' | xargs -0r rm -f task: [index-same] touch /work/examples/rsync/.run/index-same.txt /work/examples/rsync/.run/index-make.new && mv /work/examples/rsync/.run/index-make.new /work/examples/rsync/.run/index-make.txt task: [index] test ! -s /work/examples/rsync/.run/index-make.txt || aws s3 cp --no-progress --cli-connect-timeout 60 --cli-read-timeout 300 --recursive --content-type 'text/html; charset=utf-8' /work/examples/rsync/staging/ s3://example/ task: [index] for d in /work/examples/rsync/.run/slash/*/; do test -d "$d" || continue; aws s3 cp --no-progress --cli-connect-timeout 60 --cli-read-timeout 300 --recursive --content-type 'text/html; charset=utf-8' "$d" s3://example/; done task: [index-copy] while IFS="$(printf '\t')" read -r s k; do aws s3api copy-object --bucket example --copy-source "example/$s" --key "$k" --metadata-directive COPY > /dev/null; echo "copy: $s to $k"; done < /work/examples/rsync/.run/index-real.txt task: [remove] rm -f /work/examples/rsync/.run/del-*; split -l 1000 -a 4 /work/examples/rsync/.run/index-gone.txt /work/examples/rsync/.run/del- task: [remove] for f in /work/examples/rsync/.run/del-*; do awk 'BEGIN { printf "{\"Objects\":[" } { gsub(/\\/, "\\\\"); gsub(/"/, "\\\"") - printf "%s{\"Key\":\"%s\"}", (NR > 1 ? "," : ""), $0 } END { print "],\"Quiet\":true}" }' "$f" \ -> "$f.json" && aws s3api delete-objects --bucket example --delete "file://$f.json" | tee -a /work/examples/rsync/.run/publish.txt | { test "$(grep -c Errors)" = 0; }; done + printf "%s{\"Key\":\"%s\"}", (NR > 1 ? "," : ""), $0 } END { print "],\"Quiet\":true}" }' "$f" > "$f.json" && aws s3api delete-objects --bucket example --delete "file://$f.json" | tee -a /work/examples/rsync/.run/publish.txt | { test "$(grep -c Errors)" = 0; }; done task: [push] xz -T0 -c /work/examples/rsync/.run/indexed.new > /work/examples/rsync/.run/indexed.new.xz task: [push] aws s3 cp --no-progress --cli-connect-timeout 60 --cli-read-timeout 300 /work/examples/rsync/.run/indexed.new.xz s3://example/.state/indexed.txt.xz task: [smoke] { awk -F'\t' '!seen[$2]++' /work/examples/rsync/.run/labeled.txt 2>/dev/null; } | while IFS="$(printf '\t')" read -r k t; do u=$(printf '%s\n' "$k" | awk 'BEGIN { for (i = 0; i < 256; i++) ord[sprintf("%c", i)] = i } From 604e6d78bb122caf54df8cfec59b60f56a3f4e1f Mon Sep 17 00:00:00 2001 From: Josh Vaughen Date: Thu, 22 Oct 2026 13:58:33 -0700 Subject: [PATCH 2/6] refactor(rsync): one copy of each helper and each default - HEADERS, HEADROOM_KIB, NOT_WHOLE and MTIME_RE name the ranged read, the 1 GiB that split and fetch keep free, the whole-number rule of split and fresh, and the date pattern of a listing line. Each one had two or three copies. - split reads BATCH_GB through one task var, BATCH_SIZE. Its default was written two times. - Each awk program finds its first input with FILENAME == ARGV[1]. The three NR == FNR left (diff, TLPDB_SUMS, PAGE_ENTRIES) were safe only because of a guard. - report-engine is one script, and each row counts lines with n(). - The header gives the README table "The vars" as the one list of the vars that a mirror sets. No behavior change: the render differs only at the three FILENAME lines, offline passes, and parity shows 115 of 115 scenarios identical against v2.9.1, also the five report-engine scenarios. --- CLAUDE.md | 4 +- CONTRIBUTING.md | 8 ++-- README.md | 13 ++++-- engines/rsync.yml | 96 ++++++++++++++++----------------------- examples/rsync/render.txt | 6 +-- 5 files changed, 58 insertions(+), 69 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 048402d..bf0d2bb 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -25,7 +25,9 @@ Read that section before you write. command, the call var `TL_KEY` is available. - Some vars have a different name and read a var that a mirror can set, for example `BATCHES_MAX` from `MAX_BATCHES`. Such a var also gets the value on the command line - (verified). Thus, write each default one time, inline. + (verified). A task var with a different name, for example `BATCH_SIZE` in `split`, also + gets a call var (verified: `split-refuses` gives `BATCH_GB` as a call var). A global var + does not get a call var. Thus, write each default one time, inline or in a task var. - Do not put `dir:` on an engine verb. Task can read a flattened Taskfile of `includes:` from a path. In that condition, it adds each `dir:`, also an absolute one, to the directory of that Taskfile (verified, 3.53.1). Thus, `verify` starts each command with diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 9d1ca03..7455d32 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -5,9 +5,11 @@ are applicable to this repository. This repository adds three rules: - **Do not put a `vars:` default in `toolbox.yml` or in an engine for a var that a mirror sets.** If a `vars:` block has a default, the run uses that default, not the root value - of the mirror. Put each default inline in the command: `{{.X | default N}}`. `IMAGE` is - different: each engine sets `IMAGE: '{{.IMAGE | default "..."}}'` in `vars:`. Thus, if - the toolbox entry of `includes:` gives `IMAGE`, the run uses that value. + of the mirror. Put each default inline in the command: `{{.X | default N}}`. If two + commands of a verb read the var, put the default in a task var with a different name + (`BATCH_SIZE: '{{.BATCH_GB | default 4}}'` in `split`). `IMAGE` is different: each + engine sets `IMAGE: '{{.IMAGE | default "..."}}'` in `vars:`. Thus, if the toolbox entry + of `includes:` gives `IMAGE`, the run uses that value. - **Do not change the name of a verb in `toolbox.yml` or in an engine.** A mirror includes these files with `flatten: true`. Then the host verbs and the verbs in the container use one namespace. diff --git a/README.md b/README.md index 70f6a5a..c5495eb 100644 --- a/README.md +++ b/README.md @@ -288,8 +288,9 @@ different verb. Thus, the pipeline of a mirror has the same verbs with each engi An engine reads the root vars of the mirror and the environment. It has no `vars:` default for a var that a mirror sets. With such a default, the run uses the default, not the value -of the mirror. Each var that a mirror can set has its default inline: -`{{.X | default N}}`. +of the mirror. Each var that a mirror can set has its default in one location: inline, as +`{{.X | default N}}`, or in a task var with a different name, for example `BATCH_SIZE` in +`split`. `IMAGE` is different. The engine sets it in `vars:` as `{{.IMAGE | default "..."}}`. Thus, the run uses an `IMAGE` in the toolbox entry of `includes:`, not the default of the engine. @@ -603,8 +604,8 @@ A control byte is not a NUL. Thus, a `.sty` file of CTAN with a `^Z` at its end #### The vars -A mirror always sets `SOURCE`, `BUCKET` and `HOST` in its root vars. All other vars have an -inline default, and a mirror sets only the vars that are different: +A mirror always sets `SOURCE`, `BUCKET` and `HOST` in its root vars. All other vars have a +default in the engine, and a mirror sets only the vars that are different: | Var | Default | Function | |---|---|---| @@ -1386,7 +1387,9 @@ these rules: - A new hook is an empty task, with a `desc` that gives the mirror that fills it. The verb that runs the hook gives it the vars that it uses. - A new var that a mirror can set is an inline `{{.X | default N}}` in the command that - reads it. Write the default one time. Add a row to the table in [The vars](#the-vars). + reads it. If two commands of a verb read it, put the default in a task var with a + different name, for example `BATCH_SIZE` in `split`. Write the default one time. Add a + row to the table in [The vars](#the-vars). - Put each awk or Python program of more than two lines in the `programs` part of `vars:`. Write a comment above it that gives its function and its inputs. The command that runs the program gives it each input with awk `-v`, `ENVIRON` or an argument. A program has diff --git a/engines/rsync.yml b/engines/rsync.yml index 3c514b4..8546133 100644 --- a/engines/rsync.yml +++ b/engines/rsync.yml @@ -11,41 +11,8 @@ # a reconcile is due. RECONCILE and RECONCILE_HOURS are the vars of due. # The mirror puts these verbs in sequence in its pipeline. Its root vars identify it. # -# The vars that the engine reads from the mirror: -# - SOURCE, BUCKET and HOST. These vars are necessary. -# - These vars have an inline default: -# - CEILING_GB. A value of 0 (or no value) gives no ceiling. -# - BATCH_GB: 4. Digits, with no 0 first. -# - MAX_BATCHES: 4. -# - LIST_FLOOR. A value of 0 (or no value) gives no guard for a truncated listing. A -# listing must have more than LIST_FLOOR lines. If not, the run stops. -# - RETRY_BASE: 15 s. -# - TL and TL_KEY: a signed TeX Live subtree, and the fingerprint of the key that -# gives its signature. An empty value gives no signature checks. -# - FILTER: the rsync filter arguments for the listing. An empty value gives the full -# tree. -# - OWN: the keys at the bucket root that the mirror supplies, not upstream. reconcile -# does not remove them. Put a space between each two keys. An empty value gives -# none. -# - INDEX: the key suffix of the directory pages. When INDEX is set, index makes a -# page for each directory. -# - INDEX_FILES: the names of the upstream index files, in sequence, with a space -# between each two names. When a directory has such a file, its page key is a copy -# of the first one, and its key without the slash keeps the listing. An empty value -# gives a listing for each directory. -# - UPSTREAM: the URL of the upstream tree. The footer of each page gives it as text, -# with the path of the directory. An empty value gives a footer with no upstream. -# - CONTACT: the e-mail address for problem reports, in the footer of each page. An -# empty value gives the first label of HOST, "@" and the other labels of HOST. -# - PAGE_FOOT: HTML for the footer of each page, in place of the footer from UPSTREAM -# and CONTACT, with one %s for the encoded path of the directory and %% for a %. -# An empty value gives the footer from UPSTREAM and CONTACT. -# - CANARY: a file with bytes that the HTML rewriters of the zone can change. When -# CANARY is set, smoke reads the file through the domain as a Perl client. -# - FRESH_KEY: a file in which upstream records its clock. An empty value gives no -# check of that clock. -# - FRESH_HOURS: 24. Digits, with no 0 first. When the clock in FRESH_KEY is more -# than FRESH_HOURS before the time of the check, smoke stops the run with a failure. +# A mirror sets SOURCE, BUCKET and HOST. The table "The vars" in the README of lib gives +# each other var that the engine reads from the mirror, with its default. # # The label verb selects the Content-Type of each object. The AWS CLI does not select it. # @@ -95,6 +62,14 @@ vars: RSYNC: rsync --timeout=300 --contimeout=60 --no-h CURL: curl -fsS --connect-timeout 15 --max-time 60 --retry 6 --retry-connrefused --retry-max-time 600 AWS_FLAGS: --no-progress --cli-connect-timeout 60 --cli-read-timeout 300 + # The headers of a ranged read of one byte. SIZE and CTYPE read them. + HEADERS: '{{.CURL}} -r 0-0 -D - -o /dev/null' + # The KiB that split and fetch keep free on the disk of staging: 1 GiB for the text files + # of RUN and for the OS. + HEADROOM_KIB: '1048576' + # A case pattern for a value that is not a whole number: empty, a 0 first, or a character + # that is not a digit. Shell arithmetic reads 08 as octal and 0x1 as hexadecimal. + NOT_WHOLE: "''|0?*|*[!0-9]*" # ---- programs ---- # The awk and Python programs of the verbs. A program gets each input from awk -v, @@ -149,6 +124,9 @@ vars: if ($i ~ /^[0-9]+$/ && length($i) >= 9) { print $i; exit } if ($i ~ /^[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]-[0-9][0-9]-[0-9][0-9]$/) { split($i, d, "-"); print d[1] "-" d[2] "-" d[3] " " d[4] ":" d[5]; exit } } }' + # normalise: the mtime column of a line of rsync --list-only, with a space before and after + # it. It always has the same width. + MTIME_RE: ' [0-9][0-9][0-9][0-9]\/[0-9][0-9]\/[0-9][0-9] [0-9][0-9]:[0-9][0-9]:[0-9][0-9] ' # normalise: a line "path TAB size TAB mtime" for each regular file of a listing (rsync # --list-only). The size column can contain commas (rsync 3.x), or it can be empty for an # empty file (openrsync). Thus, the program finds the size from the date, which always has @@ -159,7 +137,7 @@ vars: # to it. -L resolves the two, and the mirror sends the two. A mirror that wants only the # stable names makes the listing smaller with FILTER. tlnet does this. LIST_FILES: |- - /^-/ && match($0, / [0-9][0-9][0-9][0-9]\/[0-9][0-9]\/[0-9][0-9] [0-9][0-9]:[0-9][0-9]:[0-9][0-9] /) { + /^-/ && match($0, /{{.MTIME_RE}}/) { path = substr($0, RSTART + RLENGTH); mtime = substr($0, RSTART + 1, RLENGTH - 2) size = substr($0, 11, RSTART - 11); gsub(/[ ,]/, "", size) print path "\t" size + 0 "\t" mtime } @@ -167,7 +145,7 @@ vars: # root (".") is not in the output. pages makes a page for each directory, also for a # directory with no file. LIST_DIRS: |- - /^d/ && match($0, / [0-9][0-9][0-9][0-9]\/[0-9][0-9]\/[0-9][0-9] [0-9][0-9]:[0-9][0-9]:[0-9][0-9] /) { + /^d/ && match($0, /{{.MTIME_RE}}/) { path = substr($0, RSTART + RLENGTH); if (path != ".") print path "/" } # split: divide the delta (changed.txt) into the batch files DIR/batch-NNNN.txt. CAP is the # number of bytes for each batch. The program adds the sizes in the listing. It uses only @@ -198,7 +176,7 @@ vars: # The two names have the same bytes, and thus the same checksum. The END clause is a second # check: it rejects a container that is not in the tlpdb. TLPDB_SUMS: |- - NR == FNR { if (index($1, TL "/archive/") == 1) want[$1] = 1; next } + FILENAME == ARGV[1] { if (index($1, TL "/archive/") == 1) want[$1] = 1; next } /^name /{split($0, a, " "); n = a[2]} /^revision /{split($0, a, " "); r = a[2]} /^containerchecksum /{split($0, a, " "); emit(a[2], n)} /^doccontainerchecksum /{split($0, a, " "); emit(a[2], n ".doc")} @@ -322,7 +300,7 @@ vars: # for that directory. The e entry opens the page and adds no row. Thus, a directory with no # file also gets a page. PAGE_ENTRIES: |- - NR == FNR { want[$0] = 1; next } + FILENAME == ARGV[1] { want[$0] = 1; next } { n = split($1, c, "/"); d = "" for (i = 1; i < n; i++) { if ((d in want) && !((d c[i]) in seen)) { seen[d c[i]] = 1; print d, "d", c[i], "", "" }; d = d c[i] "/" } if (d in want) print d, (c[n] == "" ? "e" : "f"), c[n], $2, $3 } @@ -552,19 +530,22 @@ tasks: # into the delta. Thus, verify examines it with the new sha512 in staging. The # pattern is the same as in verify. - >- - awk -F'\t' -v TL="{{.TL}}" 'NR == FNR { if ($1 ~ "^" TL "/(tlpkg/texlive\\.tlpdb|[^/]+)\\.sha512$") want[substr($1, 1, length($1) - 7)]; next } + awk -F'\t' -v TL="{{.TL}}" 'FILENAME == ARGV[1] { if ($1 ~ "^" TL "/(tlpkg/texlive\\.tlpdb|[^/]+)\\.sha512$") want[substr($1, 1, length($1) - 7)]; next } $1 in want' {{.RUN}}/changed.txt {{.RUN}}/upstream.txt | LC_ALL=C sort -u -o {{.RUN}}/changed.txt - {{.RUN}}/changed.txt - cut -f1 {{.RUN}}/upstream.txt > {{.RUN}}/paths.txt - cut -f1 {{.RUN}}/applied.txt | LC_ALL=C comm -23 - {{.RUN}}/paths.txt > {{.RUN}}/deleted.txt split: desc: Refuse a tree over CEILING_GB (when set) or a file the disk cannot hold; split changed.txt into RUN/batch-NNNN.txt + vars: + # A task var, not a global var, because a caller can give BATCH_GB as a call var. + BATCH_SIZE: '{{.BATCH_GB | default 4}}' cmds: # BATCH_GB goes into shell arithmetic below. There, 08 is 0 and 0x1 is 1. Thus, this # command stops the run on each value that is not a whole number. - >- - b={{shellQuote (print (.BATCH_GB | default 4))}}; - case "$b" in ''|0?*|*[!0-9]*) echo "split: BATCH_GB is a whole number, not $b" >&2; exit 1 ;; esac + b={{shellQuote .BATCH_SIZE}}; + case "$b" in {{.NOT_WHOLE}}) echo "split: BATCH_GB is a whole number, not $b" >&2; exit 1 ;; esac # CEILING_GB unset or 0 gives no ceiling. The command always prints the size. - >- {{if .CEILING_GB}}awk -F'\t' -v gb={{.CEILING_GB}} '{ s += $2 } @@ -576,14 +557,14 @@ tasks: # RUN and for the OS. - >- awk -F'\t' -v kb="$(df -Pk {{.STAGING}} | awk 'NR == 2 { print $4 }')" \ - '$2 / 1024 > kb - 1048576 { print "no room for " $1 " (" $2 " bytes)"; bad = 1 } END { exit bad }' \ + '$2 / 1024 > kb - {{.HEADROOM_KIB}} { print "no room for " $1 " (" $2 " bytes)"; bad = 1 } END { exit bad }' \ {{.RUN}}/changed.txt - rm -f {{.RUN}}/batch-*.txt # BATCH_GB goes into the command with no quotes, because it is in $(( )), where a # quoted operand is a syntax error. The first command of split stops the run before # this line on each value that is not a whole number. - >- - awk -F'\t' -v OFS='\t' -v CAP=$(({{.BATCH_GB | default 4}} * 1000000000)) -v DIR={{.RUN}} -v TL="{{.TL}}" + awk -F'\t' -v OFS='\t' -v CAP=$(({{.BATCH_SIZE}} * 1000000000)) -v DIR={{.RUN}} -v TL="{{.TL}}" '{{.CUT_BATCHES}}' {{.RUN}}/changed.txt # ---- the batches ---- @@ -630,7 +611,7 @@ tasks: internal: true requires: {vars: [B]} preconditions: - - sh: test "$(df -Pk {{.STAGING}} | awk 'NR == 2 { print $4 }')" -gt "$(awk -F'\t' '{ s += $2 } END { print int(s / 1024) + 1048576 }' {{.B}})" + - sh: test "$(df -Pk {{.STAGING}} | awk 'NR == 2 { print $4 }')" -gt "$(awk -F'\t' '{ s += $2 } END { print int(s / 1024) + {{.HEADROOM_KIB}} }' {{.B}})" msg: "not enough free disk for {{.B}}" cmds: - find {{.STAGING}} -mindepth 1 -delete @@ -1091,7 +1072,7 @@ tasks: | while IFS="$(printf '\t')" read -r k t; do u=$(printf '%s\n' "$k" | {{.ENCODE}} | cut -f2); want=$(awk -F'\t' -v k="$k" '$1 == k { print $2 }' {{.RUN}}/upstream.txt); - h=$({{.CURL}} -r 0-0 -D - -o /dev/null "{{.URL}}/$u"); + h=$({{.HEADERS}} "{{.URL}}/$u"); got=$(printf '%s\n' "$h" | {{.SIZE}}); ct=$(printf '%s\n' "$h" | {{.CTYPE}}); echo "$k: $got bytes (listing: $want), served as ${ct:-no type} (label: ${t:-none})"; test "$got" = "$want" || exit 1; @@ -1119,7 +1100,7 @@ tasks: # together. - >- {{if .INDEX}}test ! -s {{.RUN}}/index-make.txt || { d=$(tail -1 {{.RUN}}/index-make.txt); - r() { h=$({{.CURL}} -r 0-0 -D - -o /dev/null "{{.URL}}/$(printf '%s\n' "$1" | {{.ENCODE}} | cut -f2)"); + r() { h=$({{.HEADERS}} "{{.URL}}/$(printf '%s\n' "$1" | {{.ENCODE}} | cut -f2)"); echo "$(printf '%s\n' "$h" | {{.SIZE}}) bytes, type $(printf '%s\n' "$h" | {{.CTYPE}})"; }; s=$(awk -F'\t' -v k="${d}{{.INDEX}}" '$2 == k { print $1 }' {{.RUN}}/index-real.txt); if test -n "$s"; then want=$(r "$s"); @@ -1164,7 +1145,7 @@ tasks: g=$(awk -F'\t' '$1 ~ /\.tar\.gz$/ && $2 > 0 { print $1; exit }' {{.RUN}}/applied.txt 2>/dev/null); case "{{.URL}}" in http*) ;; *) g= ;; esac; if test -z "$g"; then echo "content-encoding check skipped (no non-empty .tar.gz in the state, or the URL is not http)"; else - u=$(printf '%s\n' "$g" | {{.ENCODE}} | cut -f2) && h=$({{.CURL}} -r 0-0 -D - -o /dev/null "{{.URL}}/$u") && + u=$(printf '%s\n' "$g" | {{.ENCODE}} | cut -f2) && h=$({{.HEADERS}} "{{.URL}}/$u") && if printf '%s' "$h" | grep -qi '^content-encoding:'; then echo "$g: served with a content-encoding" >&2; exit 1; fi && echo "$g: served with no content-encoding"; fi # A path in the listing of upstream that rsync cannot send (exit 23, which retry @@ -1192,7 +1173,7 @@ tasks: {{if .FRESH_KEY}}k={{shellQuote .FRESH_KEY}}; if ! awk -F'\t' -v k="$k" '$1 == k { f = 1 } END { exit !f }' {{.RUN}}/applied.txt 2>/dev/null; then echo "freshness skipped ($k is not in the state)"; else - h={{shellQuote .FRESH_H}}; case "$h" in ''|0?*|*[!0-9]*) echo "fresh: FRESH_HOURS is a whole number, not $h" >&2; exit 1 ;; esac; + h={{shellQuote .FRESH_H}}; case "$h" in {{.NOT_WHOLE}}) echo "fresh: FRESH_HOURS is a whole number, not $h" >&2; exit 1 ;; esac; u=$(printf '%s\n' "$k" | {{.ENCODE}} | cut -f2); t=$({{.CURL}} "{{.URL}}/$u" | {{.STAMP}}); case "$t" in *-*) t=$(date -u -d "$t" +%s) ;; esac; test -n "$t"; @@ -1211,20 +1192,21 @@ tasks: - | n() { test -f "$1" && wc -l < "$1" || echo 0; } gb() { test -f "$1" && awk -F'\t' '{ s += $2 } END { printf "%.2f", s / 1e9 }' "$1" || echo 0.00; } - cat >> "${GITHUB_STEP_SUMMARY:-/dev/stdout}" <> "$out" </dev/null | wc -l | awk -v m={{.BATCHES_MAX}} '{ print $1 " batches" ($1 > m ? ", " $1 - m " left for the next run" : "") }'); $(n {{.RUN}}/deleted.txt) deleted, $(n {{.RUN}}/orphans.txt) orphaned | | Published to R2 | $(awk '/^upload:/ { n++ } END { print n + 0 }' {{.RUN}}/publish.txt 2>/dev/null || echo 0) uploaded, $(cut -f2 {{.RUN}}/labeled.txt 2>/dev/null | sort -u | wc -l | tr -d ' ') types | | State | $(n {{.RUN}}/applied.txt) lines in {{.STATE}} | | Storage | $(test -f {{.RUN}}/bucket.txt && echo "$(gb {{.RUN}}/bucket.txt) GB in $(n {{.RUN}}/bucket.txt) objects (reconciled)" || echo "not measured this run"){{if .CEILING_GB}} of the {{.CEILING_GB}} GB ceiling{{end}} | EOF - # prepare made RUN/tl. Thus, the delta had a path in the signed subtree. - - test ! -d {{.RUN}}/tl || echo "| Signature | $(test -f {{.RUN}}/tl/tlpkg/texlive.tlpdb && echo "tlpdb signed by {{.TL_KEY}}; $(test -f {{.RUN}}/tl/missing.txt && wc -l < {{.RUN}}/tl/missing.txt || echo 0) named containers missing from the bucket" || echo "the tlpdb did not verify") |" >> "${GITHUB_STEP_SUMMARY:-/dev/stdout}" - # With INDEX set, the pages that index made and the keys that it removed. - - test -z "{{.INDEX}}" || echo "| Directory pages | $(test -f {{.RUN}}/index-make.txt && wc -l < {{.RUN}}/index-make.txt || echo 0) redrawn and sent, $(test -f {{.RUN}}/index-same.txt && wc -l < {{.RUN}}/index-same.txt || echo 0) redrawn and not sent (the bucket had the same bytes), $(test -f {{.RUN}}/index-real.txt && wc -l < {{.RUN}}/index-real.txt || echo 0) with a copy of an index file as the page key, $(test -f {{.RUN}}/index-gone.txt && wc -l < {{.RUN}}/index-gone.txt || echo 0) keys removed |" >> "${GITHUB_STEP_SUMMARY:-/dev/stdout}" - # Rows that show only when they have data. - - test ! -s {{.RUN}}/unlanded.txt || echo "| Never landed | $(wc -l < {{.RUN}}/unlanded.txt | tr -d ' ') listed upstream that rsync could not send, fetched again every run; the first are warnings on the run |" >> "${GITHUB_STEP_SUMMARY:-/dev/stdout}" - - test ! -f {{.RUN}}/fresh.txt || echo "| Upstream clock | $(awk '{ printf "%d h %d min", $1 / 3600, $1 % 3600 / 60 }' {{.RUN}}/fresh.txt) old in {{.FRESH_KEY}}, limit {{.FRESH_H}} h |" >> "${GITHUB_STEP_SUMMARY:-/dev/stdout}" + # prepare made RUN/tl. Thus, the delta had a path in the signed subtree. + test ! -d {{.RUN}}/tl || echo "| Signature | $(test -f {{.RUN}}/tl/tlpkg/texlive.tlpdb && echo "tlpdb signed by {{.TL_KEY}}; $(n {{.RUN}}/tl/missing.txt) named containers missing from the bucket" || echo "the tlpdb did not verify") |" >> "$out" + # With INDEX set, the pages that index made and the keys that it removed. + test -z "{{.INDEX}}" || echo "| Directory pages | $(n {{.RUN}}/index-make.txt) redrawn and sent, $(n {{.RUN}}/index-same.txt) redrawn and not sent (the bucket had the same bytes), $(n {{.RUN}}/index-real.txt) with a copy of an index file as the page key, $(n {{.RUN}}/index-gone.txt) keys removed |" >> "$out" + # Rows that show only when they have data. + test ! -s {{.RUN}}/unlanded.txt || echo "| Never landed | $(wc -l < {{.RUN}}/unlanded.txt | tr -d ' ') listed upstream that rsync could not send, fetched again every run; the first are warnings on the run |" >> "$out" + test ! -f {{.RUN}}/fresh.txt || echo "| Upstream clock | $(awk '{ printf "%d h %d min", $1 / 3600, $1 % 3600 / 60 }' {{.RUN}}/fresh.txt) old in {{.FRESH_KEY}}, limit {{.FRESH_H}} h |" >> "$out" retry: desc: Run CMD, retrying rsync's transport exit codes (5 10 12 30 35) five times with backoff and jitter; 23 (a path it could not read, a dangling symlink under -L) and 24 (file vanished mid-transfer) are successes diff --git a/examples/rsync/render.txt b/examples/rsync/render.txt index 65f43cd..7085eda 100644 --- a/examples/rsync/render.txt +++ b/examples/rsync/render.txt @@ -51,7 +51,7 @@ task: [push] aws s3 cp --no-progress --cli-connect-timeout 60 --cli-read-timeout task: [rebuild] mv /work/examples/rsync/.run/applied.new /work/examples/rsync/.run/applied.txt task: [rebuild] rm -f /work/examples/rsync/.run/rebuild-now task: [diff] LC_ALL=C comm -13 /work/examples/rsync/.run/applied.txt /work/examples/rsync/.run/upstream.txt > /work/examples/rsync/.run/changed.txt -task: [diff] awk -F'\t' -v TL="" 'NR == FNR { if ($1 ~ "^" TL "/(tlpkg/texlive\\.tlpdb|[^/]+)\\.sha512$") want[substr($1, 1, length($1) - 7)]; next } $1 in want' /work/examples/rsync/.run/changed.txt /work/examples/rsync/.run/upstream.txt | LC_ALL=C sort -u -o /work/examples/rsync/.run/changed.txt - /work/examples/rsync/.run/changed.txt +task: [diff] awk -F'\t' -v TL="" 'FILENAME == ARGV[1] { if ($1 ~ "^" TL "/(tlpkg/texlive\\.tlpdb|[^/]+)\\.sha512$") want[substr($1, 1, length($1) - 7)]; next } $1 in want' /work/examples/rsync/.run/changed.txt /work/examples/rsync/.run/upstream.txt | LC_ALL=C sort -u -o /work/examples/rsync/.run/changed.txt - /work/examples/rsync/.run/changed.txt task: [diff] cut -f1 /work/examples/rsync/.run/upstream.txt > /work/examples/rsync/.run/paths.txt task: [diff] cut -f1 /work/examples/rsync/.run/applied.txt | LC_ALL=C comm -23 - /work/examples/rsync/.run/paths.txt > /work/examples/rsync/.run/deleted.txt task: [split] b=4; case "$b" in ''|0?*|*[!0-9]*) echo "split: BATCH_GB is a whole number, not $b" >&2; exit 1 ;; esac @@ -91,7 +91,7 @@ task: [retry] for i in 1 2 3 4 5; do done; exit $rc task: [verify] cd /work/examples/rsync/staging && cut -f1 | grep -E '^/(tlpkg/texlive\.tlpdb|[^/]+)\.sha512$' | while IFS= read -r f; do (cd "$(dirname "$f")" && f=$(basename "$f") && shasum -a 512 -c "$f" && gpgv --status-fd 1 --keyring /work/examples/rsync/.run/tl/tlpkg/gpg/pubring.gpg "$f.asc" "$f" | awk -v k= '/^\[GNUPG:\] GOODSIG /{g=1} /^\[GNUPG:\] VALIDSIG / && $NF == k {v=1} END{exit !(g&&v)}') || exit 1; done -task: [verify] cd /work/examples/rsync/staging && { test "$(grep -c '^/archive/' )" = 0 || { awk -F'\t' -v TL="" 'NR == FNR { if (index($1, TL "/archive/") == 1) want[$1] = 1; next } +task: [verify] cd /work/examples/rsync/staging && { test "$(grep -c '^/archive/' )" = 0 || { awk -F'\t' -v TL="" 'FILENAME == ARGV[1] { if (index($1, TL "/archive/") == 1) want[$1] = 1; next } /^name /{split($0, a, " "); n = a[2]} /^revision /{split($0, a, " "); r = a[2]} /^containerchecksum /{split($0, a, " "); emit(a[2], n)} /^doccontainerchecksum /{split($0, a, " "); emit(a[2], n ".doc")} @@ -289,7 +289,7 @@ task: [pages] test ! -s /work/examples/rsync/.run/index-make.txt || awk -v s="/w task: [pages] test ! -s /work/examples/rsync/.run/index-make.txt || awk -v r="/work/examples/rsync/.run/slash/" '$0 != "" { n = split($0, c, "/"); d = "" for (i = 1; i < n - 1; i++) d = d c[i] "/"; print r (n - 1) "/" d }' /work/examples/rsync/.run/index-make.txt \ | LC_ALL=C sort -u | tr '\n' '\0' | xargs -0r mkdir -p -task: [pages] test ! -s /work/examples/rsync/.run/index-make.txt || awk -F'\t' -v OFS='\t' 'NR == FNR { want[$0] = 1; next } +task: [pages] test ! -s /work/examples/rsync/.run/index-make.txt || awk -F'\t' -v OFS='\t' 'FILENAME == ARGV[1] { want[$0] = 1; next } { n = split($1, c, "/"); d = "" for (i = 1; i < n; i++) { if ((d in want) && !((d c[i]) in seen)) { seen[d c[i]] = 1; print d, "d", c[i], "", "" }; d = d c[i] "/" } if (d in want) print d, (c[n] == "" ? "e" : "f"), c[n], $2, $3 }' /work/examples/rsync/.run/index-make.txt /work/examples/rsync/.run/page-state.txt | LC_ALL=C sort -t "$(printf '\t')" -k1,1 -k2,2 -k3,3V | awk -F'\t' -v S="/work/examples/rsync/staging/" -v R="/work/examples/rsync/.run/slash/" -v I="example.katoptra.org.directory.index.html" -v F='' -v N='index.html index.htm' -v X="/work/examples/rsync/.run/index-real.txt" -v U='' -v C='' -v H="example.katoptra.org" -v css=':root{color-scheme:light dark;--bg:#f6f5f2;--fg:#1c1c28;--mut:#6e6e7e;--line:#e2e0da;--hov:#edebe5;--a:#3b45d9}@media (prefers-color-scheme:dark){:root{--bg:#121218;--fg:#f0f0f5;--mut:#8b8b9c;--line:#262630;--hov:#1a1a22;--a:#8b93ff}}body{margin:0 auto;max-width:60rem;padding:2rem 1rem 3rem;background:var(--bg);color:var(--fg);font:15px/1.45 system-ui,-apple-system,sans-serif}h1{font-size:1.35rem;font-weight:650;margin:0;overflow-wrap:anywhere}p{color:var(--mut);margin:.2rem 0 1.25rem;font-size:.9rem}table{width:100%;border-collapse:collapse;font:13.5px/1.4 ui-monospace,SFMono-Regular,Menlo,Consolas,monospace}th{font:600 .7rem/1 system-ui,sans-serif;letter-spacing:.06em;text-transform:uppercase;color:var(--mut);text-align:left;padding:0 .5rem .6rem;white-space:nowrap}td{padding:.4rem .5rem;border-top:1px solid var(--line);vertical-align:baseline}td:first-child{overflow-wrap:anywhere;width:100%}td+td{white-space:nowrap;color:var(--mut);font-variant-numeric:tabular-nums}th:last-child,td:last-child{text-align:right}tr:hover td{background:var(--hov)}a{color:var(--a);text-decoration:none}a:hover{text-decoration:underline}.s a{color:var(--mut)}.k{font-size:.72rem;margin-left:.6rem;padding:.05rem .4rem;border:1px solid var(--line);border-radius:.3rem;color:var(--mut)}.k:hover{color:var(--a);border-color:var(--a);text-decoration:none}footer{margin-top:2rem;padding-top:1rem;border-top:1px solid var(--line);color:var(--mut);font-size:.85rem;overflow-wrap:anywhere}footer a{color:inherit;text-decoration:underline}@media (max-width:36rem){thead{display:none}tr{display:grid;grid-template-columns:1fr auto;border-top:1px solid var(--line)}td{border:0;padding:.1rem .5rem}td:first-child{grid-column:1/-1;padding-top:.45rem}td+td{padding-bottom:.45rem;font-size:.8rem}}' 'BEGIN { for (i = 0; i < 256; i++) ord[sprintf("%c", i)] = i } From 7dad6353eba1157a70f88073fdff8e8cc2450176 Mon Sep 17 00:00:00 2001 From: Josh Vaughen Date: Thu, 22 Oct 2026 14:03:52 -0700 Subject: [PATCH 3/6] refactor(rsync): split smoke, verify and pages into named steps Each check of smoke, each check of verify, and the two halves of pages become an internal step with a desc: - smoke: smoke-sample, smoke-tlpdb, smoke-page, smoke-canary, smoke-encoding and smoke-unlanded, then fresh and smoke-mirror as before - verify: verify-signed, verify-containers and verify-decision - pages: pages-select (the directories and the keys) and pages-draw A step does not see the call vars of its verb. Thus, each verb gives its steps the vars that they read with one YAML anchor. The anchors hold only text vars, because a root X: 0 goes through as "0". No behavior change: the render has the same commands in the same order, with the step names as labels. offline passes. Parity shows the same tree and exit code in 115 of 115 scenarios against v2.9.1. In 18 scenarios that fail on purpose, go-task's failure line also names the step: task: Failed to run task "smoke": task: Failed to run task "smoke-canary" Nothing in lib or the workflows reads that line. --- CLAUDE.md | 32 ++++++---- README.md | 32 ++++++---- engines/rsync.yml | 130 +++++++++++++++++++++++++++++++++----- examples/rsync/render.txt | 40 ++++++------ toolbox.yml | 9 ++- 5 files changed, 183 insertions(+), 60 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index bf0d2bb..941d2cb 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -30,8 +30,18 @@ Read that section before you write. does not get a call var. Thus, write each default one time, inline or in a task var. - Do not put `dir:` on an engine verb. Task can read a flattened Taskfile of `includes:` from a path. In that condition, it adds each `dir:`, also an absolute one, to the - directory of that Taskfile (verified, 3.53.1). Thus, `verify` starts each command with - `cd {{.STAGING}}`. + directory of that Taskfile (verified, 3.53.1). Thus, each step of `verify` starts each + command with `cd {{.STAGING}}`. +- A task that a verb starts does not get the call vars of that verb. Thus, `smoke`, + `verify` and `pages` give their steps the vars that the steps read, with a YAML anchor + (`vars: &smoke`, then `vars: *smoke`). Give only text vars in this method. A call var is + text: a root `X: 0` goes through as "0", and then `{{if .X}}` is true (verified, 3.53.1). +- A top-level `env:` in an included Taskfile goes to each task of the mirror, also to the + tasks of the root Taskfile and to the host verbs (verified, 3.53.1). Thus, each command + that sorts or compares in the engine sets `LC_ALL=C` itself. +- go-task ignores a top-level `set:` in an included Taskfile, also for the tasks of that + file (verified, 3.53.1). Thus, each engine verb that needs pipefail has + `set: [pipefail]`. - Use only the keys `trusted-hosts` and `cache-expiry` in `.taskrc.yml`. Task ignores a key that it does not know, and it shows no warning. In that condition, it downloads the remote Taskfiles again each time that it runs. @@ -75,13 +85,13 @@ Read that section before you write. `-v2`. The two tags move. When they move, all mirrors get the release. - Give `-r` to each `xargs` if a filter can make its input empty. GNU `xargs` runs its command one time on empty input. A guard on the file that a pipe reads is not a guard on - the input of `xargs`. For example, the `SLASH` awk of `pages` removes the root, which has + the input of `xargs`. For example, the `SLASH` awk of `pages-draw` removes the root, which has no key without a slash. Thus, if the only changed directory of a run is the root, that awk sends no line to `xargs`. - A `>-` folded block keeps a newline if a line that continues the line before it starts with more spaces than the lines around it. The two-line awk programs in the commands of - `split` and `pages` use this. Put a backslash at the end of a shell line that must - continue. `task check` shows how the block renders. + `split`, `pages` and `pages-draw` use this. Put a backslash at the end of a shell line + that must continue. `task check` shows how the block renders. - On a macOS disk, two directory names that are different only in uppercase and lowercase letters are one directory. Thus, the pages that a laptop writes from a listing of upstream are not complete if upstream has two such directories. For example, CTAN has @@ -95,9 +105,9 @@ Read that section before you write. show this problem. - The `status:` check of `prepare` reads all of `changed.txt`, not the batch. Thus, `prepare` runs on almost all runs. But if the delta of a run has no `TL` path, `prepare` - does not run, and `.run/tl` is not there. Each branch of `verify` reads the batch. The - container branch and the decision-batch branch use `.run/tl` only if the batch has a - `TL` path. + does not run, and `.run/tl` is not there. Each step of `verify` reads the batch. + `verify-containers` and `verify-decision` use `.run/tl` only if the batch has a `TL` + path. - Do not make a task internal if a host verb starts it with its name in the image. go-task does not start an internal task from the command line. For example, `empty-trash` starts `empty-trash-pipeline` with `task op`. Thus, `empty-trash-pipeline` is not internal. @@ -121,9 +131,9 @@ Read that section before you write. of the directories that do not change, and they show the old format. The stamp in `.state/indexed.txt.xz` includes `PAGE_FORMAT`. A new value makes each mirror write all of its pages again, one time. -- `PAGE_HTML`, the page awk of `pages`, writes each page directly to its two files. Do not - make the page with a string that gets a row at a time. That method copies the full string - for each row. The `archive/` directory of TeX Live has approximately 34k rows. +- `PAGE_HTML`, the page awk of `pages-draw`, writes each page directly to its two files. Do + not make the page with a string that gets a row at a time. That method copies the full + string for each row. The `archive/` directory of TeX Live has approximately 34k rows. - In a check that reads the output of a task with `grep`, find text that only the output has. go-task shows each command before it runs it, and that line also has the text of each `echo` in the command. For example, `offline` finds `not 08` in the output of diff --git a/README.md b/README.md index c5495eb..1b676c9 100644 --- a/README.md +++ b/README.md @@ -344,16 +344,16 @@ The verbs with a broken line are hooks. These hooks do no work until a mirror se | `prepare` | A hook. If `TL_KEY` has a value and the delta has a `TL` path: it fetches the tlpdb and compares its signature with the pinned key | | `batches` | Runs `fetch`, `verify`, `publish` and `checkpoint` for each of the first `MAX_BATCHES` batches. If there are remaining batches, it makes the file `.run/chain` | | `fetch` | Copies the files of the batch into `staging/` with rsync, and copies the target of each symlink. If upstream deletes a path after the listing, `fetch` does not copy it | -| `verify` | A hook. If `TL_KEY` has a value: it compares each signed file and each container in the batch with the tlpdb | +| `verify` | A hook. If `TL_KEY` has a value: it compares each signed file and each container in the batch with the tlpdb. It has three steps: `verify-signed`, `verify-containers` and `verify-decision` | | `publish` | Runs `label` on the batch. Then it runs `aws s3 cp --recursive` one time for each type, with the files of each type in a different tree. It sends each file as one PutObject, and it makes no listing of the bucket. The engine does not use `aws s3 sync`. The `FRESH_KEY` file goes last | | `label` | Writes `.run/labels.txt`: the Content-Type of each file in staging, from its name and its first KiB. A text type gets a charset if the file is UTF-8. Refer to [Content types](#content-types) | | `checkpoint` | Runs `merge` to add the uploaded files to the state. Then it uploads the state as one PutObject, and it deletes the files in staging | | `delete` | Removes the keys in `deleted.txt` from the bucket, 1,000 in each call, and removes them from the state. It waits for the run that uploads the last batch | | `reconcile` | If `due` wrote `.run/reconcile`, it runs `rebuild`. Then it deletes each key that is not a path of upstream, a key in `OWN`, or a directory of the page state. Then it runs `reconciled` | | `index` | A hook. If `INDEX` has a value: it runs `pages` and `index-same`, uploads the two sets of keys, runs `index-copy`, which copies the index files of `.run/index-real.txt` to their page keys, and removes the keys of the directories that upstream removed. Then it writes the stamp and the page state to `.state/indexed.txt.xz`. A mirror with a different landing page overrides `index` | -| `pages` | Writes a page for each directory that the run changed, from the page state, into staging. If the stamp of INDEXED does not agree, it writes a page for each directory. The keys with no slash at the end go in one tree for each depth. A directory with a file in `INDEX_FILES` gets only the key with no slash, and a line in `.run/index-real.txt` | +| `pages` | Writes a page for each directory that the run changed, from the page state, into staging. If the stamp of INDEXED does not agree, it writes a page for each directory. The keys with no slash at the end go in one tree for each depth. A directory with a file in `INDEX_FILES` gets only the key with no slash, and a line in `.run/index-real.txt`. Its steps are `pages-select`, which finds the directories, and `pages-draw` | | `index-same` | Operates only if the stamp of INDEXED does not agree. It reads the ETag of each key in the bucket. It removes from staging and from `.run/index-make.txt` each directory whose keys have the same MD5, and writes that directory to `.run/index-same.txt`. If the listing fails, `index` sends each page | -| `smoke` | Reads a sample back from `HOST`. The list below the table gives each item of the sample. Then it runs `fresh` and `smoke-mirror` | +| `smoke` | Reads a sample back from `HOST`. The list below the table gives each item of the sample. Its steps are `smoke-sample`, `smoke-tlpdb`, `smoke-page`, `smoke-canary`, `smoke-encoding` and `smoke-unlanded`, which finds the paths that rsync did not send. Then it runs `fresh` and `smoke-mirror` | | `fresh` | If `FRESH_KEY` has a value: it reads the clock of upstream in that file, from `HOST`. If the clock is more than `FRESH_HOURS` before the time of the run, it stops the run | | `smoke-mirror` | A hook, empty here. A mirror with more items to read back writes it | | `report-engine` | A hook: the rows of the engine in the run summary | @@ -637,9 +637,15 @@ change that var. #### When a run fails -The job log shows the step that stopped the run. The name of that step is the verb. The -table in this section gives each verb of the engine that can stop a run, with the cause and -the repair. The README of a mirror gives only the values of that mirror: its ceiling, its +The job log shows the verb that stopped the run. If the verb has steps, the log also shows +the step. For example: + +```text +task: Failed to run task "pipeline": task: Failed to run task "smoke": task: Failed to run task "smoke-canary": exit status 1 +``` + +The table in this section gives each verb or step of the engine that can stop a run, with +the cause and the repair. The README of a mirror gives only the values of that mirror: its ceiling, its upstream, and the steps that only that mirror has. To find the step, and the number of files that the mirror does not have, use these @@ -672,9 +678,11 @@ the work. On a laptop, | `split`: the ceiling | The tree is larger than `CEILING_GB`. The mirror gets no new files until you increase `CEILING_GB`. A larger ceiling also increases the cost | Increase `CEILING_GB` | | `split`: the disk | One file of the delta is too large for the disk of the runner, with 1 GiB free for the other data. The log shows `no room for `. `CEILING_GB` has no effect on this check | Remove the file from the listing with `FILTER`, or give the run a larger disk | | `prepare`, `verify` | A signature or a checksum did not agree. The engine uploaded no file from that batch, and clients continue to get the last good copy | If TeX Live changed its key, compare `TL_KEY` with https://www.tug.org/texlive/verify.html. Do not use a different source for the key | -| `smoke`: the canary | The domain served bytes that are not the bytes in the bucket, or it rejected a Perl client. A zone rule is missing, or the scope of a zone rule became wider. The bucket is correct | Examine the Configuration Rule of the zone. Refer to [The parts of a bucket](#the-parts-of-a-bucket) | -| `smoke`: the Content-Encoding | A `.tar.gz` had a `Content-Encoding` when `smoke` read it. Clients then decompress it while they download it | Examine `label` and the AWS CLI version in `toolchain.lock.toml`. Refer to [Content types](#content-types) | -| `smoke`: the type or the length | A key of the sample had a Content-Type that is not its label, or a length that is not its length in the listing. The CLI, R2 or the zone changed the object | Examine `label` and the AWS CLI version in `toolchain.lock.toml`. Then examine the rules of the zone | +| `smoke-canary` | The domain served bytes that are not the bytes in the bucket, or it rejected a Perl client. A zone rule is missing, or the scope of a zone rule became wider. The bucket is correct | Examine the Configuration Rule of the zone. Refer to [The parts of a bucket](#the-parts-of-a-bucket) | +| `smoke-encoding` | A `.tar.gz` had a `Content-Encoding` when `smoke` read it. Clients then decompress it while they download it | Examine `label` and the AWS CLI version in `toolchain.lock.toml`. Refer to [Content types](#content-types) | +| `smoke-sample` | A key of the sample had a Content-Type that is not its label, or a length that is not its length in the listing. The CLI, R2 or the zone changed the object | Examine `label` and the AWS CLI version in `toolchain.lock.toml`. Then examine the rules of the zone | +| `smoke-tlpdb` | The domain served a `texlive.tlpdb.sha512` that is not the copy that `prepare` verified | Start the run again. If the problem continues, examine the rules of the zone | +| `smoke-page` | A page that the run wrote did not have the length of the page in staging, at its page key or at its key with no slash. If the page key is a copy of an index file, its length or its Content-Type was not that of the file. R2 or the zone changed the page | Examine the rules of the zone. Refer to [Directory pages](#directory-pages) | | `fresh`: the clock stopped | The clock in the `FRESH_KEY` file did not change for more than `FRESH_HOURS`. Upstream stopped, or the mirror that `SOURCE` gives stopped. A secondary mirror can stop its sync, and rsync can continue to connect to it. The run copied all the bytes that it got | Examine upstream. If `SOURCE` is a secondary mirror, set `SOURCE` to a different secondary mirror. If upstream stopped, no change in the mirror is necessary | These procedures repair other problems: @@ -1337,10 +1345,12 @@ diff, and that diff is the review. - **A task that a verb runs does not get the call vars of that verb.** Also, a root var cannot read a var of an included Taskfile. A verb can run a hook with a `RUN` or a `STAGING` that a caller can change. In that condition, the verb must give these vars to - the hook. `smoke` does this. + the hook. `smoke` does this. `smoke`, `verify` and `pages` also give their steps the vars + that the steps read, with a YAML anchor. - **Do not put `dir:` on a verb of an included Taskfile.** If task reads a flattened Taskfile of `includes:` from a path, it adds each `dir:` to the directory of that - Taskfile, also an absolute `dir:`. Thus, `verify` starts each command with `cd`. + Taskfile, also an absolute `dir:`. Thus, each step of `verify` starts each command with + `cd`. - **`render` puts an empty directory on `.run`.** Thus, a `sh:` var that reads `.run` when task reads the Taskfile gives the same render in each checkout. A verb that reads `.run` at that time must accept an empty `.run`. diff --git a/engines/rsync.yml b/engines/rsync.yml index 8546133..1584035 100644 --- a/engines/rsync.yml +++ b/engines/rsync.yml @@ -632,9 +632,27 @@ tasks: # Not internal. Thus, a mirror can start it manually on a batch from a fixture. requires: {vars: [B]} status: ['test -z "{{.TL_KEY}}"'] # the no-op hook until a mirror pins a key - # Each command starts with a cd to STAGING, not dir:. go-task resolves dir: against - # the directory of an include that it reads from a path, and no call var can change - # that. + # Each command of its steps starts with a cd to STAGING, not dir:. go-task resolves + # dir: against the directory of an include that it reads from a path, and no call var + # can change that. The steps operate without pipefail, because the grep in + # verify-signed can correctly find nothing. + cmds: + # A task does not see the call vars of the task that starts it. Thus, verify gives its + # steps the vars that they read, with the anchor batch. In a run, these are the values + # that verify has. An offline check gives different values. + - task: verify-signed + vars: &batch + B: '{{.B}}' + RUN: '{{.RUN}}' + STAGING: '{{.STAGING}}' + TL: '{{.TL}}' + TL_KEY: '{{.TL_KEY}}' + - {task: verify-containers, vars: *batch} + - {task: verify-decision, vars: *batch} + + verify-signed: + desc: Each signed sha512 of batch B, with shasum, then gpgv against the keyring that prepare pinned + internal: true cmds: # This command examines each signed sha512 in this batch (the installers and # updaters at the root, and the tlpdb) with the keyring that prepare pinned. The @@ -644,12 +662,16 @@ tasks: (cd "$(dirname "$f")" && f=$(basename "$f") && shasum -a 512 -c "$f" && gpgv --status-fd 1 --keyring {{.RUN}}/tl/tlpkg/gpg/pubring.gpg "$f.asc" "$f" | awk -v k={{.TL_KEY}} '{{.GPGCHECK}}') || exit 1; done + + verify-containers: + desc: Each TL container of batch B against its checksum in the tlpdb that prepare verified + internal: true + cmds: # Each TL container in this batch must agree with the checksum that the tlpdb gives # for it (TLPDB_SUMS). prepare examined that tlpdb. # # The awk program writes to a file, not to a pipe. Thus, the status of the command - # is the status of awk. verify operates without pipefail, because the grep in the - # first command of verify can correctly find nothing. + # is the status of awk. # # A batch with no container does not do this check. prepare makes RUN/tl, and # prepare operates only when the delta has a path in TL. Thus, in an hour with no @@ -665,6 +687,11 @@ tasks: cd {{.STAGING}} && { test "$(grep -c '^{{.TL}}/archive/' {{.B}})" = 0 || { awk -F'\t' -v TL="{{.TL}}" '{{.TLPDB_SUMS}}' {{.B}} {{.RUN}}/tl/tlpkg/texlive.tlpdb > {{.RUN}}/tl/sums.txt && shasum -a 512 -c --quiet {{.RUN}}/tl/sums.txt; }; } + + verify-decision: + desc: When batch B holds the tlpdb, the tlpdb against prepare's, its .xz against it, and each container that it names against the bucket after this run + internal: true + cmds: # The decision batch. For this batch, these conditions must be correct: # - The tlpdb in the batch is the tlpdb that prepare examined at the start of the # run. @@ -971,6 +998,27 @@ tasks: PAGE_FOOT={{shellQuote (.PAGE_FOOT | default "")}} awk 'BEGIN { g = ENVIRON["PAGE_FOOT"]; gsub(/%%/, "", g) if (g !~ /^[^%]*(%s[^%]*)?$/) { print "pages: PAGE_FOOT can have one %s, and %%, not " ENVIRON["PAGE_FOOT"] > "/dev/stderr"; exit 1 } }' - 'mkdir -p {{.STAGING}} {{.SLASH}} && find {{.STAGING}} {{.SLASH}} -mindepth 1 -delete && : > {{.RUN}}/index-real.txt' + # A task does not see the call vars of the task that starts it. Thus, pages gives its + # steps the vars that they read, with the anchor pages. In a run, these are the values + # that pages has. An offline check gives different values. + - task: pages-select + vars: &pages + RUN: '{{.RUN}}' + STAGING: '{{.STAGING}}' + SLASH: '{{.SLASH}}' + HOST: '{{.HOST}}' + INDEX: '{{.INDEX}}' + INDEX_FILES: '{{.INDEX_FILES}}' + UPSTREAM: '{{.UPSTREAM}}' + CONTACT: '{{.CONTACT}}' + PAGE_FOOT: '{{.PAGE_FOOT}}' + - {task: pages-draw, vars: *pages} + + pages-select: + desc: The page state, the stamp and RUN/indexed.new; then RUN/index-gone.txt, the keys to remove, and RUN/index-make.txt, the directories to draw + internal: true + set: [pipefail] + cmds: # The page state is the state and a line "dir/ TAB TAB" for each directory of # upstream. Thus, a directory with no file also gets a page, and the page of its # parent shows it. @@ -1013,6 +1061,12 @@ tasks: | awk -v i="{{.INDEX}}" '{ print $0 i } $0 != "" { print substr($0, 1, length($0) - 1) }' | awk -F'\t' 'FILENAME == ARGV[1] { f[$1] = 1; next } !($0 in f)' {{.RUN}}/page-state.txt - > {{.RUN}}/index-gone.txt - LC_ALL=C comm -12 {{.RUN}}/index-dirty.txt {{.RUN}}/index-dirs.txt > {{.RUN}}/index-make.txt + + pages-draw: + desc: A page in STAGING for each directory of RUN/index-make.txt, and its slashless copy in SLASH + internal: true + set: [pipefail] + cmds: - test ! -s {{.RUN}}/index-make.txt || awk -v s="{{.STAGING}}/" '{ print s $0 }' {{.RUN}}/index-make.txt | tr '\n' '\0' | xargs -0 mkdir -p # SLASH// holds the slashless copies of the pages at that depth. All files in # one tree have the same number of components. Thus, no file is the parent directory @@ -1044,13 +1098,37 @@ tasks: smoke: desc: Read a key of every type this run published back through the domain, size against the listing and type against the label; with TL set the tlpdb sha512 against the verified copy; with INDEX set one redrawn page under both keys; with CANARY set the canary, as a Perl client, against the bucket's copy; no Content-Encoding on a .tar.gz; warn of paths that never land; fresh; then the mirror's own checks, smoke-mirror + cmds: + # A task does not see the call vars of the task that starts it. Thus, smoke gives its + # steps the vars that they read, with the anchor smoke. It gives fresh and + # smoke-mirror the three vars that a check reads. In a run, these are the values that + # smoke has. An offline check gives different values. + - task: smoke-sample + vars: &smoke + RUN: '{{.RUN}}' + STAGING: '{{.STAGING}}' + URL: '{{.URL}}' + TL: '{{.TL}}' + INDEX: '{{.INDEX}}' + CANARY: '{{.CANARY}}' + FRESH_KEY: '{{.FRESH_KEY}}' + - {task: smoke-tlpdb, vars: *smoke} + - {task: smoke-page, vars: *smoke} + - {task: smoke-canary, vars: *smoke} + - {task: smoke-encoding, vars: *smoke} + - {task: smoke-unlanded, vars: *smoke} + - {task: fresh, vars: {RUN: '{{.RUN}}', URL: '{{.URL}}'}} + - {task: smoke-mirror, vars: {RUN: '{{.RUN}}', STAGING: '{{.STAGING}}', URL: '{{.URL}}'}} + + smoke-sample: + desc: The first key of each type that label gave this run, and the FRESH_KEY file when the state has it, read back through the domain; size against the listing, and with http the type against the label + internal: true vars: - # The sample in the first command of smoke reads the FRESH_KEY file only when the - # state shows that the file is in the bucket. The file is in the decision batch, - # which is the last batch. A first fill that stops at MAX_BATCHES does not do that - # batch. The bucket then gives 404 for the key, because no run put it in the bucket. - # The run that puts the file in the bucket, and each run after it, include the file - # in the sample. + # The sample reads the FRESH_KEY file only when the state shows that the file is in + # the bucket. The file is in the decision batch, which is the last batch. A first + # fill that stops at MAX_BATCHES does not do that batch. The bucket then gives 404 for + # the key, because no run put it in the bucket. The run that puts the file in the + # bucket, and each run after it, include the file in the sample. # # An {{if}} on a string goes to {{else}} only when the string is empty. Thus, the # path is the flag and also the key. @@ -1077,6 +1155,11 @@ tasks: ct=$(printf '%s\n' "$h" | {{.CTYPE}}); echo "$k: $got bytes (listing: $want), served as ${ct:-no type} (label: ${t:-none})"; test "$got" = "$want" || exit 1; case "{{.URL}}" in http*) test -z "$t" || test "$ct" = "$t" || exit 1 ;; esac; done + + smoke-tlpdb: + desc: With TL set and a tlpdb that prepare verified, the tlpdb sha512 read back through the domain against the verified copy + internal: true + cmds: # The tlpdb key is in the decision batch, the last batch. Thus, a run that stopped # at MAX_BATCHES did not publish the copy of this hour, and it has nothing to # compare. @@ -1084,6 +1167,11 @@ tasks: {{if .TL}}test ! -f {{.RUN}}/tl/tlpkg/texlive.tlpdb.sha512 || test -f {{.RUN}}/chain || {{.CURL}} {{.URL}}/{{.TL}}/tlpkg/texlive.tlpdb.sha512 | cmp - {{.RUN}}/tl/tlpkg/texlive.tlpdb.sha512{{else}}echo "no TL subtree: no tlpdb to read back"{{end}} + + smoke-page: + desc: With INDEX set, the last directory of RUN/index-make.txt read back through the domain at its page key, and with http at its slashless key + internal: true + cmds: # This command reads one of the pages that this run made, at its two keys, with each # key in the URL (no zone rule is necessary). r gives the size and the type of a # key, with the same ranged read of one byte as the sample. @@ -1111,6 +1199,11 @@ tasks: want=$(wc -c < "{{.RUN}}/slash/$(printf '%s\n' "$p" | awk -F/ '{ print NF }')/$p" | tr -d ' '); test -n "$want"; got=$(r "$p"); got=${got%%,*}; echo "$p: $got (listing: $want bytes)"; test "$got" = "$want bytes"; } ;; esac; }{{else}}echo "no INDEX: no directory pages to read back"{{end}} + + smoke-canary: + desc: With CANARY set, in the state and over http, the canary read as a Perl client against the bucket's copy + internal: true + cmds: # The HTML rewriters of Cloudflare are between R2 and the client. Thus, the check # compares the object with the file that the domain sends. A size check is not # sufficient. Automatic HTTPS Rewrites can make a link one byte longer, while the @@ -1131,6 +1224,11 @@ tasks: aws s3 cp {{.AWS_FLAGS}} "{{.S3}}/$h" {{.RUN}}/canary.html && {{.CURL}} -A "libwww-perl/6.67" "{{.URL}}/$h" | cmp - {{.RUN}}/canary.html && echo "$h: $n bytes, served to a libwww-perl client as the bucket holds them"; fi{{else}}echo "no CANARY: the edge's HTML rewriters and client filter go unchecked"{{end}} + + smoke-encoding: + desc: Over http, the first non-empty .tar.gz of the state read back with no Content-Encoding + internal: true + cmds: # A Content-Encoding on an object tells each client to decompress the file during # the download. Thus, the client gets a tarball as a tar. The mirror guidelines of # GNU recommend no Content-Encoding. For foo.tar.gz, guess_type in Python gives @@ -1148,6 +1246,11 @@ tasks: u=$(printf '%s\n' "$g" | {{.ENCODE}} | cut -f2) && h=$({{.HEADERS}} "{{.URL}}/$u") && if printf '%s' "$h" | grep -qi '^content-encoding:'; then echo "$g: served with a content-encoding" >&2; exit 1; fi && echo "$g: served with no content-encoding"; fi + + smoke-unlanded: + desc: RUN/unlanded.txt, the lines of the delta that the state lacks after the last batch; the first 20 as warnings on the run + internal: true + cmds: # A path in the listing of upstream that rsync cannot send (exit 23, which retry # accepts) does not go into the bucket. Thus, each run downloads it again, and the # run shows no failure. When the run did all the batches, the lines of the delta @@ -1157,11 +1260,6 @@ tasks: if test -f {{.RUN}}/changed.txt && test ! -f {{.RUN}}/chain; then LC_ALL=C comm -23 {{.RUN}}/changed.txt {{.RUN}}/applied.txt > {{.RUN}}/unlanded.txt; else : > {{.RUN}}/unlanded.txt; fi; cut -f1 {{.RUN}}/unlanded.txt | head -20 | while IFS= read -r p; do echo "::warning::listed upstream, never fetched: $p"; done - # A task does not see the call vars of the task that starts it. Thus, smoke gives - # the three vars that a check reads. In a run, they are the values that smoke has. - # An offline check gives different values. - - {task: fresh, vars: {RUN: '{{.RUN}}', URL: '{{.URL}}'}} - - {task: smoke-mirror, vars: {RUN: '{{.RUN}}', STAGING: '{{.STAGING}}', URL: '{{.URL}}'}} fresh: desc: With FRESH_KEY set and in the state, read it through the domain and fail the run when the upstream clock in it is FRESH_HOURS (24) old; a source that stalls otherwise leaves every run green diff --git a/examples/rsync/render.txt b/examples/rsync/render.txt index 7085eda..6fa53e5 100644 --- a/examples/rsync/render.txt +++ b/examples/rsync/render.txt @@ -90,8 +90,8 @@ task: [retry] for i in 1 2 3 4 5; do s=$((15 * 2 ** i + RANDOM % (2 * 15 + 1))); echo "rsync exit $rc, retry $i in ${s}s" >&2; sleep $s done; exit $rc -task: [verify] cd /work/examples/rsync/staging && cut -f1 | grep -E '^/(tlpkg/texlive\.tlpdb|[^/]+)\.sha512$' | while IFS= read -r f; do (cd "$(dirname "$f")" && f=$(basename "$f") && shasum -a 512 -c "$f" && gpgv --status-fd 1 --keyring /work/examples/rsync/.run/tl/tlpkg/gpg/pubring.gpg "$f.asc" "$f" | awk -v k= '/^\[GNUPG:\] GOODSIG /{g=1} /^\[GNUPG:\] VALIDSIG / && $NF == k {v=1} END{exit !(g&&v)}') || exit 1; done -task: [verify] cd /work/examples/rsync/staging && { test "$(grep -c '^/archive/' )" = 0 || { awk -F'\t' -v TL="" 'FILENAME == ARGV[1] { if (index($1, TL "/archive/") == 1) want[$1] = 1; next } +task: [verify-signed] cd /work/examples/rsync/staging && cut -f1 | grep -E '^/(tlpkg/texlive\.tlpdb|[^/]+)\.sha512$' | while IFS= read -r f; do (cd "$(dirname "$f")" && f=$(basename "$f") && shasum -a 512 -c "$f" && gpgv --status-fd 1 --keyring /work/examples/rsync/.run/tl/tlpkg/gpg/pubring.gpg "$f.asc" "$f" | awk -v k= '/^\[GNUPG:\] GOODSIG /{g=1} /^\[GNUPG:\] VALIDSIG / && $NF == k {v=1} END{exit !(g&&v)}') || exit 1; done +task: [verify-containers] cd /work/examples/rsync/staging && { test "$(grep -c '^/archive/' )" = 0 || { awk -F'\t' -v TL="" 'FILENAME == ARGV[1] { if (index($1, TL "/archive/") == 1) want[$1] = 1; next } /^name /{split($0, a, " "); n = a[2]} /^revision /{split($0, a, " "); r = a[2]} /^containerchecksum /{split($0, a, " "); emit(a[2], n)} /^doccontainerchecksum /{split($0, a, " "); emit(a[2], n ".doc")} @@ -100,7 +100,7 @@ function emit(sum, base, p, q) { p = TL "/archive/" base ".tar.xz"; q = TL "/a if (p in want) { print sum " " p; delete want[p] } if (q in want) { print sum " " q; delete want[q] } } END { for (p in want) { print "no checksum in the tlpdb for " p > "/dev/stderr"; bad = 1 } exit bad }' /work/examples/rsync/.run/tl/tlpkg/texlive.tlpdb > /work/examples/rsync/.run/tl/sums.txt && shasum -a 512 -c --quiet /work/examples/rsync/.run/tl/sums.txt; }; } -task: [verify] cd /work/examples/rsync/staging && { test ! -f /tlpkg/texlive.tlpdb || { cmp /tlpkg/texlive.tlpdb /work/examples/rsync/.run/tl/tlpkg/texlive.tlpdb && xz -dc /tlpkg/texlive.tlpdb.xz | cmp - /tlpkg/texlive.tlpdb && { cut -f1 /work/examples/rsync/.run/applied.txt; cut -f1 ; } | LC_ALL=C sort -u | LC_ALL=C comm -23 - /work/examples/rsync/.run/deleted.txt > /work/examples/rsync/.run/tl/after.txt && awk -v TL="" '/^name /{n=$2} +task: [verify-decision] cd /work/examples/rsync/staging && { test ! -f /tlpkg/texlive.tlpdb || { cmp /tlpkg/texlive.tlpdb /work/examples/rsync/.run/tl/tlpkg/texlive.tlpdb && xz -dc /tlpkg/texlive.tlpdb.xz | cmp - /tlpkg/texlive.tlpdb && { cut -f1 /work/examples/rsync/.run/applied.txt; cut -f1 ; } | LC_ALL=C sort -u | LC_ALL=C comm -23 - /work/examples/rsync/.run/deleted.txt > /work/examples/rsync/.run/tl/after.txt && awk -v TL="" '/^name /{n=$2} /^containerchecksum /{print TL "/archive/" n ".tar.xz"} /^doccontainerchecksum /{print TL "/archive/" n ".doc.tar.xz"} /^srccontainerchecksum /{print TL "/archive/" n ".source.tar.xz"}' /tlpkg/texlive.tlpdb | LC_ALL=C sort | LC_ALL=C comm -23 - /work/examples/rsync/.run/tl/after.txt > /work/examples/rsync/.run/tl/missing.txt && test ! -s /work/examples/rsync/.run/tl/missing.txt; }; } @@ -275,21 +275,21 @@ esac task: [pages] PAGE_FOOT='' awk 'BEGIN { g = ENVIRON["PAGE_FOOT"]; gsub(/%%/, "", g) if (g !~ /^[^%]*(%s[^%]*)?$/) { print "pages: PAGE_FOOT can have one %s, and %%, not " ENVIRON["PAGE_FOOT"] > "/dev/stderr"; exit 1 } }' task: [pages] mkdir -p /work/examples/rsync/staging /work/examples/rsync/.run/slash && find /work/examples/rsync/staging /work/examples/rsync/.run/slash -mindepth 1 -delete && : > /work/examples/rsync/.run/index-real.txt -task: [pages] { cat /work/examples/rsync/.run/applied.txt; awk '{ print $0 "\t\t" }' /work/examples/rsync/.run/upstream-dirs.txt; } | LC_ALL=C sort > /work/examples/rsync/.run/page-state.txt -task: [pages] printf '%s\n' 3 example.katoptra.org.directory.index.html 'index.html index.htm' '' '' '' example.katoptra.org | sha256sum | awk '{ print "#pages " substr($1, 1, 16) }' > /work/examples/rsync/.run/index-stamp.txt -task: [pages] awk 'NR == 1 && /^#pages / { next } { print }' /work/examples/rsync/.run/indexed.txt > /work/examples/rsync/.run/index-old.txt -task: [pages] { LC_ALL=C comm -3 /work/examples/rsync/.run/index-old.txt /work/examples/rsync/.run/page-state.txt | awk -F'\t' '{ print ($1 == "" ? $2 : $1) }'; head -1 /work/examples/rsync/.run/indexed.txt | cmp -s - /work/examples/rsync/.run/index-stamp.txt || cat /work/examples/rsync/.run/page-state.txt; } | awk -F'\t' '!("" in s) { s[""] = 1; print "" } +task: [pages-select] { cat /work/examples/rsync/.run/applied.txt; awk '{ print $0 "\t\t" }' /work/examples/rsync/.run/upstream-dirs.txt; } | LC_ALL=C sort > /work/examples/rsync/.run/page-state.txt +task: [pages-select] printf '%s\n' 3 example.katoptra.org.directory.index.html 'index.html index.htm' '' '' '' example.katoptra.org | sha256sum | awk '{ print "#pages " substr($1, 1, 16) }' > /work/examples/rsync/.run/index-stamp.txt +task: [pages-select] awk 'NR == 1 && /^#pages / { next } { print }' /work/examples/rsync/.run/indexed.txt > /work/examples/rsync/.run/index-old.txt +task: [pages-select] { LC_ALL=C comm -3 /work/examples/rsync/.run/index-old.txt /work/examples/rsync/.run/page-state.txt | awk -F'\t' '{ print ($1 == "" ? $2 : $1) }'; head -1 /work/examples/rsync/.run/indexed.txt | cmp -s - /work/examples/rsync/.run/index-stamp.txt || cat /work/examples/rsync/.run/page-state.txt; } | awk -F'\t' '!("" in s) { s[""] = 1; print "" } { n = split($1, c, "/"); d = ""; for (i = 1; i < n; i++) { d = d c[i] "/"; if (!(d in s)) { s[d] = 1; print d } } }' | LC_ALL=C sort > /work/examples/rsync/.run/index-dirty.txt -task: [pages] awk -F'\t' '!("" in s) { s[""] = 1; print "" } +task: [pages-select] awk -F'\t' '!("" in s) { s[""] = 1; print "" } { n = split($1, c, "/"); d = ""; for (i = 1; i < n; i++) { d = d c[i] "/"; if (!(d in s)) { s[d] = 1; print d } } }' /work/examples/rsync/.run/page-state.txt | LC_ALL=C sort > /work/examples/rsync/.run/index-dirs.txt -task: [pages] cat /work/examples/rsync/.run/index-stamp.txt /work/examples/rsync/.run/page-state.txt > /work/examples/rsync/.run/indexed.new -task: [pages] LC_ALL=C comm -23 /work/examples/rsync/.run/index-dirty.txt /work/examples/rsync/.run/index-dirs.txt | awk -v i="example.katoptra.org.directory.index.html" '{ print $0 i } $0 != "" { print substr($0, 1, length($0) - 1) }' | awk -F'\t' 'FILENAME == ARGV[1] { f[$1] = 1; next } !($0 in f)' /work/examples/rsync/.run/page-state.txt - > /work/examples/rsync/.run/index-gone.txt -task: [pages] LC_ALL=C comm -12 /work/examples/rsync/.run/index-dirty.txt /work/examples/rsync/.run/index-dirs.txt > /work/examples/rsync/.run/index-make.txt -task: [pages] test ! -s /work/examples/rsync/.run/index-make.txt || awk -v s="/work/examples/rsync/staging/" '{ print s $0 }' /work/examples/rsync/.run/index-make.txt | tr '\n' '\0' | xargs -0 mkdir -p -task: [pages] test ! -s /work/examples/rsync/.run/index-make.txt || awk -v r="/work/examples/rsync/.run/slash/" '$0 != "" { n = split($0, c, "/"); d = "" +task: [pages-select] cat /work/examples/rsync/.run/index-stamp.txt /work/examples/rsync/.run/page-state.txt > /work/examples/rsync/.run/indexed.new +task: [pages-select] LC_ALL=C comm -23 /work/examples/rsync/.run/index-dirty.txt /work/examples/rsync/.run/index-dirs.txt | awk -v i="example.katoptra.org.directory.index.html" '{ print $0 i } $0 != "" { print substr($0, 1, length($0) - 1) }' | awk -F'\t' 'FILENAME == ARGV[1] { f[$1] = 1; next } !($0 in f)' /work/examples/rsync/.run/page-state.txt - > /work/examples/rsync/.run/index-gone.txt +task: [pages-select] LC_ALL=C comm -12 /work/examples/rsync/.run/index-dirty.txt /work/examples/rsync/.run/index-dirs.txt > /work/examples/rsync/.run/index-make.txt +task: [pages-draw] test ! -s /work/examples/rsync/.run/index-make.txt || awk -v s="/work/examples/rsync/staging/" '{ print s $0 }' /work/examples/rsync/.run/index-make.txt | tr '\n' '\0' | xargs -0 mkdir -p +task: [pages-draw] test ! -s /work/examples/rsync/.run/index-make.txt || awk -v r="/work/examples/rsync/.run/slash/" '$0 != "" { n = split($0, c, "/"); d = "" for (i = 1; i < n - 1; i++) d = d c[i] "/"; print r (n - 1) "/" d }' /work/examples/rsync/.run/index-make.txt \ | LC_ALL=C sort -u | tr '\n' '\0' | xargs -0r mkdir -p -task: [pages] test ! -s /work/examples/rsync/.run/index-make.txt || awk -F'\t' -v OFS='\t' 'FILENAME == ARGV[1] { want[$0] = 1; next } +task: [pages-draw] test ! -s /work/examples/rsync/.run/index-make.txt || awk -F'\t' -v OFS='\t' 'FILENAME == ARGV[1] { want[$0] = 1; next } { n = split($1, c, "/"); d = "" for (i = 1; i < n; i++) { if ((d in want) && !((d c[i]) in seen)) { seen[d c[i]] = 1; print d, "d", c[i], "", "" }; d = d c[i] "/" } if (d in want) print d, (c[n] == "" ? "e" : "f"), c[n], $2, $3 }' /work/examples/rsync/.run/index-make.txt /work/examples/rsync/.run/page-state.txt | LC_ALL=C sort -t "$(printf '\t')" -k1,1 -k2,2 -k3,3V | awk -F'\t' -v S="/work/examples/rsync/staging/" -v R="/work/examples/rsync/.run/slash/" -v I="example.katoptra.org.directory.index.html" -v F='' -v N='index.html index.htm' -v X="/work/examples/rsync/.run/index-real.txt" -v U='' -v C='' -v H="example.katoptra.org" -v css=':root{color-scheme:light dark;--bg:#f6f5f2;--fg:#1c1c28;--mut:#6e6e7e;--line:#e2e0da;--hov:#edebe5;--a:#3b45d9}@media (prefers-color-scheme:dark){:root{--bg:#121218;--fg:#f0f0f5;--mut:#8b8b9c;--line:#262630;--hov:#1a1a22;--a:#8b93ff}}body{margin:0 auto;max-width:60rem;padding:2rem 1rem 3rem;background:var(--bg);color:var(--fg);font:15px/1.45 system-ui,-apple-system,sans-serif}h1{font-size:1.35rem;font-weight:650;margin:0;overflow-wrap:anywhere}p{color:var(--mut);margin:.2rem 0 1.25rem;font-size:.9rem}table{width:100%;border-collapse:collapse;font:13.5px/1.4 ui-monospace,SFMono-Regular,Menlo,Consolas,monospace}th{font:600 .7rem/1 system-ui,sans-serif;letter-spacing:.06em;text-transform:uppercase;color:var(--mut);text-align:left;padding:0 .5rem .6rem;white-space:nowrap}td{padding:.4rem .5rem;border-top:1px solid var(--line);vertical-align:baseline}td:first-child{overflow-wrap:anywhere;width:100%}td+td{white-space:nowrap;color:var(--mut);font-variant-numeric:tabular-nums}th:last-child,td:last-child{text-align:right}tr:hover td{background:var(--hov)}a{color:var(--a);text-decoration:none}a:hover{text-decoration:underline}.s a{color:var(--mut)}.k{font-size:.72rem;margin-left:.6rem;padding:.05rem .4rem;border:1px solid var(--line);border-radius:.3rem;color:var(--mut)}.k:hover{color:var(--a);border-color:var(--a);text-decoration:none}footer{margin-top:2rem;padding-top:1rem;border-top:1px solid var(--line);color:var(--mut);font-size:.85rem;overflow-wrap:anywhere}footer a{color:inherit;text-decoration:underline}@media (max-width:36rem){thead{display:none}tr{display:grid;grid-template-columns:1fr auto;border-top:1px solid var(--line)}td{border:0;padding:.1rem .5rem}td:first-child{grid-column:1/-1;padding-top:.45rem}td+td{padding-bottom:.45rem;font-size:.8rem}}' 'BEGIN { for (i = 0; i < 256; i++) ord[sprintf("%c", i)] = i } @@ -352,24 +352,24 @@ task: [remove] for f in /work/examples/rsync/.run/del-*; do awk 'BEGIN { printf printf "%s{\"Key\":\"%s\"}", (NR > 1 ? "," : ""), $0 } END { print "],\"Quiet\":true}" }' "$f" > "$f.json" && aws s3api delete-objects --bucket example --delete "file://$f.json" | tee -a /work/examples/rsync/.run/publish.txt | { test "$(grep -c Errors)" = 0; }; done task: [push] xz -T0 -c /work/examples/rsync/.run/indexed.new > /work/examples/rsync/.run/indexed.new.xz task: [push] aws s3 cp --no-progress --cli-connect-timeout 60 --cli-read-timeout 300 /work/examples/rsync/.run/indexed.new.xz s3://example/.state/indexed.txt.xz -task: [smoke] { awk -F'\t' '!seen[$2]++' /work/examples/rsync/.run/labeled.txt 2>/dev/null; } | while IFS="$(printf '\t')" read -r k t; do u=$(printf '%s\n' "$k" | awk 'BEGIN { for (i = 0; i < 256; i++) ord[sprintf("%c", i)] = i } +task: [smoke-sample] { awk -F'\t' '!seen[$2]++' /work/examples/rsync/.run/labeled.txt 2>/dev/null; } | while IFS="$(printf '\t')" read -r k t; do u=$(printf '%s\n' "$k" | awk 'BEGIN { for (i = 0; i < 256; i++) ord[sprintf("%c", i)] = i } function enc(s, out, j, c) { out = ""; for (j = 1; j <= length(s); j++) { c = substr(s, j, 1) out = out ((c ~ /[A-Za-z0-9\/._~-]/) ? c : sprintf("%%%02X", ord[c])) } return out } { print $0 "\t" enc($0) }' | cut -f2); want=$(awk -F'\t' -v k="$k" '$1 == k { print $2 }' /work/examples/rsync/.run/upstream.txt); h=$(curl -fsS --connect-timeout 15 --max-time 60 --retry 6 --retry-connrefused --retry-max-time 600 -r 0-0 -D - -o /dev/null "https://example.katoptra.org/$u"); got=$(printf '%s\n' "$h" | awk 'tolower($1) == "content-range:" { split($3, a, "/"); r = a[2] } tolower($1) == "content-length:" { c = $2 } $2 == "416" { z = 1 } END { sub(/\r/, "", r); sub(/\r/, "", c); print (r != "" ? r : (z ? 0 : c)) }'); ct=$(printf '%s\n' "$h" | awk 'tolower($1) == "content-type:" { sub(/\r$/, ""); sub(/^[^:]*:[ \t]*/, ""); print }'); echo "$k: $got bytes (listing: $want), served as ${ct:-no type} (label: ${t:-none})"; test "$got" = "$want" || exit 1; case "https://example.katoptra.org" in http*) test -z "$t" || test "$ct" = "$t" || exit 1 ;; esac; done -task: [smoke] echo "no TL subtree: no tlpdb to read back" -task: [smoke] test ! -s /work/examples/rsync/.run/index-make.txt || { d=$(tail -1 /work/examples/rsync/.run/index-make.txt); r() { h=$(curl -fsS --connect-timeout 15 --max-time 60 --retry 6 --retry-connrefused --retry-max-time 600 -r 0-0 -D - -o /dev/null "https://example.katoptra.org/$(printf '%s\n' "$1" | awk 'BEGIN { for (i = 0; i < 256; i++) ord[sprintf("%c", i)] = i } +task: [smoke-tlpdb] echo "no TL subtree: no tlpdb to read back" +task: [smoke-page] test ! -s /work/examples/rsync/.run/index-make.txt || { d=$(tail -1 /work/examples/rsync/.run/index-make.txt); r() { h=$(curl -fsS --connect-timeout 15 --max-time 60 --retry 6 --retry-connrefused --retry-max-time 600 -r 0-0 -D - -o /dev/null "https://example.katoptra.org/$(printf '%s\n' "$1" | awk 'BEGIN { for (i = 0; i < 256; i++) ord[sprintf("%c", i)] = i } function enc(s, out, j, c) { out = ""; for (j = 1; j <= length(s); j++) { c = substr(s, j, 1) out = out ((c ~ /[A-Za-z0-9\/._~-]/) ? c : sprintf("%%%02X", ord[c])) } return out } { print $0 "\t" enc($0) }' | cut -f2)"); echo "$(printf '%s\n' "$h" | awk 'tolower($1) == "content-range:" { split($3, a, "/"); r = a[2] } tolower($1) == "content-length:" { c = $2 } $2 == "416" { z = 1 } END { sub(/\r/, "", r); sub(/\r/, "", c); print (r != "" ? r : (z ? 0 : c)) }') bytes, type $(printf '%s\n' "$h" | awk 'tolower($1) == "content-type:" { sub(/\r$/, ""); sub(/^[^:]*:[ \t]*/, ""); print }')"; }; s=$(awk -F'\t' -v k="${d}example.katoptra.org.directory.index.html" '$2 == k { print $1 }' /work/examples/rsync/.run/index-real.txt); if test -n "$s"; then want=$(r "$s"); else want=$(wc -c < "/work/examples/rsync/staging/${d}example.katoptra.org.directory.index.html" | tr -d ' '); test -n "$want"; want="$want bytes"; fi; got=$(r "${d}example.katoptra.org.directory.index.html"); test -n "$s" || got=${got%%,*}; echo "${d}example.katoptra.org.directory.index.html: $got (${s:-page}: $want)"; test "$got" = "$want"; case "https://example.katoptra.org" in http*) test -z "$d" || { p=${d%/}; want=$(wc -c < "/work/examples/rsync/.run/slash/$(printf '%s\n' "$p" | awk -F/ '{ print NF }')/$p" | tr -d ' '); test -n "$want"; got=$(r "$p"); got=${got%%,*}; echo "$p: $got (listing: $want bytes)"; test "$got" = "$want bytes"; } ;; esac; } -task: [smoke] h=pub/index.html; n=$(awk -F'\t' -v k="$h" '$1 == k { print $2 }' /work/examples/rsync/.run/applied.txt 2>/dev/null); case "https://example.katoptra.org" in http*) ;; *) n= ;; esac; if test -z "$n"; then echo "canary skipped ($h is not in the state, or the URL is not http)"; else aws s3 cp --no-progress --cli-connect-timeout 60 --cli-read-timeout 300 "s3://example/$h" /work/examples/rsync/.run/canary.html && curl -fsS --connect-timeout 15 --max-time 60 --retry 6 --retry-connrefused --retry-max-time 600 -A "libwww-perl/6.67" "https://example.katoptra.org/$h" | cmp - /work/examples/rsync/.run/canary.html && echo "$h: $n bytes, served to a libwww-perl client as the bucket holds them"; fi -task: [smoke] g=$(awk -F'\t' '$1 ~ /\.tar\.gz$/ && $2 > 0 { print $1; exit }' /work/examples/rsync/.run/applied.txt 2>/dev/null); case "https://example.katoptra.org" in http*) ;; *) g= ;; esac; if test -z "$g"; then echo "content-encoding check skipped (no non-empty .tar.gz in the state, or the URL is not http)"; else u=$(printf '%s\n' "$g" | awk 'BEGIN { for (i = 0; i < 256; i++) ord[sprintf("%c", i)] = i } +task: [smoke-canary] h=pub/index.html; n=$(awk -F'\t' -v k="$h" '$1 == k { print $2 }' /work/examples/rsync/.run/applied.txt 2>/dev/null); case "https://example.katoptra.org" in http*) ;; *) n= ;; esac; if test -z "$n"; then echo "canary skipped ($h is not in the state, or the URL is not http)"; else aws s3 cp --no-progress --cli-connect-timeout 60 --cli-read-timeout 300 "s3://example/$h" /work/examples/rsync/.run/canary.html && curl -fsS --connect-timeout 15 --max-time 60 --retry 6 --retry-connrefused --retry-max-time 600 -A "libwww-perl/6.67" "https://example.katoptra.org/$h" | cmp - /work/examples/rsync/.run/canary.html && echo "$h: $n bytes, served to a libwww-perl client as the bucket holds them"; fi +task: [smoke-encoding] g=$(awk -F'\t' '$1 ~ /\.tar\.gz$/ && $2 > 0 { print $1; exit }' /work/examples/rsync/.run/applied.txt 2>/dev/null); case "https://example.katoptra.org" in http*) ;; *) g= ;; esac; if test -z "$g"; then echo "content-encoding check skipped (no non-empty .tar.gz in the state, or the URL is not http)"; else u=$(printf '%s\n' "$g" | awk 'BEGIN { for (i = 0; i < 256; i++) ord[sprintf("%c", i)] = i } function enc(s, out, j, c) { out = ""; for (j = 1; j <= length(s); j++) { c = substr(s, j, 1) out = out ((c ~ /[A-Za-z0-9\/._~-]/) ? c : sprintf("%%%02X", ord[c])) } return out } { print $0 "\t" enc($0) }' | cut -f2) && h=$(curl -fsS --connect-timeout 15 --max-time 60 --retry 6 --retry-connrefused --retry-max-time 600 -r 0-0 -D - -o /dev/null "https://example.katoptra.org/$u") && if printf '%s' "$h" | grep -qi '^content-encoding:'; then echo "$g: served with a content-encoding" >&2; exit 1; fi && echo "$g: served with no content-encoding"; fi -task: [smoke] if test -f /work/examples/rsync/.run/changed.txt && test ! -f /work/examples/rsync/.run/chain; then LC_ALL=C comm -23 /work/examples/rsync/.run/changed.txt /work/examples/rsync/.run/applied.txt > /work/examples/rsync/.run/unlanded.txt; else : > /work/examples/rsync/.run/unlanded.txt; fi; cut -f1 /work/examples/rsync/.run/unlanded.txt | head -20 | while IFS= read -r p; do echo "::warning::listed upstream, never fetched: $p"; done +task: [smoke-unlanded] if test -f /work/examples/rsync/.run/changed.txt && test ! -f /work/examples/rsync/.run/chain; then LC_ALL=C comm -23 /work/examples/rsync/.run/changed.txt /work/examples/rsync/.run/applied.txt > /work/examples/rsync/.run/unlanded.txt; else : > /work/examples/rsync/.run/unlanded.txt; fi; cut -f1 /work/examples/rsync/.run/unlanded.txt | head -20 | while IFS= read -r p; do echo "::warning::listed upstream, never fetched: $p"; done task: [fresh] k=timestamp; if ! awk -F'\t' -v k="$k" '$1 == k { f = 1 } END { exit !f }' /work/examples/rsync/.run/applied.txt 2>/dev/null; then echo "freshness skipped ($k is not in the state)"; else h=24; case "$h" in ''|0?*|*[!0-9]*) echo "fresh: FRESH_HOURS is a whole number, not $h" >&2; exit 1 ;; esac; u=$(printf '%s\n' "$k" | awk 'BEGIN { for (i = 0; i < 256; i++) ord[sprintf("%c", i)] = i } function enc(s, out, j, c) { out = ""; for (j = 1; j <= length(s); j++) { c = substr(s, j, 1) out = out ((c ~ /[A-Za-z0-9\/._~-]/) ? c : sprintf("%%%02X", ord[c])) } return out } { print $0 "\t" enc($0) }' | cut -f2); t=$(curl -fsS --connect-timeout 15 --max-time 60 --retry 6 --retry-connrefused --retry-max-time 600 "https://example.katoptra.org/$u" | awk '!/^#/ { for (i = 1; i <= NF; i++) { diff --git a/toolbox.yml b/toolbox.yml index 8da81ee..7787eb7 100644 --- a/toolbox.yml +++ b/toolbox.yml @@ -16,8 +16,13 @@ # engine include adds are also not available: # - The rsync engine: pipeline, plan-pipeline, list, normalise, state, pull, push, # rebuild, diff and split. Also batches, batch, fetch, publish, label, -# label-trees, merge, checkpoint, remove, delete, reconcile, retry, pages, smoke -# and fresh. Its hooks are prepare, verify, index, smoke-mirror and report-engine. +# label-trees, merge, checkpoint, remove, delete, reconcile, retry, pages, +# index-same, index-copy, smoke and fresh. Its hooks are prepare, verify, index, +# smoke-mirror and report-engine. The steps of three verbs add these names: +# - verify: verify-signed, verify-containers and verify-decision +# - pages: pages-select and pages-draw +# - smoke: smoke-sample, smoke-tlpdb, smoke-page, smoke-canary, smoke-encoding +# and smoke-unlanded. # - The proton engine: pipeline, plan-pipeline, session, session-push, seal, # session-seal, age, pull, push, pd, destination, upload, confirm, list-folder, # trash, empty-trash and empty-trash-pipeline. Its hooks are stage, prune and From a88a2d24dcf517a902b1dcf62bb8c2d3c36f0e0c Mon Sep 17 00:00:00 2001 From: Josh Vaughen Date: Thu, 22 Oct 2026 14:38:58 -0700 Subject: [PATCH 4/6] refactor(rsync): give the text values of a mirror to commands in env FRESH_KEY, CANARY, PAGE_FOOT, UPSTREAM, CONTACT and INDEX_FILES go to the commands that read them in env:, not in templates. The shell reads "$X", an awk -v reads "$X", and the label-trees Python (now TREES_PY) gets FRESH_KEY as an argument. The smoke-sample sh: var TS becomes part of its command. This is the upgrade path of the FRESH_KEY ponytail: a quotation mark or a non-ASCII character in the key no longer breaks the awk string or the Python bytes literal. A narrower ponytail stays for the backslash escapes of awk -v. The render shows "$FRESH_KEY" and the other names, not their values. No behavior change for a mirror. For every value that one of the rsync mirrors sets, each command gets the same bytes. Proof: - offline passes, also the page stamp in indexed.new and the pages of run-pages, byte for byte. - Parity against v2.9.1: same tree and exit code in 115 of 115 scenarios, and in four new end-to-end scenarios with the HOST, BUCKET, INDEX, INDEX_FILES, UPSTREAM and FRESH_KEY of ctan, gnu, nongnu and gnu-alpha (tree, aws calls and output identical). A var of the environment with the same name wins over a task env: in go-task. The container gets only the PASS_ENV names, and none of them is one of these six. CLAUDE.md gives the hazard. --- CLAUDE.md | 9 +++ README.md | 9 ++- engines/rsync.yml | 136 +++++++++++++++++++++----------------- examples/rsync/render.txt | 26 ++++---- 4 files changed, 106 insertions(+), 74 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 941d2cb..7d1ca8a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -39,6 +39,15 @@ Read that section before you write. - A top-level `env:` in an included Taskfile goes to each task of the mirror, also to the tasks of the root Taskfile and to the host verbs (verified, 3.53.1). Thus, each command that sorts or compares in the engine sets `LC_ALL=C` itself. +- A task `env:` gets the call vars, and the command gets each value with no change + (verified, 3.53.1). Thus, the text values of a mirror go to the commands in `env:`, and + a quotation mark or a `$` in a value does not change a command. The render shows + `"$FRESH_KEY"`, not the value. `awk -v` changes each backslash escape in a value, and + `ENVIRON` does not. +- A var of the environment wins over a task `env:` with the same name, also an empty var + (verified, 3.53.1). Then `{{.FRESH_KEY}}` and `"$FRESH_KEY"` can give two different + values in one command. The container gets only the names in `PASS_ENV`, and none of + them is a var of the engine. Do not add a var of the engine to `PASS` or to `op.env`. - go-task ignores a top-level `set:` in an included Taskfile, also for the tasks of that file (verified, 3.53.1). Thus, each engine verb that needs pipefail has `set: [pipefail]`. diff --git a/README.md b/README.md index 1b676c9..9491a97 100644 --- a/README.md +++ b/README.md @@ -1336,7 +1336,9 @@ updates: Then run `task render-update` one time. Commit `render.txt`. After that, use `task check`. If a pull request changes the commands of the mirror, `render.txt` shows a -diff, and that diff is the review. +diff, and that diff is the review. The engine gives some text values of the mirror to a +command in `env:`, for example `FRESH_KEY`. The render shows these values as names +(`"$FRESH_KEY"`), and the root vars of the Taskfile give the values. ### Rules a mirror keeps @@ -1400,6 +1402,11 @@ these rules: reads it. If two commands of a verb read it, put the default in a task var with a different name, for example `BATCH_SIZE` in `split`. Write the default one time. Add a row to the table in [The vars](#the-vars). +- Give a text value of a mirror, for example `FRESH_KEY`, to a command in `env:`, not in a + template. The shell reads `"$X"`, awk gets the value with `-v` or `ENVIRON`, and Python + gets it as an argument. The render then shows the name of the value, not the value. A + var of the environment with the same name wins over `env:`. Thus, do not pass such a + name into the container (`PASS`). - Put each awk or Python program of more than two lines in the `programs` part of `vars:`. Write a comment above it that gives its function and its inputs. The command that runs the program gives it each input with awk `-v`, `ENVIRON` or an argument. A program has diff --git a/engines/rsync.yml b/engines/rsync.yml index 1584035..569b064 100644 --- a/engines/rsync.yml +++ b/engines/rsync.yml @@ -33,6 +33,13 @@ # MAX_BATCHES. # - A var with a different name that reads the tunable continues to see the command # line. Thus, each default is in one location only, in the vars and verbs that follow. +# - A text value of the mirror (FRESH_KEY, CANARY, PAGE_FOOT, UPSTREAM, CONTACT and +# INDEX_FILES) goes to a command in `env:`, not in a template. The shell reads "$X", +# awk gets the value with -v or ENVIRON, and Python gets it as an argument. Thus, no +# value goes into the text of a program, and a quotation mark in a value cannot +# change a command. A var of the environment with the same name wins over `env:`. +# The container gets only the names in PASS_ENV of the toolbox. Thus, do not pass +# one of these names into the container. # The other vars get their values from the vars of the mirror, or they are helper vars of # the engine. version: '3' @@ -275,6 +282,39 @@ vars: if t.startswith("text/") and utf8(p): t += "; charset=utf-8" out.write(os.fsencode(os.path.relpath(p, root)) + b"\t" + t.encode() + b"\n") + # label-trees: with "up", move each file of the labels out of staging into one tree for + # each type, and write the type of each tree to N.type. With "staging", move the files + # back. The arguments are staging, the up directory, the labels, the direction and the + # name of the FRESH_KEY file, which stays in staging. + TREES_PY: |- + import os, shutil, sys + + staging, up, labels, to, fresh = (os.fsencode(a) for a in sys.argv[1:6]) + + def move(a, b): + os.makedirs(os.path.dirname(b), exist_ok=True) + os.rename(a, b) + + if to == b"up": + shutil.rmtree(up, ignore_errors=True) + trees = {} + with open(labels, "rb") as f: + for line in f: + path, t = line.rstrip(b"\n").split(b"\t") + if path != fresh: + n = trees.setdefault(t, str(len(trees) + 1).encode()) + move(os.path.join(staging, path), os.path.join(up, n, path)) + for t, n in trees.items(): + with open(os.path.join(up, n + b".type"), "wb") as f: + f.write(t) + else: + for n in os.listdir(up) if os.path.isdir(up) else (): + tree = os.path.join(up, n) + for d, _, files in os.walk(tree) if os.path.isdir(tree) else (): + for f in files: + p = os.path.join(d, f) + move(p, os.path.join(staging, os.path.relpath(p, tree))) + shutil.rmtree(up, ignore_errors=True) # remove: the JSON body of one DeleteObjects call, for the keys of a file, one key on each # line. The program escapes each backslash and each double quotation mark of a key. DELETE_JSON: |- @@ -712,6 +752,7 @@ tasks: internal: true requires: {vars: [B]} set: [pipefail] # tee must not change the exit of an upload that has a failure + env: {FRESH_KEY: '{{.FRESH_KEY}}'} cmds: - {task: label} - cat {{.RUN}}/labels.txt >> {{.RUN}}/labeled.txt @@ -729,15 +770,13 @@ tasks: # The FRESH_KEY file goes to the bucket last, in one more command. It gives the date # of the tree, and monitors (for example, mirmon) read it. # - # ponytail: this command and label-trees put FRESH_KEY, with no quotes, into an awk - # string and a Python string. Ceiling: a key with a quote, a backslash or a - # non-ASCII character makes these strings incorrect. Upgrade path: give it to awk - # with -v, and to Python as an argument. Then do `task render-update` in lib and in - # each rsync mirror. + # ponytail: awk -v changes each backslash escape in FRESH_KEY, here, in smoke-sample + # and in fresh. Ceiling: a key with a backslash. Upgrade path: read the key with + # ENVIRON in these three commands. - >- - {{if .FRESH_KEY}}test ! -f {{.STAGING}}/{{shellQuote .FRESH_KEY}} || - aws s3 cp {{.AWS_FLAGS}} --content-type "$(awk -F'\t' '$1 == "{{.FRESH_KEY}}" { print $2 }' {{.RUN}}/labels.txt)" - {{.STAGING}}/{{shellQuote .FRESH_KEY}} {{.S3}}/{{shellQuote .FRESH_KEY}} | tee -a {{.RUN}}/publish.txt{{end}} + {{if .FRESH_KEY}}test ! -f {{.STAGING}}/"$FRESH_KEY" || + aws s3 cp {{.AWS_FLAGS}} --content-type "$(awk -F'\t' -v k="$FRESH_KEY" '$1 == k { print $2 }' {{.RUN}}/labels.txt)" + {{.STAGING}}/"$FRESH_KEY" {{.S3}}/"$FRESH_KEY" | tee -a {{.RUN}}/publish.txt{{end}} label: desc: RUN/labels.txt, "path TAB type" for every file under DIR (STAGING), byte-sorted; the type is decided from the name and the first KiB, never left to the CLI @@ -759,6 +798,7 @@ tasks: desc: 'TO=up: rename every file of RUN/labels.txt but the FRESH_KEY file out of STAGING into RUN/up/N/, one tree per type, its type in RUN/up/N.type. TO=staging: rename them back' internal: true requires: {vars: [TO]} + env: {FRESH_KEY: '{{.FRESH_KEY}}'} cmds: # Renames on one filesystem. Thus, the verb makes no copy of a byte, and staging has # all its files again for merge. If a run stops between TO=up and TO=staging, files @@ -768,35 +808,8 @@ tasks: # The FRESH_KEY file stays in staging, and publish sends it last. With no FRESH_KEY, # the empty name agrees with no file. Thus, all files move. - | - python3 -I - {{.STAGING}} {{.RUN}}/up {{.RUN}}/labels.txt {{.TO}} <<'PY' - import os, shutil, sys - - staging, up, labels, to = (os.fsencode(a) for a in sys.argv[1:5]) - - def move(a, b): - os.makedirs(os.path.dirname(b), exist_ok=True) - os.rename(a, b) - - if to == b"up": - shutil.rmtree(up, ignore_errors=True) - trees = {} - with open(labels, "rb") as f: - for line in f: - path, t = line.rstrip(b"\n").split(b"\t") - if path != b"{{.FRESH_KEY}}": - n = trees.setdefault(t, str(len(trees) + 1).encode()) - move(os.path.join(staging, path), os.path.join(up, n, path)) - for t, n in trees.items(): - with open(os.path.join(up, n + b".type"), "wb") as f: - f.write(t) - else: - for n in os.listdir(up) if os.path.isdir(up) else (): - tree = os.path.join(up, n) - for d, _, files in os.walk(tree) if os.path.isdir(tree) else (): - for f in files: - p = os.path.join(d, f) - move(p, os.path.join(staging, os.path.relpath(p, tree))) - shutil.rmtree(up, ignore_errors=True) + python3 -I - {{.STAGING}} {{.RUN}}/up {{.RUN}}/labels.txt {{.TO}} "$FRESH_KEY" <<'PY' + {{.TREES_PY}} PY merge: @@ -977,6 +990,11 @@ tasks: pages: desc: RUN/indexed.txt against the page state (RUN/applied.txt and RUN/upstream-dirs.txt) -> a page under both of its keys for every directory a changed line touches, or for every directory when the stamp differs; RUN/index-gone.txt for the keys of directories that upstream removed; RUN/index-real.txt for the directories with an index file; RUN/indexed.new for INDEXED set: [pipefail] + env: &pagetext + INDEX_FILES: '{{.INDEX_FILES}}' + UPSTREAM: '{{.UPSTREAM}}' + CONTACT: '{{.CONTACT}}' + PAGE_FOOT: '{{.PAGE_FOOT}}' vars: # The verb writes each page to two keys: # - /INDEX. For a directory URL, the rewrite of the zone sends this key. @@ -995,7 +1013,7 @@ tasks: # command stops the run on such a PAGE_FOOT. awk reads the value from the # environment, because awk -v changes each backslash escape. - >- - PAGE_FOOT={{shellQuote (.PAGE_FOOT | default "")}} awk 'BEGIN { g = ENVIRON["PAGE_FOOT"]; gsub(/%%/, "", g) + awk 'BEGIN { g = ENVIRON["PAGE_FOOT"]; gsub(/%%/, "", g) if (g !~ /^[^%]*(%s[^%]*)?$/) { print "pages: PAGE_FOOT can have one %s, and %%, not " ENVIRON["PAGE_FOOT"] > "/dev/stderr"; exit 1 } }' - 'mkdir -p {{.STAGING}} {{.SLASH}} && find {{.STAGING}} {{.SLASH}} -mindepth 1 -delete && : > {{.RUN}}/index-real.txt' # A task does not see the call vars of the task that starts it. Thus, pages gives its @@ -1018,6 +1036,7 @@ tasks: desc: The page state, the stamp and RUN/indexed.new; then RUN/index-gone.txt, the keys to remove, and RUN/index-make.txt, the directories to draw internal: true set: [pipefail] + env: *pagetext cmds: # The page state is the state and a line "dir/ TAB TAB" for each directory of # upstream. Thus, a directory with no file also gets a page, and the page of its @@ -1029,8 +1048,7 @@ tasks: # a page. INDEXED starts with the stamp of the pages that it records. If INDEXED has # no stamp line, the stamps do not agree. - >- - printf '%s\n' {{shellQuote .PAGE_FORMAT}} {{shellQuote .INDEX}} {{shellQuote (.INDEX_FILES | default "")}} - {{shellQuote (.PAGE_FOOT | default "")}} {{shellQuote (.UPSTREAM | default "")}} {{shellQuote (.CONTACT | default "")}} + printf '%s\n' {{shellQuote .PAGE_FORMAT}} {{shellQuote .INDEX}} "$INDEX_FILES" "$PAGE_FOOT" "$UPSTREAM" "$CONTACT" {{shellQuote .HOST}} | sha256sum | awk '{ print "#pages " substr($1, 1, 16) }' > {{.RUN}}/index-stamp.txt - awk 'NR == 1 && /^#pages / { next } { print }' {{.RUN}}/indexed.txt > {{.RUN}}/index-old.txt # A page changes when a file in its directory, or a directory below it, is added, @@ -1066,6 +1084,7 @@ tasks: desc: A page in STAGING for each directory of RUN/index-make.txt, and its slashless copy in SLASH internal: true set: [pipefail] + env: *pagetext cmds: - test ! -s {{.RUN}}/index-make.txt || awk -v s="{{.STAGING}}/" '{ print s $0 }' {{.RUN}}/index-make.txt | tr '\n' '\0' | xargs -0 mkdir -p # SLASH// holds the slashless copies of the pages at that depth. All files in @@ -1092,8 +1111,7 @@ tasks: test ! -s {{.RUN}}/index-make.txt || awk -F'\t' -v OFS='\t' '{{.PAGE_ENTRIES}}' {{.RUN}}/index-make.txt {{.RUN}}/page-state.txt | LC_ALL=C sort -t "$(printf '\t')" -k1,1 -k2,2 -k3,3V | awk -F'\t' -v S="{{.STAGING}}/" -v R="{{.SLASH}}/" -v I="{{.INDEX}}" - -v F={{shellQuote (.PAGE_FOOT | default "")}} -v N={{shellQuote (.INDEX_FILES | default "")}} -v X="{{.RUN}}/index-real.txt" - -v U={{shellQuote (.UPSTREAM | default "")}} -v C={{shellQuote (.CONTACT | default "")}} -v H="{{.HOST}}" + -v F="$PAGE_FOOT" -v N="$INDEX_FILES" -v X="{{.RUN}}/index-real.txt" -v U="$UPSTREAM" -v C="$CONTACT" -v H="{{.HOST}}" -v css={{shellQuote .PAGE_CSS}} '{{.PAGE_HTML}}' smoke: @@ -1123,18 +1141,7 @@ tasks: smoke-sample: desc: The first key of each type that label gave this run, and the FRESH_KEY file when the state has it, read back through the domain; size against the listing, and with http the type against the label internal: true - vars: - # The sample reads the FRESH_KEY file only when the state shows that the file is in - # the bucket. The file is in the decision batch, which is the last batch. A first - # fill that stops at MAX_BATCHES does not do that batch. The bucket then gives 404 for - # the key, because no run put it in the bucket. The run that puts the file in the - # bucket, and each run after it, include the file in the sample. - # - # An {{if}} on a string goes to {{else}} only when the string is empty. Thus, the - # path is the flag and also the key. - TS: - sh: |- - {{if .FRESH_KEY}}awk -F'\t' -v k={{shellQuote .FRESH_KEY}} '$1 == k { print $1 }' {{.RUN}}/applied.txt 2>/dev/null || true{{end}} + env: {FRESH_KEY: '{{.FRESH_KEY}}'} cmds: # The sample is the first key of each type that label gave in this run. The command # reads each key again with its encoded URL, and finds its line in the listing with @@ -1142,10 +1149,16 @@ tasks: # Content-Type with the label. This finds the CLI, R2 or the zone when one of them # sends a type that is different from the selected type. # - # The FRESH_KEY file is in the sample for its size only. A key in the sample can - # contain a space. All other keys can also contain a space. + # The FRESH_KEY file is in the sample for its size only. The sample reads it only when + # the state shows that the file is in the bucket. The file is in the decision batch, + # which is the last batch. A first fill that stops at MAX_BATCHES does not do that + # batch. The bucket then gives 404 for the key, because no run put it in the bucket. + # The run that puts the file in the bucket, and each run after it, include the file + # in the sample. + # + # A key in the sample can contain a space. All other keys can also contain a space. - >- - { {{if .TS}}printf '%s\t\n' {{shellQuote .TS}};{{end}} + { {{if .FRESH_KEY}}awk -F'\t' -v k="$FRESH_KEY" '$1 == k { print $1 "\t" }' {{.RUN}}/applied.txt 2>/dev/null || true;{{end}} awk -F'\t' '!seen[$2]++' {{.RUN}}/labeled.txt 2>/dev/null; } | while IFS="$(printf '\t')" read -r k t; do u=$(printf '%s\n' "$k" | {{.ENCODE}} | cut -f2); @@ -1203,6 +1216,7 @@ tasks: smoke-canary: desc: With CANARY set, in the state and over http, the canary read as a Perl client against the bucket's copy internal: true + env: {CANARY: '{{.CANARY}}'} cmds: # The HTML rewriters of Cloudflare are between R2 and the client. Thus, the check # compares the object with the file that the domain sends. A size check is not @@ -1218,7 +1232,7 @@ tasks: # The command does not do this check with file://, which has no bucket to read. It # also does not do it before the state holds the path. - >- - {{if .CANARY}}h={{shellQuote .CANARY}}; n=$(awk -F'\t' -v k="$h" '$1 == k { print $2 }' {{.RUN}}/applied.txt 2>/dev/null); + {{if .CANARY}}h="$CANARY"; n=$(awk -F'\t' -v k="$h" '$1 == k { print $2 }' {{.RUN}}/applied.txt 2>/dev/null); case "{{.URL}}" in http*) ;; *) n= ;; esac; if test -z "$n"; then echo "canary skipped ($h is not in the state, or the URL is not http)"; else aws s3 cp {{.AWS_FLAGS}} "{{.S3}}/$h" {{.RUN}}/canary.html && @@ -1263,12 +1277,13 @@ tasks: fresh: desc: With FRESH_KEY set and in the state, read it through the domain and fail the run when the upstream clock in it is FRESH_HOURS (24) old; a source that stalls otherwise leaves every run green + env: {FRESH_KEY: '{{.FRESH_KEY}}'} cmds: # The run moved each byte that it got. Thus, only the clock that upstream wrote can # show that the bytes are expired. Upstream can stop. A secondary mirror can stop its # sync while its rsync daemon continues to operate. - >- - {{if .FRESH_KEY}}k={{shellQuote .FRESH_KEY}}; + {{if .FRESH_KEY}}k="$FRESH_KEY"; if ! awk -F'\t' -v k="$k" '$1 == k { f = 1 } END { exit !f }' {{.RUN}}/applied.txt 2>/dev/null; then echo "freshness skipped ($k is not in the state)"; else h={{shellQuote .FRESH_H}}; case "$h" in {{.NOT_WHOLE}}) echo "fresh: FRESH_HOURS is a whole number, not $h" >&2; exit 1 ;; esac; @@ -1286,6 +1301,7 @@ tasks: report-engine: desc: The engine's rows of the run summary, appended after the toolbox's silent: true + env: {FRESH_KEY: '{{.FRESH_KEY}}'} cmds: - | n() { test -f "$1" && wc -l < "$1" || echo 0; } @@ -1304,7 +1320,7 @@ tasks: test -z "{{.INDEX}}" || echo "| Directory pages | $(n {{.RUN}}/index-make.txt) redrawn and sent, $(n {{.RUN}}/index-same.txt) redrawn and not sent (the bucket had the same bytes), $(n {{.RUN}}/index-real.txt) with a copy of an index file as the page key, $(n {{.RUN}}/index-gone.txt) keys removed |" >> "$out" # Rows that show only when they have data. test ! -s {{.RUN}}/unlanded.txt || echo "| Never landed | $(wc -l < {{.RUN}}/unlanded.txt | tr -d ' ') listed upstream that rsync could not send, fetched again every run; the first are warnings on the run |" >> "$out" - test ! -f {{.RUN}}/fresh.txt || echo "| Upstream clock | $(awk '{ printf "%d h %d min", $1 / 3600, $1 % 3600 / 60 }' {{.RUN}}/fresh.txt) old in {{.FRESH_KEY}}, limit {{.FRESH_H}} h |" >> "$out" + test ! -f {{.RUN}}/fresh.txt || echo "| Upstream clock | $(awk '{ printf "%d h %d min", $1 / 3600, $1 % 3600 / 60 }' {{.RUN}}/fresh.txt) old in $FRESH_KEY, limit {{.FRESH_H}} h |" >> "$out" retry: desc: Run CMD, retrying rsync's transport exit codes (5 10 12 30 35) five times with backoff and jitter; 23 (a path it could not read, a dangling symlink under -L) and 24 (file vanished mid-transfer) are successes diff --git a/examples/rsync/render.txt b/examples/rsync/render.txt index 6fa53e5..604394a 100644 --- a/examples/rsync/render.txt +++ b/examples/rsync/render.txt @@ -163,10 +163,10 @@ PY task: [label] LC_ALL=C sort /work/examples/rsync/.run/labels.raw > /work/examples/rsync/.run/labels.txt && rm /work/examples/rsync/.run/labels.raw task: [publish] cat /work/examples/rsync/.run/labels.txt >> /work/examples/rsync/.run/labeled.txt -task: [label-trees] python3 -I - /work/examples/rsync/staging /work/examples/rsync/.run/up /work/examples/rsync/.run/labels.txt up <<'PY' +task: [label-trees] python3 -I - /work/examples/rsync/staging /work/examples/rsync/.run/up /work/examples/rsync/.run/labels.txt up "$FRESH_KEY" <<'PY' import os, shutil, sys -staging, up, labels, to = (os.fsencode(a) for a in sys.argv[1:5]) +staging, up, labels, to, fresh = (os.fsencode(a) for a in sys.argv[1:6]) def move(a, b): os.makedirs(os.path.dirname(b), exist_ok=True) @@ -178,7 +178,7 @@ if to == b"up": with open(labels, "rb") as f: for line in f: path, t = line.rstrip(b"\n").split(b"\t") - if path != b"timestamp": + if path != fresh: n = trees.setdefault(t, str(len(trees) + 1).encode()) move(os.path.join(staging, path), os.path.join(up, n, path)) for t, n in trees.items(): @@ -195,10 +195,10 @@ else: PY task: [publish] for d in /work/examples/rsync/.run/up/*/; do test -d "$d" || continue; aws s3 cp --no-progress --cli-connect-timeout 60 --cli-read-timeout 300 --recursive --content-type "$(cat "${d%/}.type")" "$d" s3://example/ | tee -a /work/examples/rsync/.run/publish.txt; done -task: [label-trees] python3 -I - /work/examples/rsync/staging /work/examples/rsync/.run/up /work/examples/rsync/.run/labels.txt staging <<'PY' +task: [label-trees] python3 -I - /work/examples/rsync/staging /work/examples/rsync/.run/up /work/examples/rsync/.run/labels.txt staging "$FRESH_KEY" <<'PY' import os, shutil, sys -staging, up, labels, to = (os.fsencode(a) for a in sys.argv[1:5]) +staging, up, labels, to, fresh = (os.fsencode(a) for a in sys.argv[1:6]) def move(a, b): os.makedirs(os.path.dirname(b), exist_ok=True) @@ -210,7 +210,7 @@ if to == b"up": with open(labels, "rb") as f: for line in f: path, t = line.rstrip(b"\n").split(b"\t") - if path != b"timestamp": + if path != fresh: n = trees.setdefault(t, str(len(trees) + 1).encode()) move(os.path.join(staging, path), os.path.join(up, n, path)) for t, n in trees.items(): @@ -226,7 +226,7 @@ else: shutil.rmtree(up, ignore_errors=True) PY -task: [publish] test ! -f /work/examples/rsync/staging/timestamp || aws s3 cp --no-progress --cli-connect-timeout 60 --cli-read-timeout 300 --content-type "$(awk -F'\t' '$1 == "timestamp" { print $2 }' /work/examples/rsync/.run/labels.txt)" /work/examples/rsync/staging/timestamp s3://example/timestamp | tee -a /work/examples/rsync/.run/publish.txt +task: [publish] test ! -f /work/examples/rsync/staging/"$FRESH_KEY" || aws s3 cp --no-progress --cli-connect-timeout 60 --cli-read-timeout 300 --content-type "$(awk -F'\t' -v k="$FRESH_KEY" '$1 == k { print $2 }' /work/examples/rsync/.run/labels.txt)" /work/examples/rsync/staging/"$FRESH_KEY" s3://example/"$FRESH_KEY" | tee -a /work/examples/rsync/.run/publish.txt task: [merge] (cd /work/examples/rsync/staging && find . -type f | sed 's|^\./||' | LC_ALL=C sort) > /work/examples/rsync/.run/landed.txt task: [merge] awk -F'\t' 'FILENAME == ARGV[1] { drop[$1] = 1; next } !($1 in drop)' /work/examples/rsync/.run/applied.txt | cat - <(LC_ALL=C join -t "$(printf '\t')" /work/examples/rsync/.run/landed.txt ) | LC_ALL=C sort > /work/examples/rsync/.run/applied.new task: [push] xz -T0 -c /work/examples/rsync/.run/applied.new > /work/examples/rsync/.run/applied.new.xz @@ -272,11 +272,11 @@ case $rc in *) exit $rc ;; esac -task: [pages] PAGE_FOOT='' awk 'BEGIN { g = ENVIRON["PAGE_FOOT"]; gsub(/%%/, "", g) +task: [pages] awk 'BEGIN { g = ENVIRON["PAGE_FOOT"]; gsub(/%%/, "", g) if (g !~ /^[^%]*(%s[^%]*)?$/) { print "pages: PAGE_FOOT can have one %s, and %%, not " ENVIRON["PAGE_FOOT"] > "/dev/stderr"; exit 1 } }' task: [pages] mkdir -p /work/examples/rsync/staging /work/examples/rsync/.run/slash && find /work/examples/rsync/staging /work/examples/rsync/.run/slash -mindepth 1 -delete && : > /work/examples/rsync/.run/index-real.txt task: [pages-select] { cat /work/examples/rsync/.run/applied.txt; awk '{ print $0 "\t\t" }' /work/examples/rsync/.run/upstream-dirs.txt; } | LC_ALL=C sort > /work/examples/rsync/.run/page-state.txt -task: [pages-select] printf '%s\n' 3 example.katoptra.org.directory.index.html 'index.html index.htm' '' '' '' example.katoptra.org | sha256sum | awk '{ print "#pages " substr($1, 1, 16) }' > /work/examples/rsync/.run/index-stamp.txt +task: [pages-select] printf '%s\n' 3 example.katoptra.org.directory.index.html "$INDEX_FILES" "$PAGE_FOOT" "$UPSTREAM" "$CONTACT" example.katoptra.org | sha256sum | awk '{ print "#pages " substr($1, 1, 16) }' > /work/examples/rsync/.run/index-stamp.txt task: [pages-select] awk 'NR == 1 && /^#pages / { next } { print }' /work/examples/rsync/.run/indexed.txt > /work/examples/rsync/.run/index-old.txt task: [pages-select] { LC_ALL=C comm -3 /work/examples/rsync/.run/index-old.txt /work/examples/rsync/.run/page-state.txt | awk -F'\t' '{ print ($1 == "" ? $2 : $1) }'; head -1 /work/examples/rsync/.run/indexed.txt | cmp -s - /work/examples/rsync/.run/index-stamp.txt || cat /work/examples/rsync/.run/page-state.txt; } | awk -F'\t' '!("" in s) { s[""] = 1; print "" } { n = split($1, c, "/"); d = ""; for (i = 1; i < n; i++) { d = d c[i] "/"; if (!(d in s)) { s[d] = 1; print d } } }' | LC_ALL=C sort > /work/examples/rsync/.run/index-dirty.txt @@ -292,7 +292,7 @@ task: [pages-draw] test ! -s /work/examples/rsync/.run/index-make.txt || awk -v task: [pages-draw] test ! -s /work/examples/rsync/.run/index-make.txt || awk -F'\t' -v OFS='\t' 'FILENAME == ARGV[1] { want[$0] = 1; next } { n = split($1, c, "/"); d = "" for (i = 1; i < n; i++) { if ((d in want) && !((d c[i]) in seen)) { seen[d c[i]] = 1; print d, "d", c[i], "", "" }; d = d c[i] "/" } - if (d in want) print d, (c[n] == "" ? "e" : "f"), c[n], $2, $3 }' /work/examples/rsync/.run/index-make.txt /work/examples/rsync/.run/page-state.txt | LC_ALL=C sort -t "$(printf '\t')" -k1,1 -k2,2 -k3,3V | awk -F'\t' -v S="/work/examples/rsync/staging/" -v R="/work/examples/rsync/.run/slash/" -v I="example.katoptra.org.directory.index.html" -v F='' -v N='index.html index.htm' -v X="/work/examples/rsync/.run/index-real.txt" -v U='' -v C='' -v H="example.katoptra.org" -v css=':root{color-scheme:light dark;--bg:#f6f5f2;--fg:#1c1c28;--mut:#6e6e7e;--line:#e2e0da;--hov:#edebe5;--a:#3b45d9}@media (prefers-color-scheme:dark){:root{--bg:#121218;--fg:#f0f0f5;--mut:#8b8b9c;--line:#262630;--hov:#1a1a22;--a:#8b93ff}}body{margin:0 auto;max-width:60rem;padding:2rem 1rem 3rem;background:var(--bg);color:var(--fg);font:15px/1.45 system-ui,-apple-system,sans-serif}h1{font-size:1.35rem;font-weight:650;margin:0;overflow-wrap:anywhere}p{color:var(--mut);margin:.2rem 0 1.25rem;font-size:.9rem}table{width:100%;border-collapse:collapse;font:13.5px/1.4 ui-monospace,SFMono-Regular,Menlo,Consolas,monospace}th{font:600 .7rem/1 system-ui,sans-serif;letter-spacing:.06em;text-transform:uppercase;color:var(--mut);text-align:left;padding:0 .5rem .6rem;white-space:nowrap}td{padding:.4rem .5rem;border-top:1px solid var(--line);vertical-align:baseline}td:first-child{overflow-wrap:anywhere;width:100%}td+td{white-space:nowrap;color:var(--mut);font-variant-numeric:tabular-nums}th:last-child,td:last-child{text-align:right}tr:hover td{background:var(--hov)}a{color:var(--a);text-decoration:none}a:hover{text-decoration:underline}.s a{color:var(--mut)}.k{font-size:.72rem;margin-left:.6rem;padding:.05rem .4rem;border:1px solid var(--line);border-radius:.3rem;color:var(--mut)}.k:hover{color:var(--a);border-color:var(--a);text-decoration:none}footer{margin-top:2rem;padding-top:1rem;border-top:1px solid var(--line);color:var(--mut);font-size:.85rem;overflow-wrap:anywhere}footer a{color:inherit;text-decoration:underline}@media (max-width:36rem){thead{display:none}tr{display:grid;grid-template-columns:1fr auto;border-top:1px solid var(--line)}td{border:0;padding:.1rem .5rem}td:first-child{grid-column:1/-1;padding-top:.45rem}td+td{padding-bottom:.45rem;font-size:.8rem}}' 'BEGIN { for (i = 0; i < 256; i++) ord[sprintf("%c", i)] = i } + if (d in want) print d, (c[n] == "" ? "e" : "f"), c[n], $2, $3 }' /work/examples/rsync/.run/index-make.txt /work/examples/rsync/.run/page-state.txt | LC_ALL=C sort -t "$(printf '\t')" -k1,1 -k2,2 -k3,3V | awk -F'\t' -v S="/work/examples/rsync/staging/" -v R="/work/examples/rsync/.run/slash/" -v I="example.katoptra.org.directory.index.html" -v F="$PAGE_FOOT" -v N="$INDEX_FILES" -v X="/work/examples/rsync/.run/index-real.txt" -v U="$UPSTREAM" -v C="$CONTACT" -v H="example.katoptra.org" -v css=':root{color-scheme:light dark;--bg:#f6f5f2;--fg:#1c1c28;--mut:#6e6e7e;--line:#e2e0da;--hov:#edebe5;--a:#3b45d9}@media (prefers-color-scheme:dark){:root{--bg:#121218;--fg:#f0f0f5;--mut:#8b8b9c;--line:#262630;--hov:#1a1a22;--a:#8b93ff}}body{margin:0 auto;max-width:60rem;padding:2rem 1rem 3rem;background:var(--bg);color:var(--fg);font:15px/1.45 system-ui,-apple-system,sans-serif}h1{font-size:1.35rem;font-weight:650;margin:0;overflow-wrap:anywhere}p{color:var(--mut);margin:.2rem 0 1.25rem;font-size:.9rem}table{width:100%;border-collapse:collapse;font:13.5px/1.4 ui-monospace,SFMono-Regular,Menlo,Consolas,monospace}th{font:600 .7rem/1 system-ui,sans-serif;letter-spacing:.06em;text-transform:uppercase;color:var(--mut);text-align:left;padding:0 .5rem .6rem;white-space:nowrap}td{padding:.4rem .5rem;border-top:1px solid var(--line);vertical-align:baseline}td:first-child{overflow-wrap:anywhere;width:100%}td+td{white-space:nowrap;color:var(--mut);font-variant-numeric:tabular-nums}th:last-child,td:last-child{text-align:right}tr:hover td{background:var(--hov)}a{color:var(--a);text-decoration:none}a:hover{text-decoration:underline}.s a{color:var(--mut)}.k{font-size:.72rem;margin-left:.6rem;padding:.05rem .4rem;border:1px solid var(--line);border-radius:.3rem;color:var(--mut)}.k:hover{color:var(--a);border-color:var(--a);text-decoration:none}footer{margin-top:2rem;padding-top:1rem;border-top:1px solid var(--line);color:var(--mut);font-size:.85rem;overflow-wrap:anywhere}footer a{color:inherit;text-decoration:underline}@media (max-width:36rem){thead{display:none}tr{display:grid;grid-template-columns:1fr auto;border-top:1px solid var(--line)}td{border:0;padding:.1rem .5rem}td:first-child{grid-column:1/-1;padding-top:.45rem}td+td{padding-bottom:.45rem;font-size:.8rem}}' 'BEGIN { for (i = 0; i < 256; i++) ord[sprintf("%c", i)] = i } function enc(s, out, j, c) { out = ""; for (j = 1; j <= length(s); j++) { c = substr(s, j, 1) out = out ((c ~ /[A-Za-z0-9\/._~-]/) ? c : sprintf("%%%02X", ord[c])) } return out } BEGIN { n = split(N, nm, " "); for (i = n; i >= 1; i--) rank[nm[i]] = i @@ -352,7 +352,7 @@ task: [remove] for f in /work/examples/rsync/.run/del-*; do awk 'BEGIN { printf printf "%s{\"Key\":\"%s\"}", (NR > 1 ? "," : ""), $0 } END { print "],\"Quiet\":true}" }' "$f" > "$f.json" && aws s3api delete-objects --bucket example --delete "file://$f.json" | tee -a /work/examples/rsync/.run/publish.txt | { test "$(grep -c Errors)" = 0; }; done task: [push] xz -T0 -c /work/examples/rsync/.run/indexed.new > /work/examples/rsync/.run/indexed.new.xz task: [push] aws s3 cp --no-progress --cli-connect-timeout 60 --cli-read-timeout 300 /work/examples/rsync/.run/indexed.new.xz s3://example/.state/indexed.txt.xz -task: [smoke-sample] { awk -F'\t' '!seen[$2]++' /work/examples/rsync/.run/labeled.txt 2>/dev/null; } | while IFS="$(printf '\t')" read -r k t; do u=$(printf '%s\n' "$k" | awk 'BEGIN { for (i = 0; i < 256; i++) ord[sprintf("%c", i)] = i } +task: [smoke-sample] { awk -F'\t' -v k="$FRESH_KEY" '$1 == k { print $1 "\t" }' /work/examples/rsync/.run/applied.txt 2>/dev/null || true; awk -F'\t' '!seen[$2]++' /work/examples/rsync/.run/labeled.txt 2>/dev/null; } | while IFS="$(printf '\t')" read -r k t; do u=$(printf '%s\n' "$k" | awk 'BEGIN { for (i = 0; i < 256; i++) ord[sprintf("%c", i)] = i } function enc(s, out, j, c) { out = ""; for (j = 1; j <= length(s); j++) { c = substr(s, j, 1) out = out ((c ~ /[A-Za-z0-9\/._~-]/) ? c : sprintf("%%%02X", ord[c])) } return out } { print $0 "\t" enc($0) }' | cut -f2); want=$(awk -F'\t' -v k="$k" '$1 == k { print $2 }' /work/examples/rsync/.run/upstream.txt); h=$(curl -fsS --connect-timeout 15 --max-time 60 --retry 6 --retry-connrefused --retry-max-time 600 -r 0-0 -D - -o /dev/null "https://example.katoptra.org/$u"); got=$(printf '%s\n' "$h" | awk 'tolower($1) == "content-range:" { split($3, a, "/"); r = a[2] } tolower($1) == "content-length:" { c = $2 } @@ -365,12 +365,12 @@ function enc(s, out, j, c) { out = ""; for (j = 1; j <= length(s); j++) { c = s tolower($1) == "content-length:" { c = $2 } $2 == "416" { z = 1 } END { sub(/\r/, "", r); sub(/\r/, "", c); print (r != "" ? r : (z ? 0 : c)) }') bytes, type $(printf '%s\n' "$h" | awk 'tolower($1) == "content-type:" { sub(/\r$/, ""); sub(/^[^:]*:[ \t]*/, ""); print }')"; }; s=$(awk -F'\t' -v k="${d}example.katoptra.org.directory.index.html" '$2 == k { print $1 }' /work/examples/rsync/.run/index-real.txt); if test -n "$s"; then want=$(r "$s"); else want=$(wc -c < "/work/examples/rsync/staging/${d}example.katoptra.org.directory.index.html" | tr -d ' '); test -n "$want"; want="$want bytes"; fi; got=$(r "${d}example.katoptra.org.directory.index.html"); test -n "$s" || got=${got%%,*}; echo "${d}example.katoptra.org.directory.index.html: $got (${s:-page}: $want)"; test "$got" = "$want"; case "https://example.katoptra.org" in http*) test -z "$d" || { p=${d%/}; want=$(wc -c < "/work/examples/rsync/.run/slash/$(printf '%s\n' "$p" | awk -F/ '{ print NF }')/$p" | tr -d ' '); test -n "$want"; got=$(r "$p"); got=${got%%,*}; echo "$p: $got (listing: $want bytes)"; test "$got" = "$want bytes"; } ;; esac; } -task: [smoke-canary] h=pub/index.html; n=$(awk -F'\t' -v k="$h" '$1 == k { print $2 }' /work/examples/rsync/.run/applied.txt 2>/dev/null); case "https://example.katoptra.org" in http*) ;; *) n= ;; esac; if test -z "$n"; then echo "canary skipped ($h is not in the state, or the URL is not http)"; else aws s3 cp --no-progress --cli-connect-timeout 60 --cli-read-timeout 300 "s3://example/$h" /work/examples/rsync/.run/canary.html && curl -fsS --connect-timeout 15 --max-time 60 --retry 6 --retry-connrefused --retry-max-time 600 -A "libwww-perl/6.67" "https://example.katoptra.org/$h" | cmp - /work/examples/rsync/.run/canary.html && echo "$h: $n bytes, served to a libwww-perl client as the bucket holds them"; fi +task: [smoke-canary] h="$CANARY"; n=$(awk -F'\t' -v k="$h" '$1 == k { print $2 }' /work/examples/rsync/.run/applied.txt 2>/dev/null); case "https://example.katoptra.org" in http*) ;; *) n= ;; esac; if test -z "$n"; then echo "canary skipped ($h is not in the state, or the URL is not http)"; else aws s3 cp --no-progress --cli-connect-timeout 60 --cli-read-timeout 300 "s3://example/$h" /work/examples/rsync/.run/canary.html && curl -fsS --connect-timeout 15 --max-time 60 --retry 6 --retry-connrefused --retry-max-time 600 -A "libwww-perl/6.67" "https://example.katoptra.org/$h" | cmp - /work/examples/rsync/.run/canary.html && echo "$h: $n bytes, served to a libwww-perl client as the bucket holds them"; fi task: [smoke-encoding] g=$(awk -F'\t' '$1 ~ /\.tar\.gz$/ && $2 > 0 { print $1; exit }' /work/examples/rsync/.run/applied.txt 2>/dev/null); case "https://example.katoptra.org" in http*) ;; *) g= ;; esac; if test -z "$g"; then echo "content-encoding check skipped (no non-empty .tar.gz in the state, or the URL is not http)"; else u=$(printf '%s\n' "$g" | awk 'BEGIN { for (i = 0; i < 256; i++) ord[sprintf("%c", i)] = i } function enc(s, out, j, c) { out = ""; for (j = 1; j <= length(s); j++) { c = substr(s, j, 1) out = out ((c ~ /[A-Za-z0-9\/._~-]/) ? c : sprintf("%%%02X", ord[c])) } return out } { print $0 "\t" enc($0) }' | cut -f2) && h=$(curl -fsS --connect-timeout 15 --max-time 60 --retry 6 --retry-connrefused --retry-max-time 600 -r 0-0 -D - -o /dev/null "https://example.katoptra.org/$u") && if printf '%s' "$h" | grep -qi '^content-encoding:'; then echo "$g: served with a content-encoding" >&2; exit 1; fi && echo "$g: served with no content-encoding"; fi task: [smoke-unlanded] if test -f /work/examples/rsync/.run/changed.txt && test ! -f /work/examples/rsync/.run/chain; then LC_ALL=C comm -23 /work/examples/rsync/.run/changed.txt /work/examples/rsync/.run/applied.txt > /work/examples/rsync/.run/unlanded.txt; else : > /work/examples/rsync/.run/unlanded.txt; fi; cut -f1 /work/examples/rsync/.run/unlanded.txt | head -20 | while IFS= read -r p; do echo "::warning::listed upstream, never fetched: $p"; done -task: [fresh] k=timestamp; if ! awk -F'\t' -v k="$k" '$1 == k { f = 1 } END { exit !f }' /work/examples/rsync/.run/applied.txt 2>/dev/null; then echo "freshness skipped ($k is not in the state)"; else h=24; case "$h" in ''|0?*|*[!0-9]*) echo "fresh: FRESH_HOURS is a whole number, not $h" >&2; exit 1 ;; esac; u=$(printf '%s\n' "$k" | awk 'BEGIN { for (i = 0; i < 256; i++) ord[sprintf("%c", i)] = i } +task: [fresh] k="$FRESH_KEY"; if ! awk -F'\t' -v k="$k" '$1 == k { f = 1 } END { exit !f }' /work/examples/rsync/.run/applied.txt 2>/dev/null; then echo "freshness skipped ($k is not in the state)"; else h=24; case "$h" in ''|0?*|*[!0-9]*) echo "fresh: FRESH_HOURS is a whole number, not $h" >&2; exit 1 ;; esac; u=$(printf '%s\n' "$k" | awk 'BEGIN { for (i = 0; i < 256; i++) ord[sprintf("%c", i)] = i } function enc(s, out, j, c) { out = ""; for (j = 1; j <= length(s); j++) { c = substr(s, j, 1) out = out ((c ~ /[A-Za-z0-9\/._~-]/) ? c : sprintf("%%%02X", ord[c])) } return out } { print $0 "\t" enc($0) }' | cut -f2); t=$(curl -fsS --connect-timeout 15 --max-time 60 --retry 6 --retry-connrefused --retry-max-time 600 "https://example.katoptra.org/$u" | awk '!/^#/ { for (i = 1; i <= NF; i++) { if ($i ~ /^[0-9]+$/ && length($i) >= 9) { print $i; exit } From bbe75768bfdd9181cb5d063da5cc4a282aa4d8ad Mon Sep 17 00:00:00 2001 From: Josh Vaughen Date: Thu, 22 Oct 2026 14:19:30 -0700 Subject: [PATCH 5/6] refactor(rsync): no line over 110 characters, checked in CI 105 lines of engines/rsync.yml were longer than 110 characters, the longest 424. Each one breaks in a form that keeps its text: - A desc: becomes a folded block. task --list-all --json gives the same desc for each task. - A shell line in a >- block breaks at a space. The render of these lines is identical. - A flow map becomes a block map. - An awk line breaks after ; or {, before else, or with a backslash at the end of the line. A long string becomes two strings in sequence. - report-engine computes its long values into shell vars first, and an echo with several arguments puts one space between them. A new CI step, in the rsync job, fails on a line longer than 110 characters. CONTRIBUTING gives the methods above. No behavior change: offline passes (pages byte for byte), and parity against v2.9.1 shows the same tree and exit code in 119 of 119 scenarios, with the same summary rows in each report-engine scenario. --- .github/workflows/ci.yml | 7 + CLAUDE.md | 3 +- CONTRIBUTING.md | 14 +- README.md | 6 +- engines/rsync.yml | 398 +++++++++++++++++++++++++++----------- examples/rsync/render.txt | 77 +++++--- 6 files changed, 369 insertions(+), 136 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 90eb9cc..703f828 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -26,6 +26,13 @@ jobs: - name: The op verb tries again only when the command did not start if: matrix.variant == 'rsync' run: sh .github/op-retry-check.sh + # A line of the rsync engine that is longer than 110 characters is too long to read in + # a diff or in a review. + - name: No line of the rsync engine is longer than 110 characters + if: matrix.variant == 'rsync' + run: >- + awk 'length > 110 { print FILENAME ":" FNR ": " length " characters"; bad = 1 } END { exit bad }' + engines/rsync.yml - uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1 # Actions keeps the layers in its cache, with one scope for each variant. release # reads the same scope. Thus, for a tag, the build makes new layers only for the diff --git a/CLAUDE.md b/CLAUDE.md index 7d1ca8a..95c8361 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -98,7 +98,7 @@ Read that section before you write. no key without a slash. Thus, if the only changed directory of a run is the root, that awk sends no line to `xargs`. - A `>-` folded block keeps a newline if a line that continues the line before it starts - with more spaces than the lines around it. The two-line awk programs in the commands of + with more spaces than the lines around it. The short awk programs in the commands of `split`, `pages` and `pages-draw` use this. Put a backslash at the end of a shell line that must continue. `task check` shows how the block renders. - On a macOS disk, two directory names that are different only in uppercase and lowercase @@ -159,6 +159,7 @@ cd examples/rsync && task image-build && task run -- task tools && task check & cd examples/proton && task image-build && task run -- task tools && task check && task run -- task offline sh .github/validate-vars.sh --check && sh .github/chain-file.sh --check && sh docker/gpg-gate-check.sh sh .github/op-retry-check.sh +awk 'length > 110 { print FILENAME ":" FNR ": " length " characters"; bad = 1 } END { exit bad }' engines/rsync.yml ``` After a change to a verb, run `task render-update` to update the `render.txt` files. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 7455d32..8b6b6b9 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -32,6 +32,7 @@ are applicable to this repository. This repository adds three rules: cd examples/rsync && task image-build && task run -- task tools && task check && task run -- task offline cd examples/proton && task image-build && task run -- task tools && task check && task run -- task offline sh .github/validate-vars.sh --check && sh .github/chain-file.sh --check && sh docker/gpg-gate-check.sh +awk 'length > 110 { print FILENAME ":" FNR ": " length " characters"; bad = 1 } END { exit bad }' engines/rsync.yml ``` CI does these steps for each image, on each pull request and on each push to `main`: @@ -41,4 +42,15 @@ CI does these steps for each image, on each pull request and on each push to `ma 3. It runs `task check`. 4. It runs `task run -- task offline`. -It also runs the three guards one time, in the job of the `rsync` image. +It also runs the three guards one time, in the job of the `rsync` image. That job also +makes sure that no line of `engines/rsync.yml` is longer than 110 characters. To make a +line shorter, use the methods that the file uses: + +- A `desc:` becomes a folded block (`desc: >-`). YAML joins its lines with spaces, and the + text does not change. +- A shell line in a `>-` block breaks at a space. A line with more spaces than the lines + around it keeps its newline. Thus, do not break such a line inside a quoted string. +- A long awk line breaks after `;` or `{`, before `else`, or with a backslash at the end of + the line. A string can become two strings, one after the other. +- An `echo` gets several arguments. It puts one space between two arguments, and the + output does not change. diff --git a/README.md b/README.md index 9491a97..202ecc0 100644 --- a/README.md +++ b/README.md @@ -1497,9 +1497,9 @@ because `empty-trash` starts that task from the command line in the image. Last, encrypts and decrypts one file with `age`. CI runs the same four steps for each variant on each pull request and on each push to -`main`. It also runs the test -cases of the three guards: `.github/validate-vars.sh`, `.github/chain-file.sh` and -`docker/gpg-gate-check.sh`. +`main`. It also runs the test cases of the three guards: `.github/validate-vars.sh`, +`.github/chain-file.sh` and `docker/gpg-gate-check.sh`. It also makes sure that no line of +`engines/rsync.yml` is longer than 110 characters. When you change a verb, run `task render-update` in each example. The diff in the pull request is the review. `CONTRIBUTING.md` gives the three rules that this repository adds to diff --git a/engines/rsync.yml b/engines/rsync.yml index 569b064..eab461f 100644 --- a/engines/rsync.yml +++ b/engines/rsync.yml @@ -121,7 +121,8 @@ vars: # together give the key of the page. ANCESTORS: |- awk -F'\t' '!("" in s) { s[""] = 1; print "" } - { n = split($1, c, "/"); d = ""; for (i = 1; i < n; i++) { d = d c[i] "/"; if (!(d in s)) { s[d] = 1; print d } } }' + { n = split($1, c, "/"); d = "" + for (i = 1; i < n; i++) { d = d c[i] "/"; if (!(d in s)) { s[d] = 1; print d } } }' # The clock of upstream in a timestamp file, as Unix time or as "YYYY-MM-DD HH:MM" for # date -d. mirror-updated-timestamp.txt (gnu) and 00_TIME.txt (Savannah) start with Unix # time. The timestamp file of CTAN holds 2026-10-08-05-02, after a comment and the name @@ -188,7 +189,8 @@ vars: /^containerchecksum /{split($0, a, " "); emit(a[2], n)} /^doccontainerchecksum /{split($0, a, " "); emit(a[2], n ".doc")} /^srccontainerchecksum /{split($0, a, " "); emit(a[2], n ".source")} - function emit(sum, base, p, q) { p = TL "/archive/" base ".tar.xz"; q = TL "/archive/" base ".r" r ".tar.xz" + function emit(sum, base, p, q) { + p = TL "/archive/" base ".tar.xz"; q = TL "/archive/" base ".r" r ".tar.xz" if (p in want) { print sum " " p; delete want[p] } if (q in want) { print sum " " q; delete want[q] } } END { for (p in want) { print "no checksum in the tlpdb for " p > "/dev/stderr"; bad = 1 } exit bad } @@ -330,10 +332,13 @@ vars: # - S and R: STAGING and SLASH, each with a slash at the end. SAME_PAGES: |- FILENAME == E { gsub(/"/, "", $2); e[$1] = $2; next } - FILENAME == H { if ($2 == "s") d = $3 "/"; else if (substr($3, length($3) - length(I) + 1) == I) d = substr($3, 1, length($3) - length(I)); else next + FILENAME == H { if ($2 == "s") d = $3 "/" + else if (substr($3, length($3) - length(I) + 1) == I) d = substr($3, 1, length($3) - length(I)) + else next seen[d] = 1; if (e[$3] != $1) diff[d] = 1; next } !($0 in seen) || ($0 in diff) { print > M; next } - { print > Y; print S $0 I; if ($0 != "") { b = substr($0, 1, length($0) - 1); print R split(b, c, "/") "/" b } } + { print > Y; print S $0 I + if ($0 != "") { b = substr($0, 1, length($0) - 1); print R split(b, c, "/") "/" b } } # pages: the entries "dir TAB d|e|f TAB name TAB size TAB mtime" of each directory of the # first input (index-make.txt), from the page state (the second input): the files in the # bucket and the directories of upstream. A line "dir/" of the page state gives an e entry @@ -342,7 +347,9 @@ vars: PAGE_ENTRIES: |- FILENAME == ARGV[1] { want[$0] = 1; next } { n = split($1, c, "/"); d = "" - for (i = 1; i < n; i++) { if ((d in want) && !((d c[i]) in seen)) { seen[d c[i]] = 1; print d, "d", c[i], "", "" }; d = d c[i] "/" } + for (i = 1; i < n; i++) { + if ((d in want) && !((d c[i]) in seen)) { seen[d c[i]] = 1; print d, "d", c[i], "", "" } + d = d c[i] "/" } if (d in want) print d, (c[n] == "" ? "e" : "f"), c[n], $2, $3 } # pages: the style of each directory page. Each line ends with a backslash. Thus, YAML # joins the lines with no space, and the style is one line in each page. @@ -400,16 +407,19 @@ vars: BEGIN { n = split(N, nm, " "); for (i = n; i >= 1; i--) rank[nm[i]] = i G = "\\.(sig|sign|asc|gpg|dsig|proof|md5|md5sum|sha|sha1|sha224|sha256|sha256sum|sha512|sum|checksums)$" if (C == "" && (j = index(H, ".")) > 1) C = substr(H, 1, j - 1) "@" substr(H, j + 1) } - function esc(s) { gsub(/&/, "\\&", s); gsub(//, "\\>", s); gsub(/"/, "\\"", s); return s } + function esc(s) { gsub(/&/, "\\&", s); gsub(//, "\\>", s) + gsub(/"/, "\\"", s); return s } function grp(n, s, o) { s = sprintf("%.0f", n); o = "" - while (length(s) > 3) { o = "," substr(s, length(s) - 2) o; s = substr(s, 1, length(s) - 3) } return s o } + while (length(s) > 3) { o = "," substr(s, length(s) - 2) o; s = substr(s, 1, length(s) - 3) } + return s o } function iec(n, u, i) { if (n < 1024) return sprintf("%.0f B", n) split("KiB MiB GiB TiB", u, " "); n /= 1024; i = 1; while (n >= 1024 && i < 4) { n /= 1024; i++ } return sprintf(n < 10 ? "%.1f %s" : "%.0f %s", n, u[i]) } function ymd(t) { gsub(/\//, "-", t); return substr(t, 1, 16) } function foot( t) { if (F != "") return "
" sprintf(F, enc(cur)) "
" t = (U == "" ? "" : "Mirror of " esc(U) enc(cur) ".") - if (C != "") t = t (t == "" ? "" : " ") "Report a problem to " esc(C) "." + if (C != "") t = t (t == "" ? "" : " ") "Report a problem to " \ + "" esc(C) "." return (t == "" ? "" : "
" t "
") } function open(d) { cur = d; live = 1; real = ""; nd = 0; nfl = 0; tot = 0; split("", has) } function w(s) { if (real == "") printf("%s", s) > fa; if (cur != "") printf("%s", s) > fb } @@ -417,19 +427,26 @@ vars: for (i = 1; i <= nfl; i++) { r = fname[i]; x = "" while (match(tolower(r), G)) { r = substr(r, 1, RSTART - 1); if (r in has) x = r } if (x != "") { fold[i] = 1 - links[x] = links[x] "" esc(substr(fname[i], length(x) + 1)) "" } } + links[x] = links[x] "" \ + esc(substr(fname[i], length(x) + 1)) "" } } c = (nd ? grp(nd) (nd == 1 ? " directory" : " directories") : "") if (nfl) c = c (c == "" ? "" : ", ") grp(nfl) (nfl == 1 ? " file" : " files") ", " iec(tot) if (c == "") c = "This directory is empty." fa = S cur I; if (real != "") print cur real "\t" cur I > X if (cur != "") { b = substr(cur, 1, length(cur) - 1); k = split(b, e, "/"); fb = R k "/" b } - w("\nIndex of /" esc(cur) "\n") - w("\n\n

Index of /" esc(cur) "

" c "

\n") + w("\n" \ + "" \ + "Index of /" esc(cur) "\n") + w("\n\n" \ + "

Index of /" esc(cur) "

" c "

\n") w("\n") if (cur != "") w("\n") - for (i = 1; i <= nd; i++) w("\n") + for (i = 1; i <= nd; i++) + w("\n") for (i = 1; i <= nfl; i++) if (!(i in fold)) - w("\n") + w("" \ + "\n") w("
NameModified (UTC)Size
../
" esc(dname[i]) "/
" esc(dname[i]) "/
" esc(fname[i]) "" links[fname[i]] "" ymd(ftime[i]) "" iec(fsize[i]) "
" \ + esc(fname[i]) "" links[fname[i]] "" ymd(ftime[i]) "" iec(fsize[i]) "
" foot() "\n") if (real == "") close(fa); if (cur != "") close(fb) } !live || $1 != cur { if (live) shut(); open($1) } @@ -441,7 +458,9 @@ vars: tasks: pipeline: - desc: clock -> due -> list -> state -> rebuild? -> diff -> split -> prepare -> batches -> delete -> reconcile? -> index -> smoke -> report -> ping + desc: >- + clock -> due -> list -> state -> rebuild? -> diff -> split -> prepare -> batches -> delete -> + reconcile? -> index -> smoke -> report -> ping cmds: - {task: clock} - {task: due} @@ -470,14 +489,18 @@ tasks: # ---- the listing and the state ---- list: - desc: List upstream, narrowed by FILTER when set, and normalise it to RUN/upstream.txt as "path TAB size TAB mtime", byte-sorted + desc: >- + List upstream, narrowed by FILTER when set, and normalise it to RUN/upstream.txt as "path TAB + size TAB mtime", byte-sorted set: [pipefail] cmds: # In a run, each batch appends to publish.txt and labeled.txt. Only the verb that # uses one of the other files writes that file. A workspace that a previous run used # must start each run without these files. If not, smoke and the report count lines # from a previous run. - - rm -f {{.RUN}}/publish.txt {{.RUN}}/labeled.txt {{.RUN}}/orphans.txt {{.RUN}}/unlanded.txt {{.RUN}}/fresh.txt {{.RUN}}/index-same.txt + - >- + rm -f {{.RUN}}/publish.txt {{.RUN}}/labeled.txt {{.RUN}}/orphans.txt {{.RUN}}/unlanded.txt + {{.RUN}}/fresh.txt {{.RUN}}/index-same.txt - mkdir -p {{.RUN}} {{.STAGING}} # -L gives each symlink as the file that it points to. If the target of a symlink is # not upstream, rsync reports the symlink and ignores it. rsync then exits 23, but @@ -491,11 +514,17 @@ tasks: # a name as \#ooo. fetch then asks for a name that upstream does not have, and the # file never lands. -8 writes the bytes of the name. - task: retry - vars: {CMD: 'TZ=UTC {{.RSYNC}}{{if .FILTER}} {{.FILTER}}{{end}} -8 -rL --list-only {{.SOURCE}} > {{.RUN}}/listing.txt'} + vars: + CMD: >- + TZ=UTC {{.RSYNC}}{{if .FILTER}} {{.FILTER}}{{end}} -8 -rL --list-only {{.SOURCE}} + > {{.RUN}}/listing.txt - {task: normalise} normalise: - desc: RUN/listing.txt (rsync --list-only) to RUN/upstream.txt, every regular file as "path TAB size TAB mtime", and RUN/upstream-dirs.txt, every directory but the root as "path/"; both byte-sorted + desc: >- + RUN/listing.txt (rsync --list-only) to RUN/upstream.txt, every regular file as "path TAB size + TAB mtime", and RUN/upstream-dirs.txt, every directory but the root as "path/"; both + byte-sorted set: [pipefail] cmds: - awk '{{.LIST_FILES}}' {{.RUN}}/listing.txt | LC_ALL=C sort > {{.RUN}}/upstream.txt @@ -503,18 +532,29 @@ tasks: # A truncated listing must not become a list of deletions. LIST_FLOOR is a line # count, approximately 90% of the usual size of the listing. A correct listing # always has more lines. When LIST_FLOOR is not set, there is no guard. - - '{{if .LIST_FLOOR}}test "$(wc -l < {{.RUN}}/upstream.txt)" -gt {{.LIST_FLOOR}}{{else}}echo "no LIST_FLOOR: a truncated listing is not caught here"{{end}}' + - >- + {{if .LIST_FLOOR}}test "$(wc -l < {{.RUN}}/upstream.txt)" -gt {{.LIST_FLOOR}}{{else}}echo + "no LIST_FLOOR: a truncated listing is not caught here"{{end}} state: - desc: Fetch the state file to RUN/applied.txt; a missing one is rebuilt from a bucket listing (an empty bucket gives an empty state, and the first run fills it) + desc: >- + Fetch the state file to RUN/applied.txt; a missing one is rebuilt from a bucket listing (an + empty bucket gives an empty state, and the first run fills it) cmds: - mkdir -p {{.RUN}} # The bucket is the mirror, and the state is a cache of the bucket. Thus, a missing # state causes one listing of the bucket, not an upload. - - {task: pull, vars: {KEY: '{{.STATE}}', OUT: '{{.RUN}}/applied.txt', MISSING: 'echo "no state at {{.STATE}}: rebuilding it from the bucket" >&2; touch {{.RUN}}/rebuild-now'}} + - task: pull + vars: + KEY: '{{.STATE}}' + OUT: '{{.RUN}}/applied.txt' + MISSING: >- + echo "no state at {{.STATE}}: rebuilding it from the bucket" >&2; touch {{.RUN}}/rebuild-now pull: - desc: An xz-compressed KEY from the bucket, decompressed to OUT; MISSING runs when the bucket has no such key + desc: >- + An xz-compressed KEY from the bucket, decompressed to OUT; MISSING runs when the bucket has no + such key internal: true requires: {vars: [KEY, OUT, MISSING]} cmds: @@ -529,7 +569,9 @@ tasks: esac push: - desc: FILE, xz-compressed, to the bucket's KEY as one PutObject; the old object or the new one is in the bucket, never half + desc: >- + FILE, xz-compressed, to the bucket's KEY as one PutObject; the old object or the new one is in + the bucket, never half internal: true requires: {vars: [FILE, KEY]} cmds: @@ -561,7 +603,9 @@ tasks: - rm -f {{.RUN}}/rebuild-now diff: - desc: RUN/changed.txt (lines upstream has and the state lacks, plus the file each changed signed sha512 names), deleted.txt (paths the state has and upstream lacks), paths.txt + desc: >- + RUN/changed.txt (lines upstream has and the state lacks, plus the file each changed signed + sha512 names), deleted.txt (paths the state has and upstream lacks), paths.txt cmds: - LC_ALL=C comm -13 {{.RUN}}/applied.txt {{.RUN}}/upstream.txt > {{.RUN}}/changed.txt # Upstream can write a signed sha512 again while the file that it identifies does @@ -570,13 +614,17 @@ tasks: # into the delta. Thus, verify examines it with the new sha512 in staging. The # pattern is the same as in verify. - >- - awk -F'\t' -v TL="{{.TL}}" 'FILENAME == ARGV[1] { if ($1 ~ "^" TL "/(tlpkg/texlive\\.tlpdb|[^/]+)\\.sha512$") want[substr($1, 1, length($1) - 7)]; next } - $1 in want' {{.RUN}}/changed.txt {{.RUN}}/upstream.txt | LC_ALL=C sort -u -o {{.RUN}}/changed.txt - {{.RUN}}/changed.txt + awk -F'\t' -v TL="{{.TL}}" 'FILENAME == ARGV[1] { if ($1 ~ "^" TL + "/(tlpkg/texlive\\.tlpdb|[^/]+)\\.sha512$") want[substr($1, 1, length($1) - 7)]; next } + $1 in want' {{.RUN}}/changed.txt {{.RUN}}/upstream.txt | LC_ALL=C sort -u -o {{.RUN}}/changed.txt - + {{.RUN}}/changed.txt - cut -f1 {{.RUN}}/upstream.txt > {{.RUN}}/paths.txt - cut -f1 {{.RUN}}/applied.txt | LC_ALL=C comm -23 - {{.RUN}}/paths.txt > {{.RUN}}/deleted.txt split: - desc: Refuse a tree over CEILING_GB (when set) or a file the disk cannot hold; split changed.txt into RUN/batch-NNNN.txt + desc: >- + Refuse a tree over CEILING_GB (when set) or a file the disk cannot hold; split changed.txt + into RUN/batch-NNNN.txt vars: # A task var, not a global var, because a caller can give BATCH_GB as a call var. BATCH_SIZE: '{{.BATCH_GB | default 4}}' @@ -589,15 +637,19 @@ tasks: # CEILING_GB unset or 0 gives no ceiling. The command always prints the size. - >- {{if .CEILING_GB}}awk -F'\t' -v gb={{.CEILING_GB}} '{ s += $2 } - END { printf "upstream: %d objects, %.2f GB, ceiling %d GB\n", NR, s / 1e9, gb; exit (s > gb * 1e9) }' \ - {{.RUN}}/upstream.txt{{else}}awk -F'\t' '{ s += $2 } END { printf "upstream: %d objects, %.2f GB, no ceiling\n", NR, s / 1e9 }' {{.RUN}}/upstream.txt{{end}} + END { printf "upstream: %d objects, %.2f GB, ceiling %d GB\n", NR, s / 1e9, gb + exit (s > gb * 1e9) }' \ + {{.RUN}}/upstream.txt{{else}}awk -F'\t' + '{ s += $2 } END { printf "upstream: %d objects, %.2f GB, no ceiling\n", NR, s / 1e9 }' + {{.RUN}}/upstream.txt{{end}} - mkdir -p {{.STAGING}} && find {{.STAGING}} -mindepth 1 -delete - df -Pk {{.STAGING}} # The disk must have space for the largest file, plus 1 GiB for the text files of # RUN and for the OS. - >- awk -F'\t' -v kb="$(df -Pk {{.STAGING}} | awk 'NR == 2 { print $4 }')" \ - '$2 / 1024 > kb - {{.HEADROOM_KIB}} { print "no room for " $1 " (" $2 " bytes)"; bad = 1 } END { exit bad }' \ + '$2 / 1024 > kb - {{.HEADROOM_KIB}} { print "no room for " $1 " (" $2 " bytes)"; bad = 1 } + END { exit bad }' \ {{.RUN}}/changed.txt - rm -f {{.RUN}}/batch-*.txt # BATCH_GB goes into the command with no quotes, because it is in $(( )), where a @@ -609,19 +661,24 @@ tasks: # ---- the batches ---- prepare: - desc: With TL_KEY set and the delta touching TL, fetch its control files into RUN/tl and check the signed sha512 against the pinned key; the no-op hook otherwise + desc: >- + With TL_KEY set and the delta touching TL, fetch its control files into RUN/tl and check the + signed sha512 against the pinned key; the no-op hook otherwise status: - 'test -z "{{.TL_KEY}}" || test "$(grep -c "^{{.TL}}/" {{.RUN}}/changed.txt 2>/dev/null)" = 0' set: [pipefail] cmds: - mkdir -p {{.RUN}}/tl - - printf '%s\n' tlpkg/texlive.tlpdb.xz tlpkg/texlive.tlpdb.sha512 tlpkg/texlive.tlpdb.sha512.asc tlpkg/gpg/pubring.gpg > {{.RUN}}/tl/files.txt + - >- + printf '%s\n' tlpkg/texlive.tlpdb.xz tlpkg/texlive.tlpdb.sha512 tlpkg/texlive.tlpdb.sha512.asc + tlpkg/gpg/pubring.gpg > {{.RUN}}/tl/files.txt - task: retry vars: {CMD: '{{.RSYNC}} -Lt --files-from={{.RUN}}/tl/files.txt {{.SOURCE}}{{.TL}}/ {{.RUN}}/tl/'} - xz -dkf {{.RUN}}/tl/tlpkg/texlive.tlpdb.xz - >- cd {{.RUN}}/tl/tlpkg && shasum -a 512 -c texlive.tlpdb.sha512 && - gpgv --status-fd 1 --keyring gpg/pubring.gpg texlive.tlpdb.sha512.asc texlive.tlpdb.sha512 | awk -v k={{.TL_KEY}} '{{.GPGCHECK}}' + gpgv --status-fd 1 --keyring gpg/pubring.gpg texlive.tlpdb.sha512.asc texlive.tlpdb.sha512 | awk -v + k={{.TL_KEY}} '{{.GPGCHECK}}' batches: desc: Work the first MAX_BATCHES of RUN/batch-NNNN.txt in name order; each commits before the next starts @@ -651,7 +708,9 @@ tasks: internal: true requires: {vars: [B]} preconditions: - - sh: test "$(df -Pk {{.STAGING}} | awk 'NR == 2 { print $4 }')" -gt "$(awk -F'\t' '{ s += $2 } END { print int(s / 1024) + {{.HEADROOM_KIB}} }' {{.B}})" + - sh: >- + test "$(df -Pk {{.STAGING}} | awk 'NR == 2 { print $4 }')" + -gt "$(awk -F'\t' '{ s += $2 } END { print int(s / 1024) + {{.HEADROOM_KIB}} }' {{.B}})" msg: "not enough free disk for {{.B}}" cmds: - find {{.STAGING}} -mindepth 1 -delete @@ -666,7 +725,10 @@ tasks: # No --partial. Each batch removes all files from staging. Thus, there is nothing to # continue. Also, a file that rsync did not complete must not go into the bucket. - task: retry - vars: {CMD: '{{.RSYNC}} -Lt --files-from={{.RUN}}/files.txt --ignore-missing-args {{.SOURCE}} {{.STAGING}}/'} + vars: + CMD: >- + {{.RSYNC}} -Lt --files-from={{.RUN}}/files.txt --ignore-missing-args {{.SOURCE}} + {{.STAGING}}/ verify: # Not internal. Thus, a mirror can start it manually on a batch from a fixture. @@ -698,9 +760,11 @@ tasks: # updaters at the root, and the tlpdb) with the keyring that prepare pinned. The # status of the while loop is the status of the pipeline. - >- - cd {{.STAGING}} && cut -f1 {{.B}} | grep -E '^{{.TL}}/(tlpkg/texlive\.tlpdb|[^/]+)\.sha512$' | while IFS= read -r f; do + cd {{.STAGING}} && cut -f1 {{.B}} | grep -E '^{{.TL}}/(tlpkg/texlive\.tlpdb|[^/]+)\.sha512$' | + while IFS= read -r f; do (cd "$(dirname "$f")" && f=$(basename "$f") && shasum -a 512 -c "$f" && - gpgv --status-fd 1 --keyring {{.RUN}}/tl/tlpkg/gpg/pubring.gpg "$f.asc" "$f" | awk -v k={{.TL_KEY}} '{{.GPGCHECK}}') + gpgv --status-fd 1 --keyring {{.RUN}}/tl/tlpkg/gpg/pubring.gpg "$f.asc" "$f" | awk -v k={{.TL_KEY}} + '{{.GPGCHECK}}') || exit 1; done verify-containers: @@ -725,11 +789,14 @@ tasks: # between the two. - >- cd {{.STAGING}} && { test "$(grep -c '^{{.TL}}/archive/' {{.B}})" = 0 || { - awk -F'\t' -v TL="{{.TL}}" '{{.TLPDB_SUMS}}' {{.B}} {{.RUN}}/tl/tlpkg/texlive.tlpdb > {{.RUN}}/tl/sums.txt && + awk -F'\t' -v TL="{{.TL}}" '{{.TLPDB_SUMS}}' {{.B}} {{.RUN}}/tl/tlpkg/texlive.tlpdb > + {{.RUN}}/tl/sums.txt && shasum -a 512 -c --quiet {{.RUN}}/tl/sums.txt; }; } verify-decision: - desc: When batch B holds the tlpdb, the tlpdb against prepare's, its .xz against it, and each container that it names against the bucket after this run + desc: >- + When batch B holds the tlpdb, the tlpdb against prepare's, its .xz against it, and each + container that it names against the bucket after this run internal: true cmds: # The decision batch. For this batch, these conditions must be correct: @@ -765,7 +832,8 @@ tasks: - {task: label-trees, vars: {TO: up}} - >- for d in {{.RUN}}/up/*/; do test -d "$d" || continue; - aws s3 cp {{.AWS_FLAGS}} --recursive --content-type "$(cat "${d%/}.type")" "$d" {{.S3}}/ | tee -a {{.RUN}}/publish.txt; done + aws s3 cp {{.AWS_FLAGS}} --recursive --content-type "$(cat "${d%/}.type")" "$d" {{.S3}}/ | tee -a + {{.RUN}}/publish.txt; done - {task: label-trees, vars: {TO: staging}} # The FRESH_KEY file goes to the bucket last, in one more command. It gives the date # of the tree, and monitors (for example, mirmon) read it. @@ -775,11 +843,14 @@ tasks: # ENVIRON in these three commands. - >- {{if .FRESH_KEY}}test ! -f {{.STAGING}}/"$FRESH_KEY" || - aws s3 cp {{.AWS_FLAGS}} --content-type "$(awk -F'\t' -v k="$FRESH_KEY" '$1 == k { print $2 }' {{.RUN}}/labels.txt)" + aws s3 cp {{.AWS_FLAGS}} --content-type "$(awk -F'\t' -v k="$FRESH_KEY" '$1 == k { print $2 }' + {{.RUN}}/labels.txt)" {{.STAGING}}/"$FRESH_KEY" {{.S3}}/"$FRESH_KEY" | tee -a {{.RUN}}/publish.txt{{end}} label: - desc: RUN/labels.txt, "path TAB type" for every file under DIR (STAGING), byte-sorted; the type is decided from the name and the first KiB, never left to the CLI + desc: >- + RUN/labels.txt, "path TAB type" for every file under DIR (STAGING), byte-sorted; the type is + decided from the name and the first KiB, never left to the CLI vars: DIR: '{{if .DIR}}{{.DIR}}{{else}}{{.STAGING}}{{end}}' cmds: @@ -795,7 +866,9 @@ tasks: - LC_ALL=C sort {{.RUN}}/labels.raw > {{.RUN}}/labels.txt && rm {{.RUN}}/labels.raw label-trees: - desc: 'TO=up: rename every file of RUN/labels.txt but the FRESH_KEY file out of STAGING into RUN/up/N/, one tree per type, its type in RUN/up/N.type. TO=staging: rename them back' + desc: >- + TO=up: rename every file of RUN/labels.txt but the FRESH_KEY file out of STAGING into + RUN/up/N/, one tree per type, its type in RUN/up/N.type. TO=staging: rename them back internal: true requires: {vars: [TO]} env: {FRESH_KEY: '{{.FRESH_KEY}}'} @@ -828,7 +901,8 @@ tasks: - (cd {{.STAGING}} && find . -type f | sed 's|^\./||' | LC_ALL=C sort) > {{.RUN}}/landed.txt - >- awk -F'\t' 'FILENAME == ARGV[1] { drop[$1] = 1; next } !($1 in drop)' {{.B}} {{.RUN}}/applied.txt - | cat - <(LC_ALL=C join -t "$(printf '\t')" {{.RUN}}/landed.txt {{.B}}) | LC_ALL=C sort > {{.RUN}}/applied.new + | cat - <(LC_ALL=C join -t "$(printf '\t')" {{.RUN}}/landed.txt {{.B}}) | LC_ALL=C sort > + {{.RUN}}/applied.new checkpoint: internal: true @@ -852,11 +926,14 @@ tasks: # errors. Thus, the command uses grep to find them. - >- for f in {{.RUN}}/del-*; do - awk '{{.DELETE_JSON}}' "$f" > "$f.json" && aws s3api delete-objects --bucket {{.BUCKET}} --delete "file://$f.json" | tee -a {{.RUN}}/publish.txt + awk '{{.DELETE_JSON}}' "$f" > "$f.json" && aws s3api delete-objects --bucket {{.BUCKET}} --delete + "file://$f.json" | tee -a {{.RUN}}/publish.txt | { test "$(grep -c Errors)" = 0; }; done delete: - desc: Remove the keys in D (RUN/deleted.txt unless given), 1,000 per DeleteObjects call and free, and drop them from the state + desc: >- + Remove the keys in D (RUN/deleted.txt unless given), 1,000 per DeleteObjects call and free, + and drop them from the state vars: # reconcile gives its orphan list. Thus, the deleted.txt of the run, which report # counts, does not change after diff writes it. @@ -871,7 +948,9 @@ tasks: - {task: checkpoint, vars: {B: '{{.D}}'}} reconcile: - desc: When due left .run/reconcile, rebuild the state from a bucket listing, delete keys that neither upstream nor .state/ owns, and record the reconcile + desc: >- + When due left .run/reconcile, rebuild the state from a bucket listing, delete keys that + neither upstream nor .state/ owns, and record the reconcile # Not while a batch waits for the next run. Then delete keeps the orphans. Thus, the # run stays due, and the chained run that puts the last batch in the bucket # reconciles. A run that has a failure before the last line also records nothing. @@ -895,10 +974,13 @@ tasks: # reconcile keeps each key that ends in "/". - >- {{if .INDEX}}cat {{.RUN}}/applied.txt {{.RUN}}/upstream-dirs.txt | {{.ANCESTORS}} - | awk '$0 != "" { print substr($0, 1, length($0) - 1) }' | LC_ALL=C sort > {{.RUN}}/index-keys.txt{{else}}: > {{.RUN}}/index-keys.txt{{end}} + | awk '$0 != "" { print substr($0, 1, length($0) - 1) }' | LC_ALL=C sort > + {{.RUN}}/index-keys.txt{{else}}: > {{.RUN}}/index-keys.txt{{end}} - >- - cut -f1 {{.RUN}}/bucket.txt | grep -vE '^\.state/'{{if .OWN}} | grep -vxF -f <(printf '%s\n' {{.OWN}}){{end}} - {{if .INDEX}}| awk -v i="/{{.INDEX}}" '{ s = "/" $0 } substr(s, length(s) - length(i) + 1) != i'{{end}} + cut -f1 {{.RUN}}/bucket.txt | grep -vE '^\.state/'{{if .OWN}} | grep -vxF -f <(printf '%s\n' + {{.OWN}}){{end}} + {{if .INDEX}}| awk -v i="/{{.INDEX}}" '{ s = "/" $0 } substr(s, length(s) - length(i) + 1) != + i'{{end}} | LC_ALL=C comm -23 - {{.RUN}}/index-keys.txt | LC_ALL=C comm -23 - {{.RUN}}/paths.txt > {{.RUN}}/orphans.txt - {task: delete, vars: {D: '{{.RUN}}/orphans.txt'}} @@ -906,7 +988,10 @@ tasks: # ---- after the bytes moved ---- index: - desc: 'Hook. With INDEX set: redraw the directory pages this run changed and publish them; INDEXED, the stamp and the page state the pages last showed, moves forward only after they landed. A mirror with a landing page of its own replaces it' + desc: >- + Hook. With INDEX set: redraw the directory pages this run changed and publish them; INDEXED, + the stamp and the page state the pages last showed, moves forward only after they landed. A + mirror with a landing page of its own replaces it status: ['test -z "{{.INDEX}}"'] set: [pipefail] vars: @@ -915,14 +1000,17 @@ tasks: # With no INDEXED, there is no page. Then index makes a page for each directory one # time (for the 27k directories of ctan, this is approximately ten minutes). After # that, the file is there. - - {task: pull, vars: {KEY: '{{.INDEXED}}', OUT: '{{.RUN}}/indexed.txt', MISSING: ': > {{.RUN}}/indexed.txt'}} + - task: pull + vars: {KEY: '{{.INDEXED}}', OUT: '{{.RUN}}/indexed.txt', MISSING: ': > {{.RUN}}/indexed.txt'} - {task: pages} - {task: index-same} # At this point, staging holds only pages, because checkpoint removed the files. The # pages stay for smoke. The split verb of the next run removes them. The verb does # not append them to publish.txt, because smoke compares the size of that sample # with upstream, which has no pages. - - test ! -s {{.RUN}}/index-make.txt || aws s3 cp {{.AWS_FLAGS}} --recursive --content-type 'text/html; charset=utf-8' {{.STAGING}}/ {{.S3}}/ + - >- + test ! -s {{.RUN}}/index-make.txt || + aws s3 cp {{.AWS_FLAGS}} --recursive --content-type 'text/html; charset=utf-8' {{.STAGING}}/ {{.S3}}/ # One upload for each depth, because no tree can hold a/b and a/b/ together. A # slashless key has no suffix that the CLI can use. Thus, the command gives the # type. Without it, the CLI sends binary/octet-stream, and a browser downloads the @@ -940,7 +1028,9 @@ tasks: # Not internal: the offline check of lib starts it with its name. index-copy: - desc: Copy the index file of each directory in RUN/index-real.txt to the page key of that directory, in the bucket + desc: >- + Copy the index file of each directory in RUN/index-real.txt to the page key of that directory, + in the bucket cmds: # The page key of a directory with an index file (INDEX_FILES) is one server-side # copy of that file. COPY keeps its Content-Type, and the copy has the same bytes @@ -952,17 +1042,24 @@ tasks: # not use AWS_FLAGS. - >- while IFS="$(printf '\t')" read -r s k; do - aws s3api copy-object --bucket {{.BUCKET}} --copy-source "{{.BUCKET}}/$s" --key "$k" --metadata-directive COPY > /dev/null; + aws s3api copy-object --bucket {{.BUCKET}} --copy-source "{{.BUCKET}}/$s" --key "$k" + --metadata-directive COPY > /dev/null; echo "copy: $s to $k"; done < {{.RUN}}/index-real.txt # Not internal: the offline check of lib starts it with its name. index-same: - desc: When the stamp did not agree, read the ETag of each key in the bucket. Remove from staging and from RUN/index-make.txt each directory whose staged keys have the same MD5; RUN/index-same.txt gets these directories + desc: >- + When the stamp did not agree, read the ETag of each key in the bucket. Remove from staging and + from RUN/index-make.txt each directory whose staged keys have the same MD5; RUN/index-same.txt + gets these directories # When the stamp agrees, pages makes only the pages of the changed directories, and the # listing costs more Class A operations than it saves. The listing gives 1,000 keys # for each request. For a bucket with 571k keys, this is 572 operations, against two # for each page. With no INDEXED, the bucket has no page. - status: ['test ! -s {{.RUN}}/index-make.txt || test ! -s {{.RUN}}/indexed.txt || head -1 {{.RUN}}/indexed.txt | cmp -s - {{.RUN}}/index-stamp.txt'] + status: + - >- + test ! -s {{.RUN}}/index-make.txt || test ! -s {{.RUN}}/indexed.txt || + head -1 {{.RUN}}/indexed.txt | cmp -s - {{.RUN}}/index-stamp.txt set: [pipefail] vars: SLASH: '{{.RUN}}/slash' @@ -971,24 +1068,33 @@ tasks: # multipart upload is different, and such a page goes up again. If the listing # fails, each page goes up. - >- - aws s3api list-objects-v2 --bucket {{.BUCKET}} --query 'Contents[].[Key,ETag]' --output text > {{.RUN}}/bucket-etags.txt || + aws s3api list-objects-v2 --bucket {{.BUCKET}} --query 'Contents[].[Key,ETag]' --output text > + {{.RUN}}/bucket-etags.txt || { echo "index-same: the listing failed; index sends each page"; : > {{.RUN}}/bucket-etags.txt; } # Each line of page-md5.txt is "md5 TAB p|s TAB key": p for a page key in staging, s # for a slashless key in a SLASH tree. - >- { (cd {{.STAGING}} && find . -type f -exec md5sum {} +) | sed 's# \./#\tp\t#'; - for d in {{.SLASH}}/*/; do test -d "$d" || continue; (cd "$d" && find . -type f -exec md5sum {} +) | sed 's# \./#\ts\t#'; done; } + for d in {{.SLASH}}/*/; do test -d "$d" || continue; (cd "$d" && find . -type f -exec md5sum {} +) + | sed 's# \./#\ts\t#'; done; } > {{.RUN}}/page-md5.txt # SAME_PAGES writes the files of the same directories. xargs removes them. - >- - awk -F'\t' -v E="{{.RUN}}/bucket-etags.txt" -v H="{{.RUN}}/page-md5.txt" -v I="{{.INDEX}}" -v S="{{.STAGING}}/" -v R="{{.SLASH}}/" + awk -F'\t' -v E="{{.RUN}}/bucket-etags.txt" -v H="{{.RUN}}/page-md5.txt" -v I="{{.INDEX}}" -v + S="{{.STAGING}}/" -v R="{{.SLASH}}/" -v Y="{{.RUN}}/index-same.txt" -v M="{{.RUN}}/index-make.new" '{{.SAME_PAGES}}' {{.RUN}}/bucket-etags.txt {{.RUN}}/page-md5.txt {{.RUN}}/index-make.txt | tr '\n' '\0' | xargs -0r rm -f - - touch {{.RUN}}/index-same.txt {{.RUN}}/index-make.new && mv {{.RUN}}/index-make.new {{.RUN}}/index-make.txt + - >- + touch {{.RUN}}/index-same.txt {{.RUN}}/index-make.new && + mv {{.RUN}}/index-make.new {{.RUN}}/index-make.txt pages: - desc: RUN/indexed.txt against the page state (RUN/applied.txt and RUN/upstream-dirs.txt) -> a page under both of its keys for every directory a changed line touches, or for every directory when the stamp differs; RUN/index-gone.txt for the keys of directories that upstream removed; RUN/index-real.txt for the directories with an index file; RUN/indexed.new for INDEXED + desc: >- + RUN/indexed.txt against the page state (RUN/applied.txt and RUN/upstream-dirs.txt) -> a page + under both of its keys for every directory a changed line touches, or for every directory when + the stamp differs; RUN/index-gone.txt for the keys of directories that upstream removed; + RUN/index-real.txt for the directories with an index file; RUN/indexed.new for INDEXED set: [pipefail] env: &pagetext INDEX_FILES: '{{.INDEX_FILES}}' @@ -1014,8 +1120,12 @@ tasks: # environment, because awk -v changes each backslash escape. - >- awk 'BEGIN { g = ENVIRON["PAGE_FOOT"]; gsub(/%%/, "", g) - if (g !~ /^[^%]*(%s[^%]*)?$/) { print "pages: PAGE_FOOT can have one %s, and %%, not " ENVIRON["PAGE_FOOT"] > "/dev/stderr"; exit 1 } }' - - 'mkdir -p {{.STAGING}} {{.SLASH}} && find {{.STAGING}} {{.SLASH}} -mindepth 1 -delete && : > {{.RUN}}/index-real.txt' + if (g !~ /^[^%]*(%s[^%]*)?$/) { + print "pages: PAGE_FOOT can have one %s, and %%, not " ENVIRON["PAGE_FOOT"] > "/dev/stderr" + exit 1 } }' + - >- + mkdir -p {{.STAGING}} {{.SLASH}} && find {{.STAGING}} {{.SLASH}} -mindepth 1 -delete && + : > {{.RUN}}/index-real.txt # A task does not see the call vars of the task that starts it. Thus, pages gives its # steps the vars that they read, with the anchor pages. In a run, these are the values # that pages has. An offline check gives different values. @@ -1033,7 +1143,9 @@ tasks: - {task: pages-draw, vars: *pages} pages-select: - desc: The page state, the stamp and RUN/indexed.new; then RUN/index-gone.txt, the keys to remove, and RUN/index-make.txt, the directories to draw + desc: >- + The page state, the stamp and RUN/indexed.new; then RUN/index-gone.txt, the keys to remove, + and RUN/index-make.txt, the directories to draw internal: true set: [pipefail] env: *pagetext @@ -1048,8 +1160,10 @@ tasks: # a page. INDEXED starts with the stamp of the pages that it records. If INDEXED has # no stamp line, the stamps do not agree. - >- - printf '%s\n' {{shellQuote .PAGE_FORMAT}} {{shellQuote .INDEX}} "$INDEX_FILES" "$PAGE_FOOT" "$UPSTREAM" "$CONTACT" - {{shellQuote .HOST}} | sha256sum | awk '{ print "#pages " substr($1, 1, 16) }' > {{.RUN}}/index-stamp.txt + printf '%s\n' {{shellQuote .PAGE_FORMAT}} {{shellQuote .INDEX}} "$INDEX_FILES" "$PAGE_FOOT" + "$UPSTREAM" "$CONTACT" + {{shellQuote .HOST}} | sha256sum | awk '{ print "#pages " substr($1, 1, 16) }' > + {{.RUN}}/index-stamp.txt - awk 'NR == 1 && /^#pages / { next } { print }' {{.RUN}}/indexed.txt > {{.RUN}}/index-old.txt # A page changes when a file in its directory, or a directory below it, is added, # changed or removed. Thus, the verb makes the page of each ancestor of each changed @@ -1063,7 +1177,8 @@ tasks: # the two files. Then make a parent page again only for a new child or a removed # child. - >- - { LC_ALL=C comm -3 {{.RUN}}/index-old.txt {{.RUN}}/page-state.txt | awk -F'\t' '{ print ($1 == "" ? $2 : $1) }'; + { LC_ALL=C comm -3 {{.RUN}}/index-old.txt {{.RUN}}/page-state.txt | awk -F'\t' '{ print ($1 == "" ? + $2 : $1) }'; head -1 {{.RUN}}/indexed.txt | cmp -s - {{.RUN}}/index-stamp.txt || cat {{.RUN}}/page-state.txt; } | {{.ANCESTORS}} | LC_ALL=C sort > {{.RUN}}/index-dirty.txt - '{{.ANCESTORS}} {{.RUN}}/page-state.txt | LC_ALL=C sort > {{.RUN}}/index-dirs.txt' @@ -1077,7 +1192,8 @@ tasks: - >- LC_ALL=C comm -23 {{.RUN}}/index-dirty.txt {{.RUN}}/index-dirs.txt | awk -v i="{{.INDEX}}" '{ print $0 i } $0 != "" { print substr($0, 1, length($0) - 1) }' - | awk -F'\t' 'FILENAME == ARGV[1] { f[$1] = 1; next } !($0 in f)' {{.RUN}}/page-state.txt - > {{.RUN}}/index-gone.txt + | awk -F'\t' 'FILENAME == ARGV[1] { f[$1] = 1; next } !($0 in f)' {{.RUN}}/page-state.txt - > + {{.RUN}}/index-gone.txt - LC_ALL=C comm -12 {{.RUN}}/index-dirty.txt {{.RUN}}/index-dirs.txt > {{.RUN}}/index-make.txt pages-draw: @@ -1086,7 +1202,9 @@ tasks: set: [pipefail] env: *pagetext cmds: - - test ! -s {{.RUN}}/index-make.txt || awk -v s="{{.STAGING}}/" '{ print s $0 }' {{.RUN}}/index-make.txt | tr '\n' '\0' | xargs -0 mkdir -p + - >- + test ! -s {{.RUN}}/index-make.txt || + awk -v s="{{.STAGING}}/" '{ print s $0 }' {{.RUN}}/index-make.txt | tr '\n' '\0' | xargs -0 mkdir -p # SLASH// holds the slashless copies of the pages at that depth. All files in # one tree have the same number of components. Thus, no file is the parent directory # of a different file, and the upload can send the full tree. @@ -1110,12 +1228,19 @@ tasks: - >- test ! -s {{.RUN}}/index-make.txt || awk -F'\t' -v OFS='\t' '{{.PAGE_ENTRIES}}' {{.RUN}}/index-make.txt {{.RUN}}/page-state.txt - | LC_ALL=C sort -t "$(printf '\t')" -k1,1 -k2,2 -k3,3V | awk -F'\t' -v S="{{.STAGING}}/" -v R="{{.SLASH}}/" -v I="{{.INDEX}}" - -v F="$PAGE_FOOT" -v N="$INDEX_FILES" -v X="{{.RUN}}/index-real.txt" -v U="$UPSTREAM" -v C="$CONTACT" -v H="{{.HOST}}" + | LC_ALL=C sort -t "$(printf '\t')" -k1,1 -k2,2 -k3,3V | awk -F'\t' -v S="{{.STAGING}}/" -v + R="{{.SLASH}}/" -v I="{{.INDEX}}" + -v F="$PAGE_FOOT" -v N="$INDEX_FILES" -v X="{{.RUN}}/index-real.txt" -v U="$UPSTREAM" -v + C="$CONTACT" -v H="{{.HOST}}" -v css={{shellQuote .PAGE_CSS}} '{{.PAGE_HTML}}' smoke: - desc: Read a key of every type this run published back through the domain, size against the listing and type against the label; with TL set the tlpdb sha512 against the verified copy; with INDEX set one redrawn page under both keys; with CANARY set the canary, as a Perl client, against the bucket's copy; no Content-Encoding on a .tar.gz; warn of paths that never land; fresh; then the mirror's own checks, smoke-mirror + desc: >- + Read a key of every type this run published back through the domain, size against the listing + and type against the label; with TL set the tlpdb sha512 against the verified copy; with INDEX + set one redrawn page under both keys; with CANARY set the canary, as a Perl client, against + the bucket's copy; no Content-Encoding on a .tar.gz; warn of paths that never land; fresh; + then the mirror's own checks, smoke-mirror cmds: # A task does not see the call vars of the task that starts it. Thus, smoke gives its # steps the vars that they read, with the anchor smoke. It gives fresh and @@ -1139,7 +1264,10 @@ tasks: - {task: smoke-mirror, vars: {RUN: '{{.RUN}}', STAGING: '{{.STAGING}}', URL: '{{.URL}}'}} smoke-sample: - desc: The first key of each type that label gave this run, and the FRESH_KEY file when the state has it, read back through the domain; size against the listing, and with http the type against the label + desc: >- + The first key of each type that label gave this run, and the FRESH_KEY file when the state has + it, read back through the domain; size against the listing, and with http the type against the + label internal: true env: {FRESH_KEY: '{{.FRESH_KEY}}'} cmds: @@ -1158,7 +1286,8 @@ tasks: # # A key in the sample can contain a space. All other keys can also contain a space. - >- - { {{if .FRESH_KEY}}awk -F'\t' -v k="$FRESH_KEY" '$1 == k { print $1 "\t" }' {{.RUN}}/applied.txt 2>/dev/null || true;{{end}} + { {{if .FRESH_KEY}}awk -F'\t' -v k="$FRESH_KEY" '$1 == k { print $1 "\t" }' {{.RUN}}/applied.txt + 2>/dev/null || true;{{end}} awk -F'\t' '!seen[$2]++' {{.RUN}}/labeled.txt 2>/dev/null; } | while IFS="$(printf '\t')" read -r k t; do u=$(printf '%s\n' "$k" | {{.ENCODE}} | cut -f2); @@ -1166,11 +1295,14 @@ tasks: h=$({{.HEADERS}} "{{.URL}}/$u"); got=$(printf '%s\n' "$h" | {{.SIZE}}); ct=$(printf '%s\n' "$h" | {{.CTYPE}}); - echo "$k: $got bytes (listing: $want), served as ${ct:-no type} (label: ${t:-none})"; test "$got" = "$want" || exit 1; + echo "$k: $got bytes (listing: $want), served as ${ct:-no type} (label: ${t:-none})"; test "$got" = + "$want" || exit 1; case "{{.URL}}" in http*) test -z "$t" || test "$ct" = "$t" || exit 1 ;; esac; done smoke-tlpdb: - desc: With TL set and a tlpdb that prepare verified, the tlpdb sha512 read back through the domain against the verified copy + desc: >- + With TL set and a tlpdb that prepare verified, the tlpdb sha512 read back through the domain + against the verified copy internal: true cmds: # The tlpdb key is in the decision batch, the last batch. Thus, a run that stopped @@ -1179,10 +1311,13 @@ tasks: - >- {{if .TL}}test ! -f {{.RUN}}/tl/tlpkg/texlive.tlpdb.sha512 || test -f {{.RUN}}/chain || - {{.CURL}} {{.URL}}/{{.TL}}/tlpkg/texlive.tlpdb.sha512 | cmp - {{.RUN}}/tl/tlpkg/texlive.tlpdb.sha512{{else}}echo "no TL subtree: no tlpdb to read back"{{end}} + {{.CURL}} {{.URL}}/{{.TL}}/tlpkg/texlive.tlpdb.sha512 | cmp - + {{.RUN}}/tl/tlpkg/texlive.tlpdb.sha512{{else}}echo "no TL subtree: no tlpdb to read back"{{end}} smoke-page: - desc: With INDEX set, the last directory of RUN/index-make.txt read back through the domain at its page key, and with http at its slashless key + desc: >- + With INDEX set, the last directory of RUN/index-make.txt read back through the domain at its + page key, and with http at its slashless key internal: true cmds: # This command reads one of the pages that this run made, at its two keys, with each @@ -1205,16 +1340,21 @@ tasks: echo "$(printf '%s\n' "$h" | {{.SIZE}}) bytes, type $(printf '%s\n' "$h" | {{.CTYPE}})"; }; s=$(awk -F'\t' -v k="${d}{{.INDEX}}" '$2 == k { print $1 }' {{.RUN}}/index-real.txt); if test -n "$s"; then want=$(r "$s"); - else want=$(wc -c < "{{.STAGING}}/${d}{{.INDEX}}" | tr -d ' '); test -n "$want"; want="$want bytes"; fi; + else want=$(wc -c < "{{.STAGING}}/${d}{{.INDEX}}" | tr -d ' '); test -n "$want"; want="$want + bytes"; fi; got=$(r "${d}{{.INDEX}}"); test -n "$s" || got=${got%%,*}; echo "${d}{{.INDEX}}: $got (${s:-page}: $want)"; test "$got" = "$want"; case "{{.URL}}" in http*) test -z "$d" || { p=${d%/}; - want=$(wc -c < "{{.RUN}}/slash/$(printf '%s\n' "$p" | awk -F/ '{ print NF }')/$p" | tr -d ' '); test -n "$want"; - got=$(r "$p"); got=${got%%,*}; echo "$p: $got (listing: $want bytes)"; test "$got" = "$want bytes"; } ;; esac; + want=$(wc -c < "{{.RUN}}/slash/$(printf '%s\n' "$p" | awk -F/ '{ print NF }')/$p" | tr -d ' '); + test -n "$want"; + got=$(r "$p"); got=${got%%,*}; echo "$p: $got (listing: $want bytes)"; test "$got" = "$want bytes"; + } ;; esac; }{{else}}echo "no INDEX: no directory pages to read back"{{end}} smoke-canary: - desc: With CANARY set, in the state and over http, the canary read as a Perl client against the bucket's copy + desc: >- + With CANARY set, in the state and over http, the canary read as a Perl client against the + bucket's copy internal: true env: {CANARY: '{{.CANARY}}'} cmds: @@ -1232,12 +1372,14 @@ tasks: # The command does not do this check with file://, which has no bucket to read. It # also does not do it before the state holds the path. - >- - {{if .CANARY}}h="$CANARY"; n=$(awk -F'\t' -v k="$h" '$1 == k { print $2 }' {{.RUN}}/applied.txt 2>/dev/null); + {{if .CANARY}}h="$CANARY"; n=$(awk -F'\t' -v k="$h" '$1 == k { print $2 }' {{.RUN}}/applied.txt + 2>/dev/null); case "{{.URL}}" in http*) ;; *) n= ;; esac; if test -z "$n"; then echo "canary skipped ($h is not in the state, or the URL is not http)"; else aws s3 cp {{.AWS_FLAGS}} "{{.S3}}/$h" {{.RUN}}/canary.html && {{.CURL}} -A "libwww-perl/6.67" "{{.URL}}/$h" | cmp - {{.RUN}}/canary.html && - echo "$h: $n bytes, served to a libwww-perl client as the bucket holds them"; fi{{else}}echo "no CANARY: the edge's HTML rewriters and client filter go unchecked"{{end}} + echo "$h: $n bytes, served to a libwww-perl client as the bucket holds them"; fi{{else}}echo "no + CANARY: the edge's HTML rewriters and client filter go unchecked"{{end}} smoke-encoding: desc: Over http, the first non-empty .tar.gz of the state read back with no Content-Encoding @@ -1256,13 +1398,17 @@ tasks: - >- g=$(awk -F'\t' '$1 ~ /\.tar\.gz$/ && $2 > 0 { print $1; exit }' {{.RUN}}/applied.txt 2>/dev/null); case "{{.URL}}" in http*) ;; *) g= ;; esac; - if test -z "$g"; then echo "content-encoding check skipped (no non-empty .tar.gz in the state, or the URL is not http)"; else + if test -z "$g"; then echo "content-encoding check skipped (no non-empty .tar.gz in the state, or + the URL is not http)"; else u=$(printf '%s\n' "$g" | {{.ENCODE}} | cut -f2) && h=$({{.HEADERS}} "{{.URL}}/$u") && - if printf '%s' "$h" | grep -qi '^content-encoding:'; then echo "$g: served with a content-encoding" >&2; exit 1; fi && + if printf '%s' "$h" | grep -qi '^content-encoding:'; then echo "$g: served with a content-encoding" + >&2; exit 1; fi && echo "$g: served with no content-encoding"; fi smoke-unlanded: - desc: RUN/unlanded.txt, the lines of the delta that the state lacks after the last batch; the first 20 as warnings on the run + desc: >- + RUN/unlanded.txt, the lines of the delta that the state lacks after the last batch; the first + 20 as warnings on the run internal: true cmds: # A path in the listing of upstream that rsync cannot send (exit 23, which retry @@ -1272,11 +1418,16 @@ tasks: # first 20 is a warning on the Actions run. - >- if test -f {{.RUN}}/changed.txt && test ! -f {{.RUN}}/chain; - then LC_ALL=C comm -23 {{.RUN}}/changed.txt {{.RUN}}/applied.txt > {{.RUN}}/unlanded.txt; else : > {{.RUN}}/unlanded.txt; fi; - cut -f1 {{.RUN}}/unlanded.txt | head -20 | while IFS= read -r p; do echo "::warning::listed upstream, never fetched: $p"; done + then LC_ALL=C comm -23 {{.RUN}}/changed.txt {{.RUN}}/applied.txt > {{.RUN}}/unlanded.txt; else : > + {{.RUN}}/unlanded.txt; fi; + cut -f1 {{.RUN}}/unlanded.txt | head -20 | while IFS= read -r p; do echo "::warning::listed + upstream, never fetched: $p"; done fresh: - desc: With FRESH_KEY set and in the state, read it through the domain and fail the run when the upstream clock in it is FRESH_HOURS (24) old; a source that stalls otherwise leaves every run green + desc: >- + With FRESH_KEY set and in the state, read it through the domain and fail the run when the + upstream clock in it is FRESH_HOURS (24) old; a source that stalls otherwise leaves every run + green env: {FRESH_KEY: '{{.FRESH_KEY}}'} cmds: # The run moved each byte that it got. Thus, only the clock that upstream wrote can @@ -1286,16 +1437,21 @@ tasks: {{if .FRESH_KEY}}k="$FRESH_KEY"; if ! awk -F'\t' -v k="$k" '$1 == k { f = 1 } END { exit !f }' {{.RUN}}/applied.txt 2>/dev/null; then echo "freshness skipped ($k is not in the state)"; else - h={{shellQuote .FRESH_H}}; case "$h" in {{.NOT_WHOLE}}) echo "fresh: FRESH_HOURS is a whole number, not $h" >&2; exit 1 ;; esac; + h={{shellQuote .FRESH_H}}; case "$h" in {{.NOT_WHOLE}}) echo "fresh: FRESH_HOURS is a whole number, + not $h" >&2; exit 1 ;; esac; u=$(printf '%s\n' "$k" | {{.ENCODE}} | cut -f2); t=$({{.CURL}} "{{.URL}}/$u" | {{.STAMP}}); case "$t" in *-*) t=$(date -u -d "$t" +%s) ;; esac; test -n "$t"; a=$(( $(date -u +%s) - t )); echo "$a" > {{.RUN}}/fresh.txt; echo "$k: upstream's clock $((a / 3600)) h $((a % 3600 / 60)) min ago, limit {{.FRESH_H}} h"; - test "$a" -le $(( {{.FRESH_H}} * 3600 )) || { echo "::error::$k is $((a / 3600)) h old, past FRESH_HOURS {{.FRESH_H}}: upstream or SOURCE has stopped moving"; exit 1; }; fi{{else}}echo "no FRESH_KEY: a source that stops moving goes unnoticed"{{end}} + test "$a" -le $(( {{.FRESH_H}} * 3600 )) || { echo "::error::$k is $((a / 3600)) h old, past + FRESH_HOURS {{.FRESH_H}}: upstream or SOURCE has stopped moving"; exit 1; }; fi{{else}}echo "no + FRESH_KEY: a source that stops moving goes unnoticed"{{end}} smoke-mirror: - desc: A hook, empty here; a mirror with more to read back than the sample replaces it, and smoke runs it last + desc: >- + A hook, empty here; a mirror with more to read back than the sample replaces it, and smoke + runs it last cmds: [] report-engine: @@ -1306,29 +1462,55 @@ tasks: - | n() { test -f "$1" && wc -l < "$1" || echo 0; } gb() { test -f "$1" && awk -F'\t' '{ s += $2 } END { printf "%.2f", s / 1e9 }' "$1" || echo 0.00; } - out="${GITHUB_STEP_SUMMARY:-/dev/stdout}" + r={{.RUN}}; u={{.URL}}; out="${GITHUB_STEP_SUMMARY:-/dev/stdout}" + batches=$(ls $r/batch-*.txt 2>/dev/null | wc -l | awk -v m={{.BATCHES_MAX}} \ + '{ print $1 " batches" ($1 > m ? ", " $1 - m " left for the next run" : "") }') + deleted=$(n $r/deleted.txt); orphaned=$(n $r/orphans.txt) + uploaded=$(awk '/^upload:/ { n++ } END { print n + 0 }' $r/publish.txt 2>/dev/null || echo 0) + types=$(cut -f2 $r/labeled.txt 2>/dev/null | sort -u | wc -l | tr -d ' ') + storage="not measured this run" + test ! -f $r/bucket.txt || storage="$(gb $r/bucket.txt) GB in $(n $r/bucket.txt) objects (reconciled)" cat >> "$out" </dev/null | wc -l | awk -v m={{.BATCHES_MAX}} '{ print $1 " batches" ($1 > m ? ", " $1 - m " left for the next run" : "") }'); $(n {{.RUN}}/deleted.txt) deleted, $(n {{.RUN}}/orphans.txt) orphaned | - | Published to R2 | $(awk '/^upload:/ { n++ } END { print n + 0 }' {{.RUN}}/publish.txt 2>/dev/null || echo 0) uploaded, $(cut -f2 {{.RUN}}/labeled.txt 2>/dev/null | sort -u | wc -l | tr -d ' ') types | - | State | $(n {{.RUN}}/applied.txt) lines in {{.STATE}} | - | Storage | $(test -f {{.RUN}}/bucket.txt && echo "$(gb {{.RUN}}/bucket.txt) GB in $(n {{.RUN}}/bucket.txt) objects (reconciled)" || echo "not measured this run"){{if .CEILING_GB}} of the {{.CEILING_GB}} GB ceiling{{end}} | + | Mirror | $(n $r/upstream.txt) objects, $(gb $r/upstream.txt) GB upstream, at [$u/]($u/) | + | Delta | $(n $r/changed.txt) new or changed in $batches; $deleted deleted, $orphaned orphaned | + | Published to R2 | $uploaded uploaded, $types types | + | State | $(n $r/applied.txt) lines in {{.STATE}} | + | Storage | $storage{{if .CEILING_GB}} of the {{.CEILING_GB}} GB ceiling{{end}} | EOF # prepare made RUN/tl. Thus, the delta had a path in the signed subtree. - test ! -d {{.RUN}}/tl || echo "| Signature | $(test -f {{.RUN}}/tl/tlpkg/texlive.tlpdb && echo "tlpdb signed by {{.TL_KEY}}; $(n {{.RUN}}/tl/missing.txt) named containers missing from the bucket" || echo "the tlpdb did not verify") |" >> "$out" - # With INDEX set, the pages that index made and the keys that it removed. - test -z "{{.INDEX}}" || echo "| Directory pages | $(n {{.RUN}}/index-make.txt) redrawn and sent, $(n {{.RUN}}/index-same.txt) redrawn and not sent (the bucket had the same bytes), $(n {{.RUN}}/index-real.txt) with a copy of an index file as the page key, $(n {{.RUN}}/index-gone.txt) keys removed |" >> "$out" + if test -d $r/tl; then + s="the tlpdb did not verify" + test ! -f $r/tl/tlpkg/texlive.tlpdb || + s="tlpdb signed by {{.TL_KEY}}; $(n $r/tl/missing.txt) named containers missing from the bucket" + echo "| Signature | $s |" >> "$out" + fi + # With INDEX set, the pages that index made and the keys that it removed. echo puts one + # space between two arguments. + test -z "{{.INDEX}}" || echo "| Directory pages | $(n $r/index-make.txt) redrawn and sent," \ + "$(n $r/index-same.txt) redrawn and not sent (the bucket had the same bytes)," \ + "$(n $r/index-real.txt) with a copy of an index file as the page key," \ + "$(n $r/index-gone.txt) keys removed |" >> "$out" # Rows that show only when they have data. - test ! -s {{.RUN}}/unlanded.txt || echo "| Never landed | $(wc -l < {{.RUN}}/unlanded.txt | tr -d ' ') listed upstream that rsync could not send, fetched again every run; the first are warnings on the run |" >> "$out" - test ! -f {{.RUN}}/fresh.txt || echo "| Upstream clock | $(awk '{ printf "%d h %d min", $1 / 3600, $1 % 3600 / 60 }' {{.RUN}}/fresh.txt) old in $FRESH_KEY, limit {{.FRESH_H}} h |" >> "$out" + test ! -s $r/unlanded.txt || echo "| Never landed | $(wc -l < $r/unlanded.txt | tr -d ' ') listed" \ + "upstream that rsync could not send, fetched again every run; the first are warnings on" \ + "the run |" >> "$out" + test ! -f $r/fresh.txt || echo "| Upstream clock |" \ + "$(awk '{ printf "%d h %d min", $1 / 3600, $1 % 3600 / 60 }' $r/fresh.txt) old in $FRESH_KEY," \ + "limit {{.FRESH_H}} h |" >> "$out" retry: - desc: Run CMD, retrying rsync's transport exit codes (5 10 12 30 35) five times with backoff and jitter; 23 (a path it could not read, a dangling symlink under -L) and 24 (file vanished mid-transfer) are successes + desc: >- + Run CMD, retrying rsync's transport exit codes (5 10 12 30 35) five times with backoff and + jitter; 23 (a path it could not read, a dangling symlink under -L) and 24 (file vanished + mid-transfer) are successes cmds: - | for i in 1 2 3 4 5; do - rc=0; ( {{.CMD}} ) || rc=$? # subshell: an exit in CMD cannot stop the loop. ||: the errexit of Task cannot stop it. + # An exit in CMD stops only the subshell, not the loop. Because of ||, the errexit + # of Task does not stop the loop. + rc=0; ( {{.CMD}} ) || rc=$? case $rc in 0|23|24) exit 0;; 5|10|12|30|35) ;; *) exit $rc;; esac test $i -lt 5 || break - s=$(({{.RETRY_S}} * 2 ** i + RANDOM % (2 * {{.RETRY_S}} + 1))); echo "rsync exit $rc, retry $i in ${s}s" >&2; sleep $s + s=$(({{.RETRY_S}} * 2 ** i + RANDOM % (2 * {{.RETRY_S}} + 1))) + echo "rsync exit $rc, retry $i in ${s}s" >&2; sleep $s done; exit $rc diff --git a/examples/rsync/render.txt b/examples/rsync/render.txt index 604394a..207818c 100644 --- a/examples/rsync/render.txt +++ b/examples/rsync/render.txt @@ -23,10 +23,13 @@ echo "due: RECONCILE=$r RECONCILE_HOURS=$h, last reconcile $ago: $(test -f /work task: [list] rm -f /work/examples/rsync/.run/publish.txt /work/examples/rsync/.run/labeled.txt /work/examples/rsync/.run/orphans.txt /work/examples/rsync/.run/unlanded.txt /work/examples/rsync/.run/fresh.txt /work/examples/rsync/.run/index-same.txt task: [list] mkdir -p /work/examples/rsync/.run /work/examples/rsync/staging task: [retry] for i in 1 2 3 4 5; do - rc=0; ( TZ=UTC rsync --timeout=300 --contimeout=60 --no-h -8 -rL --list-only rsync://rsync.example.org/pub/ > /work/examples/rsync/.run/listing.txt ) || rc=$? # subshell: an exit in CMD cannot stop the loop. ||: the errexit of Task cannot stop it. + # An exit in CMD stops only the subshell, not the loop. Because of ||, the errexit + # of Task does not stop the loop. + rc=0; ( TZ=UTC rsync --timeout=300 --contimeout=60 --no-h -8 -rL --list-only rsync://rsync.example.org/pub/ > /work/examples/rsync/.run/listing.txt ) || rc=$? case $rc in 0|23|24) exit 0;; 5|10|12|30|35) ;; *) exit $rc;; esac test $i -lt 5 || break - s=$((15 * 2 ** i + RANDOM % (2 * 15 + 1))); echo "rsync exit $rc, retry $i in ${s}s" >&2; sleep $s + s=$((15 * 2 ** i + RANDOM % (2 * 15 + 1))) + echo "rsync exit $rc, retry $i in ${s}s" >&2; sleep $s done; exit $rc task: [normalise] awk '/^-/ && match($0, / [0-9][0-9][0-9][0-9]\/[0-9][0-9]\/[0-9][0-9] [0-9][0-9]:[0-9][0-9]:[0-9][0-9] /) { @@ -59,7 +62,8 @@ task: [split] awk -F'\t' '{ s += $2 } END { printf "upstream: %d objects, %.2f G task: [split] mkdir -p /work/examples/rsync/staging && find /work/examples/rsync/staging -mindepth 1 -delete task: [split] df -Pk /work/examples/rsync/staging task: [split] awk -F'\t' -v kb="$(df -Pk /work/examples/rsync/staging | awk 'NR == 2 { print $4 }')" \ - '$2 / 1024 > kb - 1048576 { print "no room for " $1 " (" $2 " bytes)"; bad = 1 } END { exit bad }' \ + '$2 / 1024 > kb - 1048576 { print "no room for " $1 " (" $2 " bytes)"; bad = 1 } + END { exit bad }' \ /work/examples/rsync/.run/changed.txt task: [split] rm -f /work/examples/rsync/.run/batch-*.txt task: [split] awk -F'\t' -v OFS='\t' -v CAP=$((4 * 1000000000)) -v DIR=/work/examples/rsync/.run -v TL="" 'index($1, TL "/tlpkg/") == 1 || $1 !~ /\// { dec[++nd] = $0; next } @@ -73,10 +77,13 @@ function flush( i, f) { if (!nb) return; f = name(++n) task: [prepare] mkdir -p /work/examples/rsync/.run/tl task: [prepare] printf '%s\n' tlpkg/texlive.tlpdb.xz tlpkg/texlive.tlpdb.sha512 tlpkg/texlive.tlpdb.sha512.asc tlpkg/gpg/pubring.gpg > /work/examples/rsync/.run/tl/files.txt task: [retry] for i in 1 2 3 4 5; do - rc=0; ( rsync --timeout=300 --contimeout=60 --no-h -Lt --files-from=/work/examples/rsync/.run/tl/files.txt rsync://rsync.example.org/pub// /work/examples/rsync/.run/tl/ ) || rc=$? # subshell: an exit in CMD cannot stop the loop. ||: the errexit of Task cannot stop it. + # An exit in CMD stops only the subshell, not the loop. Because of ||, the errexit + # of Task does not stop the loop. + rc=0; ( rsync --timeout=300 --contimeout=60 --no-h -Lt --files-from=/work/examples/rsync/.run/tl/files.txt rsync://rsync.example.org/pub// /work/examples/rsync/.run/tl/ ) || rc=$? case $rc in 0|23|24) exit 0;; 5|10|12|30|35) ;; *) exit $rc;; esac test $i -lt 5 || break - s=$((15 * 2 ** i + RANDOM % (2 * 15 + 1))); echo "rsync exit $rc, retry $i in ${s}s" >&2; sleep $s + s=$((15 * 2 ** i + RANDOM % (2 * 15 + 1))) + echo "rsync exit $rc, retry $i in ${s}s" >&2; sleep $s done; exit $rc task: [prepare] xz -dkf /work/examples/rsync/.run/tl/tlpkg/texlive.tlpdb.xz @@ -84,10 +91,13 @@ task: [prepare] cd /work/examples/rsync/.run/tl/tlpkg && shasum -a 512 -c texliv task: [fetch] find /work/examples/rsync/staging -mindepth 1 -delete task: [fetch] cut -f1 > /work/examples/rsync/.run/files.txt task: [retry] for i in 1 2 3 4 5; do - rc=0; ( rsync --timeout=300 --contimeout=60 --no-h -Lt --files-from=/work/examples/rsync/.run/files.txt --ignore-missing-args rsync://rsync.example.org/pub/ /work/examples/rsync/staging/ ) || rc=$? # subshell: an exit in CMD cannot stop the loop. ||: the errexit of Task cannot stop it. + # An exit in CMD stops only the subshell, not the loop. Because of ||, the errexit + # of Task does not stop the loop. + rc=0; ( rsync --timeout=300 --contimeout=60 --no-h -Lt --files-from=/work/examples/rsync/.run/files.txt --ignore-missing-args rsync://rsync.example.org/pub/ /work/examples/rsync/staging/ ) || rc=$? case $rc in 0|23|24) exit 0;; 5|10|12|30|35) ;; *) exit $rc;; esac test $i -lt 5 || break - s=$((15 * 2 ** i + RANDOM % (2 * 15 + 1))); echo "rsync exit $rc, retry $i in ${s}s" >&2; sleep $s + s=$((15 * 2 ** i + RANDOM % (2 * 15 + 1))) + echo "rsync exit $rc, retry $i in ${s}s" >&2; sleep $s done; exit $rc task: [verify-signed] cd /work/examples/rsync/staging && cut -f1 | grep -E '^/(tlpkg/texlive\.tlpdb|[^/]+)\.sha512$' | while IFS= read -r f; do (cd "$(dirname "$f")" && f=$(basename "$f") && shasum -a 512 -c "$f" && gpgv --status-fd 1 --keyring /work/examples/rsync/.run/tl/tlpkg/gpg/pubring.gpg "$f.asc" "$f" | awk -v k= '/^\[GNUPG:\] GOODSIG /{g=1} /^\[GNUPG:\] VALIDSIG / && $NF == k {v=1} END{exit !(g&&v)}') || exit 1; done @@ -96,7 +106,8 @@ task: [verify-containers] cd /work/examples/rsync/staging && { test "$(grep -c ' /^containerchecksum /{split($0, a, " "); emit(a[2], n)} /^doccontainerchecksum /{split($0, a, " "); emit(a[2], n ".doc")} /^srccontainerchecksum /{split($0, a, " "); emit(a[2], n ".source")} -function emit(sum, base, p, q) { p = TL "/archive/" base ".tar.xz"; q = TL "/archive/" base ".r" r ".tar.xz" +function emit(sum, base, p, q) { + p = TL "/archive/" base ".tar.xz"; q = TL "/archive/" base ".r" r ".tar.xz" if (p in want) { print sum " " p; delete want[p] } if (q in want) { print sum " " q; delete want[q] } } END { for (p in want) { print "no checksum in the tlpdb for " p > "/dev/stderr"; bad = 1 } exit bad }' /work/examples/rsync/.run/tl/tlpkg/texlive.tlpdb > /work/examples/rsync/.run/tl/sums.txt && shasum -a 512 -c --quiet /work/examples/rsync/.run/tl/sums.txt; }; } @@ -251,7 +262,8 @@ task: [push] aws s3 cp --no-progress --cli-connect-timeout 60 --cli-read-timeout task: [rebuild] mv /work/examples/rsync/.run/applied.new /work/examples/rsync/.run/applied.txt task: [rebuild] rm -f /work/examples/rsync/.run/rebuild-now task: [reconcile] cat /work/examples/rsync/.run/applied.txt /work/examples/rsync/.run/upstream-dirs.txt | awk -F'\t' '!("" in s) { s[""] = 1; print "" } - { n = split($1, c, "/"); d = ""; for (i = 1; i < n; i++) { d = d c[i] "/"; if (!(d in s)) { s[d] = 1; print d } } }' | awk '$0 != "" { print substr($0, 1, length($0) - 1) }' | LC_ALL=C sort > /work/examples/rsync/.run/index-keys.txt + { n = split($1, c, "/"); d = "" + for (i = 1; i < n; i++) { d = d c[i] "/"; if (!(d in s)) { s[d] = 1; print d } } }' | awk '$0 != "" { print substr($0, 1, length($0) - 1) }' | LC_ALL=C sort > /work/examples/rsync/.run/index-keys.txt task: [reconcile] cut -f1 /work/examples/rsync/.run/bucket.txt | grep -vE '^\.state/' | awk -v i="/example.katoptra.org.directory.index.html" '{ s = "/" $0 } substr(s, length(s) - length(i) + 1) != i' | LC_ALL=C comm -23 - /work/examples/rsync/.run/index-keys.txt | LC_ALL=C comm -23 - /work/examples/rsync/.run/paths.txt > /work/examples/rsync/.run/orphans.txt task: [remove] rm -f /work/examples/rsync/.run/del-*; split -l 1000 -a 4 /work/examples/rsync/.run/orphans.txt /work/examples/rsync/.run/del- task: [remove] for f in /work/examples/rsync/.run/del-*; do awk 'BEGIN { printf "{\"Objects\":[" } { gsub(/\\/, "\\\\"); gsub(/"/, "\\\"") @@ -273,15 +285,19 @@ case $rc in esac task: [pages] awk 'BEGIN { g = ENVIRON["PAGE_FOOT"]; gsub(/%%/, "", g) - if (g !~ /^[^%]*(%s[^%]*)?$/) { print "pages: PAGE_FOOT can have one %s, and %%, not " ENVIRON["PAGE_FOOT"] > "/dev/stderr"; exit 1 } }' + if (g !~ /^[^%]*(%s[^%]*)?$/) { + print "pages: PAGE_FOOT can have one %s, and %%, not " ENVIRON["PAGE_FOOT"] > "/dev/stderr" + exit 1 } }' task: [pages] mkdir -p /work/examples/rsync/staging /work/examples/rsync/.run/slash && find /work/examples/rsync/staging /work/examples/rsync/.run/slash -mindepth 1 -delete && : > /work/examples/rsync/.run/index-real.txt task: [pages-select] { cat /work/examples/rsync/.run/applied.txt; awk '{ print $0 "\t\t" }' /work/examples/rsync/.run/upstream-dirs.txt; } | LC_ALL=C sort > /work/examples/rsync/.run/page-state.txt task: [pages-select] printf '%s\n' 3 example.katoptra.org.directory.index.html "$INDEX_FILES" "$PAGE_FOOT" "$UPSTREAM" "$CONTACT" example.katoptra.org | sha256sum | awk '{ print "#pages " substr($1, 1, 16) }' > /work/examples/rsync/.run/index-stamp.txt task: [pages-select] awk 'NR == 1 && /^#pages / { next } { print }' /work/examples/rsync/.run/indexed.txt > /work/examples/rsync/.run/index-old.txt task: [pages-select] { LC_ALL=C comm -3 /work/examples/rsync/.run/index-old.txt /work/examples/rsync/.run/page-state.txt | awk -F'\t' '{ print ($1 == "" ? $2 : $1) }'; head -1 /work/examples/rsync/.run/indexed.txt | cmp -s - /work/examples/rsync/.run/index-stamp.txt || cat /work/examples/rsync/.run/page-state.txt; } | awk -F'\t' '!("" in s) { s[""] = 1; print "" } - { n = split($1, c, "/"); d = ""; for (i = 1; i < n; i++) { d = d c[i] "/"; if (!(d in s)) { s[d] = 1; print d } } }' | LC_ALL=C sort > /work/examples/rsync/.run/index-dirty.txt + { n = split($1, c, "/"); d = "" + for (i = 1; i < n; i++) { d = d c[i] "/"; if (!(d in s)) { s[d] = 1; print d } } }' | LC_ALL=C sort > /work/examples/rsync/.run/index-dirty.txt task: [pages-select] awk -F'\t' '!("" in s) { s[""] = 1; print "" } - { n = split($1, c, "/"); d = ""; for (i = 1; i < n; i++) { d = d c[i] "/"; if (!(d in s)) { s[d] = 1; print d } } }' /work/examples/rsync/.run/page-state.txt | LC_ALL=C sort > /work/examples/rsync/.run/index-dirs.txt + { n = split($1, c, "/"); d = "" + for (i = 1; i < n; i++) { d = d c[i] "/"; if (!(d in s)) { s[d] = 1; print d } } }' /work/examples/rsync/.run/page-state.txt | LC_ALL=C sort > /work/examples/rsync/.run/index-dirs.txt task: [pages-select] cat /work/examples/rsync/.run/index-stamp.txt /work/examples/rsync/.run/page-state.txt > /work/examples/rsync/.run/indexed.new task: [pages-select] LC_ALL=C comm -23 /work/examples/rsync/.run/index-dirty.txt /work/examples/rsync/.run/index-dirs.txt | awk -v i="example.katoptra.org.directory.index.html" '{ print $0 i } $0 != "" { print substr($0, 1, length($0) - 1) }' | awk -F'\t' 'FILENAME == ARGV[1] { f[$1] = 1; next } !($0 in f)' /work/examples/rsync/.run/page-state.txt - > /work/examples/rsync/.run/index-gone.txt task: [pages-select] LC_ALL=C comm -12 /work/examples/rsync/.run/index-dirty.txt /work/examples/rsync/.run/index-dirs.txt > /work/examples/rsync/.run/index-make.txt @@ -291,23 +307,28 @@ task: [pages-draw] test ! -s /work/examples/rsync/.run/index-make.txt || awk -v | LC_ALL=C sort -u | tr '\n' '\0' | xargs -0r mkdir -p task: [pages-draw] test ! -s /work/examples/rsync/.run/index-make.txt || awk -F'\t' -v OFS='\t' 'FILENAME == ARGV[1] { want[$0] = 1; next } { n = split($1, c, "/"); d = "" - for (i = 1; i < n; i++) { if ((d in want) && !((d c[i]) in seen)) { seen[d c[i]] = 1; print d, "d", c[i], "", "" }; d = d c[i] "/" } + for (i = 1; i < n; i++) { + if ((d in want) && !((d c[i]) in seen)) { seen[d c[i]] = 1; print d, "d", c[i], "", "" } + d = d c[i] "/" } if (d in want) print d, (c[n] == "" ? "e" : "f"), c[n], $2, $3 }' /work/examples/rsync/.run/index-make.txt /work/examples/rsync/.run/page-state.txt | LC_ALL=C sort -t "$(printf '\t')" -k1,1 -k2,2 -k3,3V | awk -F'\t' -v S="/work/examples/rsync/staging/" -v R="/work/examples/rsync/.run/slash/" -v I="example.katoptra.org.directory.index.html" -v F="$PAGE_FOOT" -v N="$INDEX_FILES" -v X="/work/examples/rsync/.run/index-real.txt" -v U="$UPSTREAM" -v C="$CONTACT" -v H="example.katoptra.org" -v css=':root{color-scheme:light dark;--bg:#f6f5f2;--fg:#1c1c28;--mut:#6e6e7e;--line:#e2e0da;--hov:#edebe5;--a:#3b45d9}@media (prefers-color-scheme:dark){:root{--bg:#121218;--fg:#f0f0f5;--mut:#8b8b9c;--line:#262630;--hov:#1a1a22;--a:#8b93ff}}body{margin:0 auto;max-width:60rem;padding:2rem 1rem 3rem;background:var(--bg);color:var(--fg);font:15px/1.45 system-ui,-apple-system,sans-serif}h1{font-size:1.35rem;font-weight:650;margin:0;overflow-wrap:anywhere}p{color:var(--mut);margin:.2rem 0 1.25rem;font-size:.9rem}table{width:100%;border-collapse:collapse;font:13.5px/1.4 ui-monospace,SFMono-Regular,Menlo,Consolas,monospace}th{font:600 .7rem/1 system-ui,sans-serif;letter-spacing:.06em;text-transform:uppercase;color:var(--mut);text-align:left;padding:0 .5rem .6rem;white-space:nowrap}td{padding:.4rem .5rem;border-top:1px solid var(--line);vertical-align:baseline}td:first-child{overflow-wrap:anywhere;width:100%}td+td{white-space:nowrap;color:var(--mut);font-variant-numeric:tabular-nums}th:last-child,td:last-child{text-align:right}tr:hover td{background:var(--hov)}a{color:var(--a);text-decoration:none}a:hover{text-decoration:underline}.s a{color:var(--mut)}.k{font-size:.72rem;margin-left:.6rem;padding:.05rem .4rem;border:1px solid var(--line);border-radius:.3rem;color:var(--mut)}.k:hover{color:var(--a);border-color:var(--a);text-decoration:none}footer{margin-top:2rem;padding-top:1rem;border-top:1px solid var(--line);color:var(--mut);font-size:.85rem;overflow-wrap:anywhere}footer a{color:inherit;text-decoration:underline}@media (max-width:36rem){thead{display:none}tr{display:grid;grid-template-columns:1fr auto;border-top:1px solid var(--line)}td{border:0;padding:.1rem .5rem}td:first-child{grid-column:1/-1;padding-top:.45rem}td+td{padding-bottom:.45rem;font-size:.8rem}}' 'BEGIN { for (i = 0; i < 256; i++) ord[sprintf("%c", i)] = i } function enc(s, out, j, c) { out = ""; for (j = 1; j <= length(s); j++) { c = substr(s, j, 1) out = out ((c ~ /[A-Za-z0-9\/._~-]/) ? c : sprintf("%%%02X", ord[c])) } return out } BEGIN { n = split(N, nm, " "); for (i = n; i >= 1; i--) rank[nm[i]] = i G = "\\.(sig|sign|asc|gpg|dsig|proof|md5|md5sum|sha|sha1|sha224|sha256|sha256sum|sha512|sum|checksums)$" if (C == "" && (j = index(H, ".")) > 1) C = substr(H, 1, j - 1) "@" substr(H, j + 1) } -function esc(s) { gsub(/&/, "\\&", s); gsub(//, "\\>", s); gsub(/"/, "\\"", s); return s } +function esc(s) { gsub(/&/, "\\&", s); gsub(//, "\\>", s) + gsub(/"/, "\\"", s); return s } function grp(n, s, o) { s = sprintf("%.0f", n); o = "" - while (length(s) > 3) { o = "," substr(s, length(s) - 2) o; s = substr(s, 1, length(s) - 3) } return s o } + while (length(s) > 3) { o = "," substr(s, length(s) - 2) o; s = substr(s, 1, length(s) - 3) } + return s o } function iec(n, u, i) { if (n < 1024) return sprintf("%.0f B", n) split("KiB MiB GiB TiB", u, " "); n /= 1024; i = 1; while (n >= 1024 && i < 4) { n /= 1024; i++ } return sprintf(n < 10 ? "%.1f %s" : "%.0f %s", n, u[i]) } function ymd(t) { gsub(/\//, "-", t); return substr(t, 1, 16) } function foot( t) { if (F != "") return "
" sprintf(F, enc(cur)) "
" t = (U == "" ? "" : "Mirror of " esc(U) enc(cur) ".") - if (C != "") t = t (t == "" ? "" : " ") "Report a problem to " esc(C) "." + if (C != "") t = t (t == "" ? "" : " ") "Report a problem to " \ + "" esc(C) "." return (t == "" ? "" : "
" t "
") } function open(d) { cur = d; live = 1; real = ""; nd = 0; nfl = 0; tot = 0; split("", has) } function w(s) { if (real == "") printf("%s", s) > fa; if (cur != "") printf("%s", s) > fb } @@ -315,19 +336,26 @@ function shut( i, r, x, c, e, k, b) { split("", fold); split("", links) for (i = 1; i <= nfl; i++) { r = fname[i]; x = "" while (match(tolower(r), G)) { r = substr(r, 1, RSTART - 1); if (r in has) x = r } if (x != "") { fold[i] = 1 - links[x] = links[x] "" esc(substr(fname[i], length(x) + 1)) "" } } + links[x] = links[x] "" \ + esc(substr(fname[i], length(x) + 1)) "" } } c = (nd ? grp(nd) (nd == 1 ? " directory" : " directories") : "") if (nfl) c = c (c == "" ? "" : ", ") grp(nfl) (nfl == 1 ? " file" : " files") ", " iec(tot) if (c == "") c = "This directory is empty." fa = S cur I; if (real != "") print cur real "\t" cur I > X if (cur != "") { b = substr(cur, 1, length(cur) - 1); k = split(b, e, "/"); fb = R k "/" b } - w("\nIndex of /" esc(cur) "\n") - w("\n\n

Index of /" esc(cur) "

" c "

\n") + w("\n" \ + "" \ + "Index of /" esc(cur) "\n") + w("\n\n" \ + "

Index of /" esc(cur) "

" c "

\n") w("\n") if (cur != "") w("\n") - for (i = 1; i <= nd; i++) w("\n") + for (i = 1; i <= nd; i++) + w("\n") for (i = 1; i <= nfl; i++) if (!(i in fold)) - w("\n") + w("" \ + "\n") w("
NameModified (UTC)Size
../
" esc(dname[i]) "/
" esc(dname[i]) "/
" esc(fname[i]) "" links[fname[i]] "" ymd(ftime[i]) "" iec(fsize[i]) "
" \ + esc(fname[i]) "" links[fname[i]] "" ymd(ftime[i]) "" iec(fsize[i]) "
" foot() "\n") if (real == "") close(fa); if (cur != "") close(fb) } !live || $1 != cur { if (live) shut(); open($1) } @@ -339,10 +367,13 @@ END { if (live) shut() }' task: [index-same] aws s3api list-objects-v2 --bucket example --query 'Contents[].[Key,ETag]' --output text > /work/examples/rsync/.run/bucket-etags.txt || { echo "index-same: the listing failed; index sends each page"; : > /work/examples/rsync/.run/bucket-etags.txt; } task: [index-same] { (cd /work/examples/rsync/staging && find . -type f -exec md5sum {} +) | sed 's# \./#\tp\t#'; for d in /work/examples/rsync/.run/slash/*/; do test -d "$d" || continue; (cd "$d" && find . -type f -exec md5sum {} +) | sed 's# \./#\ts\t#'; done; } > /work/examples/rsync/.run/page-md5.txt task: [index-same] awk -F'\t' -v E="/work/examples/rsync/.run/bucket-etags.txt" -v H="/work/examples/rsync/.run/page-md5.txt" -v I="example.katoptra.org.directory.index.html" -v S="/work/examples/rsync/staging/" -v R="/work/examples/rsync/.run/slash/" -v Y="/work/examples/rsync/.run/index-same.txt" -v M="/work/examples/rsync/.run/index-make.new" 'FILENAME == E { gsub(/"/, "", $2); e[$1] = $2; next } -FILENAME == H { if ($2 == "s") d = $3 "/"; else if (substr($3, length($3) - length(I) + 1) == I) d = substr($3, 1, length($3) - length(I)); else next +FILENAME == H { if ($2 == "s") d = $3 "/" + else if (substr($3, length($3) - length(I) + 1) == I) d = substr($3, 1, length($3) - length(I)) + else next seen[d] = 1; if (e[$3] != $1) diff[d] = 1; next } !($0 in seen) || ($0 in diff) { print > M; next } -{ print > Y; print S $0 I; if ($0 != "") { b = substr($0, 1, length($0) - 1); print R split(b, c, "/") "/" b } }' /work/examples/rsync/.run/bucket-etags.txt /work/examples/rsync/.run/page-md5.txt /work/examples/rsync/.run/index-make.txt | tr '\n' '\0' | xargs -0r rm -f +{ print > Y; print S $0 I + if ($0 != "") { b = substr($0, 1, length($0) - 1); print R split(b, c, "/") "/" b } }' /work/examples/rsync/.run/bucket-etags.txt /work/examples/rsync/.run/page-md5.txt /work/examples/rsync/.run/index-make.txt | tr '\n' '\0' | xargs -0r rm -f task: [index-same] touch /work/examples/rsync/.run/index-same.txt /work/examples/rsync/.run/index-make.new && mv /work/examples/rsync/.run/index-make.new /work/examples/rsync/.run/index-make.txt task: [index] test ! -s /work/examples/rsync/.run/index-make.txt || aws s3 cp --no-progress --cli-connect-timeout 60 --cli-read-timeout 300 --recursive --content-type 'text/html; charset=utf-8' /work/examples/rsync/staging/ s3://example/ task: [index] for d in /work/examples/rsync/.run/slash/*/; do test -d "$d" || continue; aws s3 cp --no-progress --cli-connect-timeout 60 --cli-read-timeout 300 --recursive --content-type 'text/html; charset=utf-8' "$d" s3://example/; done From 9872cfc9358e24dc07310622b8f6dec62af9344b Mon Sep 17 00:00:00 2001 From: Josh Vaughen Date: Fri, 23 Oct 2026 09:56:45 -0700 Subject: [PATCH 6/6] refactor(rsync): drop the step split and the line wrap This commit sets the branch to three changes on v2.9.1 (b9c0ff3): the named programs (3bb0169), one copy of each helper (604e6d7), and the text values of a mirror in env:, done again with ENVIRON. It removes the step split (7dad635) and the wrap at 110 columns with its CI step (bbe7576). The split added 11 task names and anchors for little gain. The wrap broke 18 lines inside a quoted string. The env change, compared with a88a2d2: - awk reads each of the six values with ENVIRON, not with -v. Thus, a backslash escape in a value does not change, and the FRESH_KEY ponytail goes. - PAGE_HTML reads PAGE_FOOT, INDEX_FILES, UPSTREAM and CONTACT from ENVIRON. The PAGE_FOOT check of pages and the footer read the same bytes. - The engine header gives BATCH_SIZE as the task var of a tunable that two commands of a verb read. - offline gives fresh a FRESH_KEY with a quotation mark, a $, a \n and a non-ASCII character. The check fails with the engine of v2.9.1. No behavior change for a mirror. No value that an rsync mirror sets has a backslash. Proof on this tree: - offline, task check and task tools in examples/rsync. - Parity against v2.9.1: 119 of 119 scenarios with the same tree, exit code and output, also the four end-to-end scenarios with the values of ctan, gnu, nongnu and gnu-alpha. - validate-vars.sh, chain-file.sh, gpg-gate-check.sh and op-retry-check.sh. --- .github/workflows/ci.yml | 7 - CLAUDE.md | 44 ++- CONTRIBUTING.md | 14 +- README.md | 47 ++-- engines/rsync.yml | 544 +++++++++--------------------------- examples/rsync/Taskfile.yml | 8 + examples/rsync/render.txt | 126 ++++----- toolbox.yml | 9 +- 8 files changed, 225 insertions(+), 574 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 703f828..90eb9cc 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -26,13 +26,6 @@ jobs: - name: The op verb tries again only when the command did not start if: matrix.variant == 'rsync' run: sh .github/op-retry-check.sh - # A line of the rsync engine that is longer than 110 characters is too long to read in - # a diff or in a review. - - name: No line of the rsync engine is longer than 110 characters - if: matrix.variant == 'rsync' - run: >- - awk 'length > 110 { print FILENAME ":" FNR ": " length " characters"; bad = 1 } END { exit bad }' - engines/rsync.yml - uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1 # Actions keeps the layers in its cache, with one scope for each variant. release # reads the same scope. Thus, for a tag, the build makes new layers only for the diff --git a/CLAUDE.md b/CLAUDE.md index 95c8361..0d830e1 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -28,29 +28,20 @@ Read that section before you write. (verified). A task var with a different name, for example `BATCH_SIZE` in `split`, also gets a call var (verified: `split-refuses` gives `BATCH_GB` as a call var). A global var does not get a call var. Thus, write each default one time, inline or in a task var. -- Do not put `dir:` on an engine verb. Task can read a flattened Taskfile of `includes:` - from a path. In that condition, it adds each `dir:`, also an absolute one, to the - directory of that Taskfile (verified, 3.53.1). Thus, each step of `verify` starts each - command with `cd {{.STAGING}}`. -- A task that a verb starts does not get the call vars of that verb. Thus, `smoke`, - `verify` and `pages` give their steps the vars that the steps read, with a YAML anchor - (`vars: &smoke`, then `vars: *smoke`). Give only text vars in this method. A call var is - text: a root `X: 0` goes through as "0", and then `{{if .X}}` is true (verified, 3.53.1). -- A top-level `env:` in an included Taskfile goes to each task of the mirror, also to the - tasks of the root Taskfile and to the host verbs (verified, 3.53.1). Thus, each command - that sorts or compares in the engine sets `LC_ALL=C` itself. - A task `env:` gets the call vars, and the command gets each value with no change (verified, 3.53.1). Thus, the text values of a mirror go to the commands in `env:`, and a quotation mark or a `$` in a value does not change a command. The render shows - `"$FRESH_KEY"`, not the value. `awk -v` changes each backslash escape in a value, and - `ENVIRON` does not. + `"$FRESH_KEY"`, not the value. In awk, read such a value with `ENVIRON`. `awk -v` + changes each backslash escape in a value (`\n` becomes a newline), and `ENVIRON` does + not. `offline` gives `fresh` a `FRESH_KEY` with a `\n` in its name. - A var of the environment wins over a task `env:` with the same name, also an empty var (verified, 3.53.1). Then `{{.FRESH_KEY}}` and `"$FRESH_KEY"` can give two different values in one command. The container gets only the names in `PASS_ENV`, and none of them is a var of the engine. Do not add a var of the engine to `PASS` or to `op.env`. -- go-task ignores a top-level `set:` in an included Taskfile, also for the tasks of that - file (verified, 3.53.1). Thus, each engine verb that needs pipefail has - `set: [pipefail]`. +- Do not put `dir:` on an engine verb. Task can read a flattened Taskfile of `includes:` + from a path. In that condition, it adds each `dir:`, also an absolute one, to the + directory of that Taskfile (verified, 3.53.1). Thus, `verify` starts each command with + `cd {{.STAGING}}`. - Use only the keys `trusted-hosts` and `cache-expiry` in `.taskrc.yml`. Task ignores a key that it does not know, and it shows no warning. In that condition, it downloads the remote Taskfiles again each time that it runs. @@ -94,13 +85,13 @@ Read that section before you write. `-v2`. The two tags move. When they move, all mirrors get the release. - Give `-r` to each `xargs` if a filter can make its input empty. GNU `xargs` runs its command one time on empty input. A guard on the file that a pipe reads is not a guard on - the input of `xargs`. For example, the `SLASH` awk of `pages-draw` removes the root, which has + the input of `xargs`. For example, the `SLASH` awk of `pages` removes the root, which has no key without a slash. Thus, if the only changed directory of a run is the root, that awk sends no line to `xargs`. - A `>-` folded block keeps a newline if a line that continues the line before it starts - with more spaces than the lines around it. The short awk programs in the commands of - `split`, `pages` and `pages-draw` use this. Put a backslash at the end of a shell line - that must continue. `task check` shows how the block renders. + with more spaces than the lines around it. The two-line awk programs in the commands of + `split` and `pages` use this. Put a backslash at the end of a shell line that must + continue. `task check` shows how the block renders. - On a macOS disk, two directory names that are different only in uppercase and lowercase letters are one directory. Thus, the pages that a laptop writes from a listing of upstream are not complete if upstream has two such directories. For example, CTAN has @@ -114,9 +105,9 @@ Read that section before you write. show this problem. - The `status:` check of `prepare` reads all of `changed.txt`, not the batch. Thus, `prepare` runs on almost all runs. But if the delta of a run has no `TL` path, `prepare` - does not run, and `.run/tl` is not there. Each step of `verify` reads the batch. - `verify-containers` and `verify-decision` use `.run/tl` only if the batch has a `TL` - path. + does not run, and `.run/tl` is not there. Each branch of `verify` reads the batch. The + container branch and the decision-batch branch use `.run/tl` only if the batch has a + `TL` path. - Do not make a task internal if a host verb starts it with its name in the image. go-task does not start an internal task from the command line. For example, `empty-trash` starts `empty-trash-pipeline` with `task op`. Thus, `empty-trash-pipeline` is not internal. @@ -140,9 +131,9 @@ Read that section before you write. of the directories that do not change, and they show the old format. The stamp in `.state/indexed.txt.xz` includes `PAGE_FORMAT`. A new value makes each mirror write all of its pages again, one time. -- `PAGE_HTML`, the page awk of `pages-draw`, writes each page directly to its two files. Do - not make the page with a string that gets a row at a time. That method copies the full - string for each row. The `archive/` directory of TeX Live has approximately 34k rows. +- `PAGE_HTML`, the page awk of `pages`, writes each page directly to its two files. Do not + make the page with a string that gets a row at a time. That method copies the full string + for each row. The `archive/` directory of TeX Live has approximately 34k rows. - In a check that reads the output of a task with `grep`, find text that only the output has. go-task shows each command before it runs it, and that line also has the text of each `echo` in the command. For example, `offline` finds `not 08` in the output of @@ -159,7 +150,6 @@ cd examples/rsync && task image-build && task run -- task tools && task check & cd examples/proton && task image-build && task run -- task tools && task check && task run -- task offline sh .github/validate-vars.sh --check && sh .github/chain-file.sh --check && sh docker/gpg-gate-check.sh sh .github/op-retry-check.sh -awk 'length > 110 { print FILENAME ":" FNR ": " length " characters"; bad = 1 } END { exit bad }' engines/rsync.yml ``` After a change to a verb, run `task render-update` to update the `render.txt` files. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 8b6b6b9..7455d32 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -32,7 +32,6 @@ are applicable to this repository. This repository adds three rules: cd examples/rsync && task image-build && task run -- task tools && task check && task run -- task offline cd examples/proton && task image-build && task run -- task tools && task check && task run -- task offline sh .github/validate-vars.sh --check && sh .github/chain-file.sh --check && sh docker/gpg-gate-check.sh -awk 'length > 110 { print FILENAME ":" FNR ": " length " characters"; bad = 1 } END { exit bad }' engines/rsync.yml ``` CI does these steps for each image, on each pull request and on each push to `main`: @@ -42,15 +41,4 @@ CI does these steps for each image, on each pull request and on each push to `ma 3. It runs `task check`. 4. It runs `task run -- task offline`. -It also runs the three guards one time, in the job of the `rsync` image. That job also -makes sure that no line of `engines/rsync.yml` is longer than 110 characters. To make a -line shorter, use the methods that the file uses: - -- A `desc:` becomes a folded block (`desc: >-`). YAML joins its lines with spaces, and the - text does not change. -- A shell line in a `>-` block breaks at a space. A line with more spaces than the lines - around it keeps its newline. Thus, do not break such a line inside a quoted string. -- A long awk line breaks after `;` or `{`, before `else`, or with a backslash at the end of - the line. A string can become two strings, one after the other. -- An `echo` gets several arguments. It puts one space between two arguments, and the - output does not change. +It also runs the three guards one time, in the job of the `rsync` image. diff --git a/README.md b/README.md index 202ecc0..b12cbfa 100644 --- a/README.md +++ b/README.md @@ -344,16 +344,16 @@ The verbs with a broken line are hooks. These hooks do no work until a mirror se | `prepare` | A hook. If `TL_KEY` has a value and the delta has a `TL` path: it fetches the tlpdb and compares its signature with the pinned key | | `batches` | Runs `fetch`, `verify`, `publish` and `checkpoint` for each of the first `MAX_BATCHES` batches. If there are remaining batches, it makes the file `.run/chain` | | `fetch` | Copies the files of the batch into `staging/` with rsync, and copies the target of each symlink. If upstream deletes a path after the listing, `fetch` does not copy it | -| `verify` | A hook. If `TL_KEY` has a value: it compares each signed file and each container in the batch with the tlpdb. It has three steps: `verify-signed`, `verify-containers` and `verify-decision` | +| `verify` | A hook. If `TL_KEY` has a value: it compares each signed file and each container in the batch with the tlpdb | | `publish` | Runs `label` on the batch. Then it runs `aws s3 cp --recursive` one time for each type, with the files of each type in a different tree. It sends each file as one PutObject, and it makes no listing of the bucket. The engine does not use `aws s3 sync`. The `FRESH_KEY` file goes last | | `label` | Writes `.run/labels.txt`: the Content-Type of each file in staging, from its name and its first KiB. A text type gets a charset if the file is UTF-8. Refer to [Content types](#content-types) | | `checkpoint` | Runs `merge` to add the uploaded files to the state. Then it uploads the state as one PutObject, and it deletes the files in staging | | `delete` | Removes the keys in `deleted.txt` from the bucket, 1,000 in each call, and removes them from the state. It waits for the run that uploads the last batch | | `reconcile` | If `due` wrote `.run/reconcile`, it runs `rebuild`. Then it deletes each key that is not a path of upstream, a key in `OWN`, or a directory of the page state. Then it runs `reconciled` | | `index` | A hook. If `INDEX` has a value: it runs `pages` and `index-same`, uploads the two sets of keys, runs `index-copy`, which copies the index files of `.run/index-real.txt` to their page keys, and removes the keys of the directories that upstream removed. Then it writes the stamp and the page state to `.state/indexed.txt.xz`. A mirror with a different landing page overrides `index` | -| `pages` | Writes a page for each directory that the run changed, from the page state, into staging. If the stamp of INDEXED does not agree, it writes a page for each directory. The keys with no slash at the end go in one tree for each depth. A directory with a file in `INDEX_FILES` gets only the key with no slash, and a line in `.run/index-real.txt`. Its steps are `pages-select`, which finds the directories, and `pages-draw` | +| `pages` | Writes a page for each directory that the run changed, from the page state, into staging. If the stamp of INDEXED does not agree, it writes a page for each directory. The keys with no slash at the end go in one tree for each depth. A directory with a file in `INDEX_FILES` gets only the key with no slash, and a line in `.run/index-real.txt` | | `index-same` | Operates only if the stamp of INDEXED does not agree. It reads the ETag of each key in the bucket. It removes from staging and from `.run/index-make.txt` each directory whose keys have the same MD5, and writes that directory to `.run/index-same.txt`. If the listing fails, `index` sends each page | -| `smoke` | Reads a sample back from `HOST`. The list below the table gives each item of the sample. Its steps are `smoke-sample`, `smoke-tlpdb`, `smoke-page`, `smoke-canary`, `smoke-encoding` and `smoke-unlanded`, which finds the paths that rsync did not send. Then it runs `fresh` and `smoke-mirror` | +| `smoke` | Reads a sample back from `HOST`. The list below the table gives each item of the sample. Then it runs `fresh` and `smoke-mirror` | | `fresh` | If `FRESH_KEY` has a value: it reads the clock of upstream in that file, from `HOST`. If the clock is more than `FRESH_HOURS` before the time of the run, it stops the run | | `smoke-mirror` | A hook, empty here. A mirror with more items to read back writes it | | `report-engine` | A hook: the rows of the engine in the run summary | @@ -637,15 +637,9 @@ change that var. #### When a run fails -The job log shows the verb that stopped the run. If the verb has steps, the log also shows -the step. For example: - -```text -task: Failed to run task "pipeline": task: Failed to run task "smoke": task: Failed to run task "smoke-canary": exit status 1 -``` - -The table in this section gives each verb or step of the engine that can stop a run, with -the cause and the repair. The README of a mirror gives only the values of that mirror: its ceiling, its +The job log shows the step that stopped the run. The name of that step is the verb. The +table in this section gives each verb of the engine that can stop a run, with the cause and +the repair. The README of a mirror gives only the values of that mirror: its ceiling, its upstream, and the steps that only that mirror has. To find the step, and the number of files that the mirror does not have, use these @@ -678,11 +672,9 @@ the work. On a laptop, | `split`: the ceiling | The tree is larger than `CEILING_GB`. The mirror gets no new files until you increase `CEILING_GB`. A larger ceiling also increases the cost | Increase `CEILING_GB` | | `split`: the disk | One file of the delta is too large for the disk of the runner, with 1 GiB free for the other data. The log shows `no room for `. `CEILING_GB` has no effect on this check | Remove the file from the listing with `FILTER`, or give the run a larger disk | | `prepare`, `verify` | A signature or a checksum did not agree. The engine uploaded no file from that batch, and clients continue to get the last good copy | If TeX Live changed its key, compare `TL_KEY` with https://www.tug.org/texlive/verify.html. Do not use a different source for the key | -| `smoke-canary` | The domain served bytes that are not the bytes in the bucket, or it rejected a Perl client. A zone rule is missing, or the scope of a zone rule became wider. The bucket is correct | Examine the Configuration Rule of the zone. Refer to [The parts of a bucket](#the-parts-of-a-bucket) | -| `smoke-encoding` | A `.tar.gz` had a `Content-Encoding` when `smoke` read it. Clients then decompress it while they download it | Examine `label` and the AWS CLI version in `toolchain.lock.toml`. Refer to [Content types](#content-types) | -| `smoke-sample` | A key of the sample had a Content-Type that is not its label, or a length that is not its length in the listing. The CLI, R2 or the zone changed the object | Examine `label` and the AWS CLI version in `toolchain.lock.toml`. Then examine the rules of the zone | -| `smoke-tlpdb` | The domain served a `texlive.tlpdb.sha512` that is not the copy that `prepare` verified | Start the run again. If the problem continues, examine the rules of the zone | -| `smoke-page` | A page that the run wrote did not have the length of the page in staging, at its page key or at its key with no slash. If the page key is a copy of an index file, its length or its Content-Type was not that of the file. R2 or the zone changed the page | Examine the rules of the zone. Refer to [Directory pages](#directory-pages) | +| `smoke`: the canary | The domain served bytes that are not the bytes in the bucket, or it rejected a Perl client. A zone rule is missing, or the scope of a zone rule became wider. The bucket is correct | Examine the Configuration Rule of the zone. Refer to [The parts of a bucket](#the-parts-of-a-bucket) | +| `smoke`: the Content-Encoding | A `.tar.gz` had a `Content-Encoding` when `smoke` read it. Clients then decompress it while they download it | Examine `label` and the AWS CLI version in `toolchain.lock.toml`. Refer to [Content types](#content-types) | +| `smoke`: the type or the length | A key of the sample had a Content-Type that is not its label, or a length that is not its length in the listing. The CLI, R2 or the zone changed the object | Examine `label` and the AWS CLI version in `toolchain.lock.toml`. Then examine the rules of the zone | | `fresh`: the clock stopped | The clock in the `FRESH_KEY` file did not change for more than `FRESH_HOURS`. Upstream stopped, or the mirror that `SOURCE` gives stopped. A secondary mirror can stop its sync, and rsync can continue to connect to it. The run copied all the bytes that it got | Examine upstream. If `SOURCE` is a secondary mirror, set `SOURCE` to a different secondary mirror. If upstream stopped, no change in the mirror is necessary | These procedures repair other problems: @@ -1347,12 +1339,10 @@ command in `env:`, for example `FRESH_KEY`. The render shows these values as nam - **A task that a verb runs does not get the call vars of that verb.** Also, a root var cannot read a var of an included Taskfile. A verb can run a hook with a `RUN` or a `STAGING` that a caller can change. In that condition, the verb must give these vars to - the hook. `smoke` does this. `smoke`, `verify` and `pages` also give their steps the vars - that the steps read, with a YAML anchor. + the hook. `smoke` does this. - **Do not put `dir:` on a verb of an included Taskfile.** If task reads a flattened Taskfile of `includes:` from a path, it adds each `dir:` to the directory of that - Taskfile, also an absolute `dir:`. Thus, each step of `verify` starts each command with - `cd`. + Taskfile, also an absolute `dir:`. Thus, `verify` starts each command with `cd`. - **`render` puts an empty directory on `.run`.** Thus, a `sh:` var that reads `.run` when task reads the Taskfile gives the same render in each checkout. A verb that reads `.run` at that time must accept an empty `.run`. @@ -1403,10 +1393,11 @@ these rules: different name, for example `BATCH_SIZE` in `split`. Write the default one time. Add a row to the table in [The vars](#the-vars). - Give a text value of a mirror, for example `FRESH_KEY`, to a command in `env:`, not in a - template. The shell reads `"$X"`, awk gets the value with `-v` or `ENVIRON`, and Python - gets it as an argument. The render then shows the name of the value, not the value. A - var of the environment with the same name wins over `env:`. Thus, do not pass such a - name into the container (`PASS`). + template. The shell reads `"$X"`, awk reads `ENVIRON["X"]`, and Python gets the value as + an argument. Do not give such a value to awk with `-v`, because `-v` changes each + backslash escape. The render then shows the name of the value, not the value. A var of + the environment with the same name wins over `env:`. Thus, do not pass such a name into + the container (`PASS`). - Put each awk or Python program of more than two lines in the `programs` part of `vars:`. Write a comment above it that gives its function and its inputs. The command that runs the program gives it each input with awk `-v`, `ENVIRON` or an argument. A program has @@ -1497,9 +1488,9 @@ because `empty-trash` starts that task from the command line in the image. Last, encrypts and decrypts one file with `age`. CI runs the same four steps for each variant on each pull request and on each push to -`main`. It also runs the test cases of the three guards: `.github/validate-vars.sh`, -`.github/chain-file.sh` and `docker/gpg-gate-check.sh`. It also makes sure that no line of -`engines/rsync.yml` is longer than 110 characters. +`main`. It also runs the test +cases of the three guards: `.github/validate-vars.sh`, `.github/chain-file.sh` and +`docker/gpg-gate-check.sh`. When you change a verb, run `task render-update` in each example. The diff in the pull request is the review. `CONTRIBUTING.md` gives the three rules that this repository adds to diff --git a/engines/rsync.yml b/engines/rsync.yml index eab461f..e0c82a4 100644 --- a/engines/rsync.yml +++ b/engines/rsync.yml @@ -33,13 +33,15 @@ # MAX_BATCHES. # - A var with a different name that reads the tunable continues to see the command # line. Thus, each default is in one location only, in the vars and verbs that follow. +# When two commands of a verb read a tunable, the default is in a task var with a +# different name, for example BATCH_SIZE in split. # - A text value of the mirror (FRESH_KEY, CANARY, PAGE_FOOT, UPSTREAM, CONTACT and # INDEX_FILES) goes to a command in `env:`, not in a template. The shell reads "$X", -# awk gets the value with -v or ENVIRON, and Python gets it as an argument. Thus, no -# value goes into the text of a program, and a quotation mark in a value cannot -# change a command. A var of the environment with the same name wins over `env:`. -# The container gets only the names in PASS_ENV of the toolbox. Thus, do not pass -# one of these names into the container. +# awk reads ENVIRON["X"], and Python gets the value as an argument. Thus, no value goes +# into the text of a program, and a quotation mark, a `$` or a backslash in a value +# cannot change a command. A var of the environment with the same name wins over +# `env:`. The container gets only the names in PASS_ENV of the toolbox. Thus, do not +# pass one of these names into the container. # The other vars get their values from the vars of the mirror, or they are helper vars of # the engine. version: '3' @@ -121,8 +123,7 @@ vars: # together give the key of the page. ANCESTORS: |- awk -F'\t' '!("" in s) { s[""] = 1; print "" } - { n = split($1, c, "/"); d = "" - for (i = 1; i < n; i++) { d = d c[i] "/"; if (!(d in s)) { s[d] = 1; print d } } }' + { n = split($1, c, "/"); d = ""; for (i = 1; i < n; i++) { d = d c[i] "/"; if (!(d in s)) { s[d] = 1; print d } } }' # The clock of upstream in a timestamp file, as Unix time or as "YYYY-MM-DD HH:MM" for # date -d. mirror-updated-timestamp.txt (gnu) and 00_TIME.txt (Savannah) start with Unix # time. The timestamp file of CTAN holds 2026-10-08-05-02, after a comment and the name @@ -189,8 +190,7 @@ vars: /^containerchecksum /{split($0, a, " "); emit(a[2], n)} /^doccontainerchecksum /{split($0, a, " "); emit(a[2], n ".doc")} /^srccontainerchecksum /{split($0, a, " "); emit(a[2], n ".source")} - function emit(sum, base, p, q) { - p = TL "/archive/" base ".tar.xz"; q = TL "/archive/" base ".r" r ".tar.xz" + function emit(sum, base, p, q) { p = TL "/archive/" base ".tar.xz"; q = TL "/archive/" base ".r" r ".tar.xz" if (p in want) { print sum " " p; delete want[p] } if (q in want) { print sum " " q; delete want[q] } } END { for (p in want) { print "no checksum in the tlpdb for " p > "/dev/stderr"; bad = 1 } exit bad } @@ -332,13 +332,10 @@ vars: # - S and R: STAGING and SLASH, each with a slash at the end. SAME_PAGES: |- FILENAME == E { gsub(/"/, "", $2); e[$1] = $2; next } - FILENAME == H { if ($2 == "s") d = $3 "/" - else if (substr($3, length($3) - length(I) + 1) == I) d = substr($3, 1, length($3) - length(I)) - else next + FILENAME == H { if ($2 == "s") d = $3 "/"; else if (substr($3, length($3) - length(I) + 1) == I) d = substr($3, 1, length($3) - length(I)); else next seen[d] = 1; if (e[$3] != $1) diff[d] = 1; next } !($0 in seen) || ($0 in diff) { print > M; next } - { print > Y; print S $0 I - if ($0 != "") { b = substr($0, 1, length($0) - 1); print R split(b, c, "/") "/" b } } + { print > Y; print S $0 I; if ($0 != "") { b = substr($0, 1, length($0) - 1); print R split(b, c, "/") "/" b } } # pages: the entries "dir TAB d|e|f TAB name TAB size TAB mtime" of each directory of the # first input (index-make.txt), from the page state (the second input): the files in the # bucket and the directories of upstream. A line "dir/" of the page state gives an e entry @@ -347,9 +344,7 @@ vars: PAGE_ENTRIES: |- FILENAME == ARGV[1] { want[$0] = 1; next } { n = split($1, c, "/"); d = "" - for (i = 1; i < n; i++) { - if ((d in want) && !((d c[i]) in seen)) { seen[d c[i]] = 1; print d, "d", c[i], "", "" } - d = d c[i] "/" } + for (i = 1; i < n; i++) { if ((d in want) && !((d c[i]) in seen)) { seen[d c[i]] = 1; print d, "d", c[i], "", "" }; d = d c[i] "/" } if (d in want) print d, (c[n] == "" ? "e" : "f"), c[n], $2, $3 } # pages: the style of each directory page. Each line ends with a backslash. Thus, YAML # joins the lines with no space, and the style is one line in each page. @@ -388,11 +383,12 @@ vars: # /dir/ and for /dir. Without it, a relative href in the second resolves against the # parent. The program percent-encodes each name in href and HTML-escapes it in text. # - # The footer is PAGE_FOOT (F), with %s for the encoded path of the directory. When - # PAGE_FOOT is empty, the program makes the footer from UPSTREAM (U) and CONTACT (C). - # CONTACT gets its default from HOST (H). + # The program reads PAGE_FOOT, INDEX_FILES, UPSTREAM and CONTACT from ENVIRON, into F, N, + # U and C. The footer is PAGE_FOOT, with %s for the encoded path of the directory. When + # PAGE_FOOT is empty, the program makes the footer from UPSTREAM and CONTACT. CONTACT + # gets its default from HOST (H). # - # rank gives the position of each name in INDEX_FILES (N). real is the index file of the + # rank gives the position of each name in INDEX_FILES. real is the index file of the # open directory with the lowest rank. Without a real, shut writes the page key, and with # a real, it writes a line to X (index-real.txt). # @@ -404,22 +400,20 @@ vars: # css (PAGE_CSS). PAGE_HTML: |- {{.URLENC}} - BEGIN { n = split(N, nm, " "); for (i = n; i >= 1; i--) rank[nm[i]] = i + BEGIN { F = ENVIRON["PAGE_FOOT"]; N = ENVIRON["INDEX_FILES"]; U = ENVIRON["UPSTREAM"]; C = ENVIRON["CONTACT"] + n = split(N, nm, " "); for (i = n; i >= 1; i--) rank[nm[i]] = i G = "\\.(sig|sign|asc|gpg|dsig|proof|md5|md5sum|sha|sha1|sha224|sha256|sha256sum|sha512|sum|checksums)$" if (C == "" && (j = index(H, ".")) > 1) C = substr(H, 1, j - 1) "@" substr(H, j + 1) } - function esc(s) { gsub(/&/, "\\&", s); gsub(//, "\\>", s) - gsub(/"/, "\\"", s); return s } + function esc(s) { gsub(/&/, "\\&", s); gsub(//, "\\>", s); gsub(/"/, "\\"", s); return s } function grp(n, s, o) { s = sprintf("%.0f", n); o = "" - while (length(s) > 3) { o = "," substr(s, length(s) - 2) o; s = substr(s, 1, length(s) - 3) } - return s o } + while (length(s) > 3) { o = "," substr(s, length(s) - 2) o; s = substr(s, 1, length(s) - 3) } return s o } function iec(n, u, i) { if (n < 1024) return sprintf("%.0f B", n) split("KiB MiB GiB TiB", u, " "); n /= 1024; i = 1; while (n >= 1024 && i < 4) { n /= 1024; i++ } return sprintf(n < 10 ? "%.1f %s" : "%.0f %s", n, u[i]) } function ymd(t) { gsub(/\//, "-", t); return substr(t, 1, 16) } function foot( t) { if (F != "") return "
" sprintf(F, enc(cur)) "
" t = (U == "" ? "" : "Mirror of " esc(U) enc(cur) ".") - if (C != "") t = t (t == "" ? "" : " ") "Report a problem to " \ - "" esc(C) "." + if (C != "") t = t (t == "" ? "" : " ") "Report a problem to " esc(C) "." return (t == "" ? "" : "
" t "
") } function open(d) { cur = d; live = 1; real = ""; nd = 0; nfl = 0; tot = 0; split("", has) } function w(s) { if (real == "") printf("%s", s) > fa; if (cur != "") printf("%s", s) > fb } @@ -427,26 +421,19 @@ vars: for (i = 1; i <= nfl; i++) { r = fname[i]; x = "" while (match(tolower(r), G)) { r = substr(r, 1, RSTART - 1); if (r in has) x = r } if (x != "") { fold[i] = 1 - links[x] = links[x] "" \ - esc(substr(fname[i], length(x) + 1)) "" } } + links[x] = links[x] "" esc(substr(fname[i], length(x) + 1)) "" } } c = (nd ? grp(nd) (nd == 1 ? " directory" : " directories") : "") if (nfl) c = c (c == "" ? "" : ", ") grp(nfl) (nfl == 1 ? " file" : " files") ", " iec(tot) if (c == "") c = "This directory is empty." fa = S cur I; if (real != "") print cur real "\t" cur I > X if (cur != "") { b = substr(cur, 1, length(cur) - 1); k = split(b, e, "/"); fb = R k "/" b } - w("\n" \ - "" \ - "Index of /" esc(cur) "\n") - w("\n\n" \ - "

Index of /" esc(cur) "

" c "

\n") + w("\nIndex of /" esc(cur) "\n") + w("\n\n

Index of /" esc(cur) "

" c "

\n") w("\n") if (cur != "") w("\n") - for (i = 1; i <= nd; i++) - w("\n") + for (i = 1; i <= nd; i++) w("\n") for (i = 1; i <= nfl; i++) if (!(i in fold)) - w("" \ - "\n") + w("\n") w("
NameModified (UTC)Size
../
" esc(dname[i]) "/
" esc(dname[i]) "/
" \ - esc(fname[i]) "" links[fname[i]] "" ymd(ftime[i]) "" iec(fsize[i]) "
" esc(fname[i]) "" links[fname[i]] "" ymd(ftime[i]) "" iec(fsize[i]) "
" foot() "\n") if (real == "") close(fa); if (cur != "") close(fb) } !live || $1 != cur { if (live) shut(); open($1) } @@ -458,9 +445,7 @@ vars: tasks: pipeline: - desc: >- - clock -> due -> list -> state -> rebuild? -> diff -> split -> prepare -> batches -> delete -> - reconcile? -> index -> smoke -> report -> ping + desc: clock -> due -> list -> state -> rebuild? -> diff -> split -> prepare -> batches -> delete -> reconcile? -> index -> smoke -> report -> ping cmds: - {task: clock} - {task: due} @@ -489,18 +474,14 @@ tasks: # ---- the listing and the state ---- list: - desc: >- - List upstream, narrowed by FILTER when set, and normalise it to RUN/upstream.txt as "path TAB - size TAB mtime", byte-sorted + desc: List upstream, narrowed by FILTER when set, and normalise it to RUN/upstream.txt as "path TAB size TAB mtime", byte-sorted set: [pipefail] cmds: # In a run, each batch appends to publish.txt and labeled.txt. Only the verb that # uses one of the other files writes that file. A workspace that a previous run used # must start each run without these files. If not, smoke and the report count lines # from a previous run. - - >- - rm -f {{.RUN}}/publish.txt {{.RUN}}/labeled.txt {{.RUN}}/orphans.txt {{.RUN}}/unlanded.txt - {{.RUN}}/fresh.txt {{.RUN}}/index-same.txt + - rm -f {{.RUN}}/publish.txt {{.RUN}}/labeled.txt {{.RUN}}/orphans.txt {{.RUN}}/unlanded.txt {{.RUN}}/fresh.txt {{.RUN}}/index-same.txt - mkdir -p {{.RUN}} {{.STAGING}} # -L gives each symlink as the file that it points to. If the target of a symlink is # not upstream, rsync reports the symlink and ignores it. rsync then exits 23, but @@ -514,17 +495,11 @@ tasks: # a name as \#ooo. fetch then asks for a name that upstream does not have, and the # file never lands. -8 writes the bytes of the name. - task: retry - vars: - CMD: >- - TZ=UTC {{.RSYNC}}{{if .FILTER}} {{.FILTER}}{{end}} -8 -rL --list-only {{.SOURCE}} - > {{.RUN}}/listing.txt + vars: {CMD: 'TZ=UTC {{.RSYNC}}{{if .FILTER}} {{.FILTER}}{{end}} -8 -rL --list-only {{.SOURCE}} > {{.RUN}}/listing.txt'} - {task: normalise} normalise: - desc: >- - RUN/listing.txt (rsync --list-only) to RUN/upstream.txt, every regular file as "path TAB size - TAB mtime", and RUN/upstream-dirs.txt, every directory but the root as "path/"; both - byte-sorted + desc: RUN/listing.txt (rsync --list-only) to RUN/upstream.txt, every regular file as "path TAB size TAB mtime", and RUN/upstream-dirs.txt, every directory but the root as "path/"; both byte-sorted set: [pipefail] cmds: - awk '{{.LIST_FILES}}' {{.RUN}}/listing.txt | LC_ALL=C sort > {{.RUN}}/upstream.txt @@ -532,29 +507,18 @@ tasks: # A truncated listing must not become a list of deletions. LIST_FLOOR is a line # count, approximately 90% of the usual size of the listing. A correct listing # always has more lines. When LIST_FLOOR is not set, there is no guard. - - >- - {{if .LIST_FLOOR}}test "$(wc -l < {{.RUN}}/upstream.txt)" -gt {{.LIST_FLOOR}}{{else}}echo - "no LIST_FLOOR: a truncated listing is not caught here"{{end}} + - '{{if .LIST_FLOOR}}test "$(wc -l < {{.RUN}}/upstream.txt)" -gt {{.LIST_FLOOR}}{{else}}echo "no LIST_FLOOR: a truncated listing is not caught here"{{end}}' state: - desc: >- - Fetch the state file to RUN/applied.txt; a missing one is rebuilt from a bucket listing (an - empty bucket gives an empty state, and the first run fills it) + desc: Fetch the state file to RUN/applied.txt; a missing one is rebuilt from a bucket listing (an empty bucket gives an empty state, and the first run fills it) cmds: - mkdir -p {{.RUN}} # The bucket is the mirror, and the state is a cache of the bucket. Thus, a missing # state causes one listing of the bucket, not an upload. - - task: pull - vars: - KEY: '{{.STATE}}' - OUT: '{{.RUN}}/applied.txt' - MISSING: >- - echo "no state at {{.STATE}}: rebuilding it from the bucket" >&2; touch {{.RUN}}/rebuild-now + - {task: pull, vars: {KEY: '{{.STATE}}', OUT: '{{.RUN}}/applied.txt', MISSING: 'echo "no state at {{.STATE}}: rebuilding it from the bucket" >&2; touch {{.RUN}}/rebuild-now'}} pull: - desc: >- - An xz-compressed KEY from the bucket, decompressed to OUT; MISSING runs when the bucket has no - such key + desc: An xz-compressed KEY from the bucket, decompressed to OUT; MISSING runs when the bucket has no such key internal: true requires: {vars: [KEY, OUT, MISSING]} cmds: @@ -569,9 +533,7 @@ tasks: esac push: - desc: >- - FILE, xz-compressed, to the bucket's KEY as one PutObject; the old object or the new one is in - the bucket, never half + desc: FILE, xz-compressed, to the bucket's KEY as one PutObject; the old object or the new one is in the bucket, never half internal: true requires: {vars: [FILE, KEY]} cmds: @@ -603,9 +565,7 @@ tasks: - rm -f {{.RUN}}/rebuild-now diff: - desc: >- - RUN/changed.txt (lines upstream has and the state lacks, plus the file each changed signed - sha512 names), deleted.txt (paths the state has and upstream lacks), paths.txt + desc: RUN/changed.txt (lines upstream has and the state lacks, plus the file each changed signed sha512 names), deleted.txt (paths the state has and upstream lacks), paths.txt cmds: - LC_ALL=C comm -13 {{.RUN}}/applied.txt {{.RUN}}/upstream.txt > {{.RUN}}/changed.txt # Upstream can write a signed sha512 again while the file that it identifies does @@ -614,17 +574,13 @@ tasks: # into the delta. Thus, verify examines it with the new sha512 in staging. The # pattern is the same as in verify. - >- - awk -F'\t' -v TL="{{.TL}}" 'FILENAME == ARGV[1] { if ($1 ~ "^" TL - "/(tlpkg/texlive\\.tlpdb|[^/]+)\\.sha512$") want[substr($1, 1, length($1) - 7)]; next } - $1 in want' {{.RUN}}/changed.txt {{.RUN}}/upstream.txt | LC_ALL=C sort -u -o {{.RUN}}/changed.txt - - {{.RUN}}/changed.txt + awk -F'\t' -v TL="{{.TL}}" 'FILENAME == ARGV[1] { if ($1 ~ "^" TL "/(tlpkg/texlive\\.tlpdb|[^/]+)\\.sha512$") want[substr($1, 1, length($1) - 7)]; next } + $1 in want' {{.RUN}}/changed.txt {{.RUN}}/upstream.txt | LC_ALL=C sort -u -o {{.RUN}}/changed.txt - {{.RUN}}/changed.txt - cut -f1 {{.RUN}}/upstream.txt > {{.RUN}}/paths.txt - cut -f1 {{.RUN}}/applied.txt | LC_ALL=C comm -23 - {{.RUN}}/paths.txt > {{.RUN}}/deleted.txt split: - desc: >- - Refuse a tree over CEILING_GB (when set) or a file the disk cannot hold; split changed.txt - into RUN/batch-NNNN.txt + desc: Refuse a tree over CEILING_GB (when set) or a file the disk cannot hold; split changed.txt into RUN/batch-NNNN.txt vars: # A task var, not a global var, because a caller can give BATCH_GB as a call var. BATCH_SIZE: '{{.BATCH_GB | default 4}}' @@ -637,19 +593,15 @@ tasks: # CEILING_GB unset or 0 gives no ceiling. The command always prints the size. - >- {{if .CEILING_GB}}awk -F'\t' -v gb={{.CEILING_GB}} '{ s += $2 } - END { printf "upstream: %d objects, %.2f GB, ceiling %d GB\n", NR, s / 1e9, gb - exit (s > gb * 1e9) }' \ - {{.RUN}}/upstream.txt{{else}}awk -F'\t' - '{ s += $2 } END { printf "upstream: %d objects, %.2f GB, no ceiling\n", NR, s / 1e9 }' - {{.RUN}}/upstream.txt{{end}} + END { printf "upstream: %d objects, %.2f GB, ceiling %d GB\n", NR, s / 1e9, gb; exit (s > gb * 1e9) }' \ + {{.RUN}}/upstream.txt{{else}}awk -F'\t' '{ s += $2 } END { printf "upstream: %d objects, %.2f GB, no ceiling\n", NR, s / 1e9 }' {{.RUN}}/upstream.txt{{end}} - mkdir -p {{.STAGING}} && find {{.STAGING}} -mindepth 1 -delete - df -Pk {{.STAGING}} # The disk must have space for the largest file, plus 1 GiB for the text files of # RUN and for the OS. - >- awk -F'\t' -v kb="$(df -Pk {{.STAGING}} | awk 'NR == 2 { print $4 }')" \ - '$2 / 1024 > kb - {{.HEADROOM_KIB}} { print "no room for " $1 " (" $2 " bytes)"; bad = 1 } - END { exit bad }' \ + '$2 / 1024 > kb - {{.HEADROOM_KIB}} { print "no room for " $1 " (" $2 " bytes)"; bad = 1 } END { exit bad }' \ {{.RUN}}/changed.txt - rm -f {{.RUN}}/batch-*.txt # BATCH_GB goes into the command with no quotes, because it is in $(( )), where a @@ -661,24 +613,19 @@ tasks: # ---- the batches ---- prepare: - desc: >- - With TL_KEY set and the delta touching TL, fetch its control files into RUN/tl and check the - signed sha512 against the pinned key; the no-op hook otherwise + desc: With TL_KEY set and the delta touching TL, fetch its control files into RUN/tl and check the signed sha512 against the pinned key; the no-op hook otherwise status: - 'test -z "{{.TL_KEY}}" || test "$(grep -c "^{{.TL}}/" {{.RUN}}/changed.txt 2>/dev/null)" = 0' set: [pipefail] cmds: - mkdir -p {{.RUN}}/tl - - >- - printf '%s\n' tlpkg/texlive.tlpdb.xz tlpkg/texlive.tlpdb.sha512 tlpkg/texlive.tlpdb.sha512.asc - tlpkg/gpg/pubring.gpg > {{.RUN}}/tl/files.txt + - printf '%s\n' tlpkg/texlive.tlpdb.xz tlpkg/texlive.tlpdb.sha512 tlpkg/texlive.tlpdb.sha512.asc tlpkg/gpg/pubring.gpg > {{.RUN}}/tl/files.txt - task: retry vars: {CMD: '{{.RSYNC}} -Lt --files-from={{.RUN}}/tl/files.txt {{.SOURCE}}{{.TL}}/ {{.RUN}}/tl/'} - xz -dkf {{.RUN}}/tl/tlpkg/texlive.tlpdb.xz - >- cd {{.RUN}}/tl/tlpkg && shasum -a 512 -c texlive.tlpdb.sha512 && - gpgv --status-fd 1 --keyring gpg/pubring.gpg texlive.tlpdb.sha512.asc texlive.tlpdb.sha512 | awk -v - k={{.TL_KEY}} '{{.GPGCHECK}}' + gpgv --status-fd 1 --keyring gpg/pubring.gpg texlive.tlpdb.sha512.asc texlive.tlpdb.sha512 | awk -v k={{.TL_KEY}} '{{.GPGCHECK}}' batches: desc: Work the first MAX_BATCHES of RUN/batch-NNNN.txt in name order; each commits before the next starts @@ -708,9 +655,7 @@ tasks: internal: true requires: {vars: [B]} preconditions: - - sh: >- - test "$(df -Pk {{.STAGING}} | awk 'NR == 2 { print $4 }')" - -gt "$(awk -F'\t' '{ s += $2 } END { print int(s / 1024) + {{.HEADROOM_KIB}} }' {{.B}})" + - sh: test "$(df -Pk {{.STAGING}} | awk 'NR == 2 { print $4 }')" -gt "$(awk -F'\t' '{ s += $2 } END { print int(s / 1024) + {{.HEADROOM_KIB}} }' {{.B}})" msg: "not enough free disk for {{.B}}" cmds: - find {{.STAGING}} -mindepth 1 -delete @@ -725,57 +670,30 @@ tasks: # No --partial. Each batch removes all files from staging. Thus, there is nothing to # continue. Also, a file that rsync did not complete must not go into the bucket. - task: retry - vars: - CMD: >- - {{.RSYNC}} -Lt --files-from={{.RUN}}/files.txt --ignore-missing-args {{.SOURCE}} - {{.STAGING}}/ + vars: {CMD: '{{.RSYNC}} -Lt --files-from={{.RUN}}/files.txt --ignore-missing-args {{.SOURCE}} {{.STAGING}}/'} verify: # Not internal. Thus, a mirror can start it manually on a batch from a fixture. requires: {vars: [B]} status: ['test -z "{{.TL_KEY}}"'] # the no-op hook until a mirror pins a key - # Each command of its steps starts with a cd to STAGING, not dir:. go-task resolves - # dir: against the directory of an include that it reads from a path, and no call var - # can change that. The steps operate without pipefail, because the grep in - # verify-signed can correctly find nothing. - cmds: - # A task does not see the call vars of the task that starts it. Thus, verify gives its - # steps the vars that they read, with the anchor batch. In a run, these are the values - # that verify has. An offline check gives different values. - - task: verify-signed - vars: &batch - B: '{{.B}}' - RUN: '{{.RUN}}' - STAGING: '{{.STAGING}}' - TL: '{{.TL}}' - TL_KEY: '{{.TL_KEY}}' - - {task: verify-containers, vars: *batch} - - {task: verify-decision, vars: *batch} - - verify-signed: - desc: Each signed sha512 of batch B, with shasum, then gpgv against the keyring that prepare pinned - internal: true + # Each command starts with a cd to STAGING, not dir:. go-task resolves dir: against + # the directory of an include that it reads from a path, and no call var can change + # that. cmds: # This command examines each signed sha512 in this batch (the installers and # updaters at the root, and the tlpdb) with the keyring that prepare pinned. The # status of the while loop is the status of the pipeline. - >- - cd {{.STAGING}} && cut -f1 {{.B}} | grep -E '^{{.TL}}/(tlpkg/texlive\.tlpdb|[^/]+)\.sha512$' | - while IFS= read -r f; do + cd {{.STAGING}} && cut -f1 {{.B}} | grep -E '^{{.TL}}/(tlpkg/texlive\.tlpdb|[^/]+)\.sha512$' | while IFS= read -r f; do (cd "$(dirname "$f")" && f=$(basename "$f") && shasum -a 512 -c "$f" && - gpgv --status-fd 1 --keyring {{.RUN}}/tl/tlpkg/gpg/pubring.gpg "$f.asc" "$f" | awk -v k={{.TL_KEY}} - '{{.GPGCHECK}}') + gpgv --status-fd 1 --keyring {{.RUN}}/tl/tlpkg/gpg/pubring.gpg "$f.asc" "$f" | awk -v k={{.TL_KEY}} '{{.GPGCHECK}}') || exit 1; done - - verify-containers: - desc: Each TL container of batch B against its checksum in the tlpdb that prepare verified - internal: true - cmds: # Each TL container in this batch must agree with the checksum that the tlpdb gives # for it (TLPDB_SUMS). prepare examined that tlpdb. # # The awk program writes to a file, not to a pipe. Thus, the status of the command - # is the status of awk. + # is the status of awk. verify operates without pipefail, because the grep in the + # first command of verify can correctly find nothing. # # A batch with no container does not do this check. prepare makes RUN/tl, and # prepare operates only when the delta has a path in TL. Thus, in an hour with no @@ -789,16 +707,8 @@ tasks: # between the two. - >- cd {{.STAGING}} && { test "$(grep -c '^{{.TL}}/archive/' {{.B}})" = 0 || { - awk -F'\t' -v TL="{{.TL}}" '{{.TLPDB_SUMS}}' {{.B}} {{.RUN}}/tl/tlpkg/texlive.tlpdb > - {{.RUN}}/tl/sums.txt && + awk -F'\t' -v TL="{{.TL}}" '{{.TLPDB_SUMS}}' {{.B}} {{.RUN}}/tl/tlpkg/texlive.tlpdb > {{.RUN}}/tl/sums.txt && shasum -a 512 -c --quiet {{.RUN}}/tl/sums.txt; }; } - - verify-decision: - desc: >- - When batch B holds the tlpdb, the tlpdb against prepare's, its .xz against it, and each - container that it names against the bucket after this run - internal: true - cmds: # The decision batch. For this batch, these conditions must be correct: # - The tlpdb in the batch is the tlpdb that prepare examined at the start of the # run. @@ -832,25 +742,17 @@ tasks: - {task: label-trees, vars: {TO: up}} - >- for d in {{.RUN}}/up/*/; do test -d "$d" || continue; - aws s3 cp {{.AWS_FLAGS}} --recursive --content-type "$(cat "${d%/}.type")" "$d" {{.S3}}/ | tee -a - {{.RUN}}/publish.txt; done + aws s3 cp {{.AWS_FLAGS}} --recursive --content-type "$(cat "${d%/}.type")" "$d" {{.S3}}/ | tee -a {{.RUN}}/publish.txt; done - {task: label-trees, vars: {TO: staging}} # The FRESH_KEY file goes to the bucket last, in one more command. It gives the date # of the tree, and monitors (for example, mirmon) read it. - # - # ponytail: awk -v changes each backslash escape in FRESH_KEY, here, in smoke-sample - # and in fresh. Ceiling: a key with a backslash. Upgrade path: read the key with - # ENVIRON in these three commands. - >- {{if .FRESH_KEY}}test ! -f {{.STAGING}}/"$FRESH_KEY" || - aws s3 cp {{.AWS_FLAGS}} --content-type "$(awk -F'\t' -v k="$FRESH_KEY" '$1 == k { print $2 }' - {{.RUN}}/labels.txt)" + aws s3 cp {{.AWS_FLAGS}} --content-type "$(awk -F'\t' '$1 == ENVIRON["FRESH_KEY"] { print $2 }' {{.RUN}}/labels.txt)" {{.STAGING}}/"$FRESH_KEY" {{.S3}}/"$FRESH_KEY" | tee -a {{.RUN}}/publish.txt{{end}} label: - desc: >- - RUN/labels.txt, "path TAB type" for every file under DIR (STAGING), byte-sorted; the type is - decided from the name and the first KiB, never left to the CLI + desc: RUN/labels.txt, "path TAB type" for every file under DIR (STAGING), byte-sorted; the type is decided from the name and the first KiB, never left to the CLI vars: DIR: '{{if .DIR}}{{.DIR}}{{else}}{{.STAGING}}{{end}}' cmds: @@ -866,9 +768,7 @@ tasks: - LC_ALL=C sort {{.RUN}}/labels.raw > {{.RUN}}/labels.txt && rm {{.RUN}}/labels.raw label-trees: - desc: >- - TO=up: rename every file of RUN/labels.txt but the FRESH_KEY file out of STAGING into - RUN/up/N/, one tree per type, its type in RUN/up/N.type. TO=staging: rename them back + desc: 'TO=up: rename every file of RUN/labels.txt but the FRESH_KEY file out of STAGING into RUN/up/N/, one tree per type, its type in RUN/up/N.type. TO=staging: rename them back' internal: true requires: {vars: [TO]} env: {FRESH_KEY: '{{.FRESH_KEY}}'} @@ -901,8 +801,7 @@ tasks: - (cd {{.STAGING}} && find . -type f | sed 's|^\./||' | LC_ALL=C sort) > {{.RUN}}/landed.txt - >- awk -F'\t' 'FILENAME == ARGV[1] { drop[$1] = 1; next } !($1 in drop)' {{.B}} {{.RUN}}/applied.txt - | cat - <(LC_ALL=C join -t "$(printf '\t')" {{.RUN}}/landed.txt {{.B}}) | LC_ALL=C sort > - {{.RUN}}/applied.new + | cat - <(LC_ALL=C join -t "$(printf '\t')" {{.RUN}}/landed.txt {{.B}}) | LC_ALL=C sort > {{.RUN}}/applied.new checkpoint: internal: true @@ -926,14 +825,11 @@ tasks: # errors. Thus, the command uses grep to find them. - >- for f in {{.RUN}}/del-*; do - awk '{{.DELETE_JSON}}' "$f" > "$f.json" && aws s3api delete-objects --bucket {{.BUCKET}} --delete - "file://$f.json" | tee -a {{.RUN}}/publish.txt + awk '{{.DELETE_JSON}}' "$f" > "$f.json" && aws s3api delete-objects --bucket {{.BUCKET}} --delete "file://$f.json" | tee -a {{.RUN}}/publish.txt | { test "$(grep -c Errors)" = 0; }; done delete: - desc: >- - Remove the keys in D (RUN/deleted.txt unless given), 1,000 per DeleteObjects call and free, - and drop them from the state + desc: Remove the keys in D (RUN/deleted.txt unless given), 1,000 per DeleteObjects call and free, and drop them from the state vars: # reconcile gives its orphan list. Thus, the deleted.txt of the run, which report # counts, does not change after diff writes it. @@ -948,9 +844,7 @@ tasks: - {task: checkpoint, vars: {B: '{{.D}}'}} reconcile: - desc: >- - When due left .run/reconcile, rebuild the state from a bucket listing, delete keys that - neither upstream nor .state/ owns, and record the reconcile + desc: When due left .run/reconcile, rebuild the state from a bucket listing, delete keys that neither upstream nor .state/ owns, and record the reconcile # Not while a batch waits for the next run. Then delete keeps the orphans. Thus, the # run stays due, and the chained run that puts the last batch in the bucket # reconciles. A run that has a failure before the last line also records nothing. @@ -974,13 +868,10 @@ tasks: # reconcile keeps each key that ends in "/". - >- {{if .INDEX}}cat {{.RUN}}/applied.txt {{.RUN}}/upstream-dirs.txt | {{.ANCESTORS}} - | awk '$0 != "" { print substr($0, 1, length($0) - 1) }' | LC_ALL=C sort > - {{.RUN}}/index-keys.txt{{else}}: > {{.RUN}}/index-keys.txt{{end}} + | awk '$0 != "" { print substr($0, 1, length($0) - 1) }' | LC_ALL=C sort > {{.RUN}}/index-keys.txt{{else}}: > {{.RUN}}/index-keys.txt{{end}} - >- - cut -f1 {{.RUN}}/bucket.txt | grep -vE '^\.state/'{{if .OWN}} | grep -vxF -f <(printf '%s\n' - {{.OWN}}){{end}} - {{if .INDEX}}| awk -v i="/{{.INDEX}}" '{ s = "/" $0 } substr(s, length(s) - length(i) + 1) != - i'{{end}} + cut -f1 {{.RUN}}/bucket.txt | grep -vE '^\.state/'{{if .OWN}} | grep -vxF -f <(printf '%s\n' {{.OWN}}){{end}} + {{if .INDEX}}| awk -v i="/{{.INDEX}}" '{ s = "/" $0 } substr(s, length(s) - length(i) + 1) != i'{{end}} | LC_ALL=C comm -23 - {{.RUN}}/index-keys.txt | LC_ALL=C comm -23 - {{.RUN}}/paths.txt > {{.RUN}}/orphans.txt - {task: delete, vars: {D: '{{.RUN}}/orphans.txt'}} @@ -988,10 +879,7 @@ tasks: # ---- after the bytes moved ---- index: - desc: >- - Hook. With INDEX set: redraw the directory pages this run changed and publish them; INDEXED, - the stamp and the page state the pages last showed, moves forward only after they landed. A - mirror with a landing page of its own replaces it + desc: 'Hook. With INDEX set: redraw the directory pages this run changed and publish them; INDEXED, the stamp and the page state the pages last showed, moves forward only after they landed. A mirror with a landing page of its own replaces it' status: ['test -z "{{.INDEX}}"'] set: [pipefail] vars: @@ -1000,17 +888,14 @@ tasks: # With no INDEXED, there is no page. Then index makes a page for each directory one # time (for the 27k directories of ctan, this is approximately ten minutes). After # that, the file is there. - - task: pull - vars: {KEY: '{{.INDEXED}}', OUT: '{{.RUN}}/indexed.txt', MISSING: ': > {{.RUN}}/indexed.txt'} + - {task: pull, vars: {KEY: '{{.INDEXED}}', OUT: '{{.RUN}}/indexed.txt', MISSING: ': > {{.RUN}}/indexed.txt'}} - {task: pages} - {task: index-same} # At this point, staging holds only pages, because checkpoint removed the files. The # pages stay for smoke. The split verb of the next run removes them. The verb does # not append them to publish.txt, because smoke compares the size of that sample # with upstream, which has no pages. - - >- - test ! -s {{.RUN}}/index-make.txt || - aws s3 cp {{.AWS_FLAGS}} --recursive --content-type 'text/html; charset=utf-8' {{.STAGING}}/ {{.S3}}/ + - test ! -s {{.RUN}}/index-make.txt || aws s3 cp {{.AWS_FLAGS}} --recursive --content-type 'text/html; charset=utf-8' {{.STAGING}}/ {{.S3}}/ # One upload for each depth, because no tree can hold a/b and a/b/ together. A # slashless key has no suffix that the CLI can use. Thus, the command gives the # type. Without it, the CLI sends binary/octet-stream, and a browser downloads the @@ -1028,9 +913,7 @@ tasks: # Not internal: the offline check of lib starts it with its name. index-copy: - desc: >- - Copy the index file of each directory in RUN/index-real.txt to the page key of that directory, - in the bucket + desc: Copy the index file of each directory in RUN/index-real.txt to the page key of that directory, in the bucket cmds: # The page key of a directory with an index file (INDEX_FILES) is one server-side # copy of that file. COPY keeps its Content-Type, and the copy has the same bytes @@ -1042,24 +925,17 @@ tasks: # not use AWS_FLAGS. - >- while IFS="$(printf '\t')" read -r s k; do - aws s3api copy-object --bucket {{.BUCKET}} --copy-source "{{.BUCKET}}/$s" --key "$k" - --metadata-directive COPY > /dev/null; + aws s3api copy-object --bucket {{.BUCKET}} --copy-source "{{.BUCKET}}/$s" --key "$k" --metadata-directive COPY > /dev/null; echo "copy: $s to $k"; done < {{.RUN}}/index-real.txt # Not internal: the offline check of lib starts it with its name. index-same: - desc: >- - When the stamp did not agree, read the ETag of each key in the bucket. Remove from staging and - from RUN/index-make.txt each directory whose staged keys have the same MD5; RUN/index-same.txt - gets these directories + desc: When the stamp did not agree, read the ETag of each key in the bucket. Remove from staging and from RUN/index-make.txt each directory whose staged keys have the same MD5; RUN/index-same.txt gets these directories # When the stamp agrees, pages makes only the pages of the changed directories, and the # listing costs more Class A operations than it saves. The listing gives 1,000 keys # for each request. For a bucket with 571k keys, this is 572 operations, against two # for each page. With no INDEXED, the bucket has no page. - status: - - >- - test ! -s {{.RUN}}/index-make.txt || test ! -s {{.RUN}}/indexed.txt || - head -1 {{.RUN}}/indexed.txt | cmp -s - {{.RUN}}/index-stamp.txt + status: ['test ! -s {{.RUN}}/index-make.txt || test ! -s {{.RUN}}/indexed.txt || head -1 {{.RUN}}/indexed.txt | cmp -s - {{.RUN}}/index-stamp.txt'] set: [pipefail] vars: SLASH: '{{.RUN}}/slash' @@ -1068,35 +944,26 @@ tasks: # multipart upload is different, and such a page goes up again. If the listing # fails, each page goes up. - >- - aws s3api list-objects-v2 --bucket {{.BUCKET}} --query 'Contents[].[Key,ETag]' --output text > - {{.RUN}}/bucket-etags.txt || + aws s3api list-objects-v2 --bucket {{.BUCKET}} --query 'Contents[].[Key,ETag]' --output text > {{.RUN}}/bucket-etags.txt || { echo "index-same: the listing failed; index sends each page"; : > {{.RUN}}/bucket-etags.txt; } # Each line of page-md5.txt is "md5 TAB p|s TAB key": p for a page key in staging, s # for a slashless key in a SLASH tree. - >- { (cd {{.STAGING}} && find . -type f -exec md5sum {} +) | sed 's# \./#\tp\t#'; - for d in {{.SLASH}}/*/; do test -d "$d" || continue; (cd "$d" && find . -type f -exec md5sum {} +) - | sed 's# \./#\ts\t#'; done; } + for d in {{.SLASH}}/*/; do test -d "$d" || continue; (cd "$d" && find . -type f -exec md5sum {} +) | sed 's# \./#\ts\t#'; done; } > {{.RUN}}/page-md5.txt # SAME_PAGES writes the files of the same directories. xargs removes them. - >- - awk -F'\t' -v E="{{.RUN}}/bucket-etags.txt" -v H="{{.RUN}}/page-md5.txt" -v I="{{.INDEX}}" -v - S="{{.STAGING}}/" -v R="{{.SLASH}}/" + awk -F'\t' -v E="{{.RUN}}/bucket-etags.txt" -v H="{{.RUN}}/page-md5.txt" -v I="{{.INDEX}}" -v S="{{.STAGING}}/" -v R="{{.SLASH}}/" -v Y="{{.RUN}}/index-same.txt" -v M="{{.RUN}}/index-make.new" '{{.SAME_PAGES}}' {{.RUN}}/bucket-etags.txt {{.RUN}}/page-md5.txt {{.RUN}}/index-make.txt | tr '\n' '\0' | xargs -0r rm -f - - >- - touch {{.RUN}}/index-same.txt {{.RUN}}/index-make.new && - mv {{.RUN}}/index-make.new {{.RUN}}/index-make.txt + - touch {{.RUN}}/index-same.txt {{.RUN}}/index-make.new && mv {{.RUN}}/index-make.new {{.RUN}}/index-make.txt pages: - desc: >- - RUN/indexed.txt against the page state (RUN/applied.txt and RUN/upstream-dirs.txt) -> a page - under both of its keys for every directory a changed line touches, or for every directory when - the stamp differs; RUN/index-gone.txt for the keys of directories that upstream removed; - RUN/index-real.txt for the directories with an index file; RUN/indexed.new for INDEXED + desc: RUN/indexed.txt against the page state (RUN/applied.txt and RUN/upstream-dirs.txt) -> a page under both of its keys for every directory a changed line touches, or for every directory when the stamp differs; RUN/index-gone.txt for the keys of directories that upstream removed; RUN/index-real.txt for the directories with an index file; RUN/indexed.new for INDEXED set: [pipefail] - env: &pagetext + env: INDEX_FILES: '{{.INDEX_FILES}}' UPSTREAM: '{{.UPSTREAM}}' CONTACT: '{{.CONTACT}}' @@ -1117,39 +984,12 @@ tasks: # PAGE_FOOT is a printf format. It can have one %s, for the encoded directory, and %% # for a %. A different directive takes the directory or prints nothing. Thus, this # command stops the run on such a PAGE_FOOT. awk reads the value from the - # environment, because awk -v changes each backslash escape. + # environment, because awk -v changes each backslash escape. PAGE_HTML reads it in + # the same way. Thus, the check and the footer read the same bytes. - >- awk 'BEGIN { g = ENVIRON["PAGE_FOOT"]; gsub(/%%/, "", g) - if (g !~ /^[^%]*(%s[^%]*)?$/) { - print "pages: PAGE_FOOT can have one %s, and %%, not " ENVIRON["PAGE_FOOT"] > "/dev/stderr" - exit 1 } }' - - >- - mkdir -p {{.STAGING}} {{.SLASH}} && find {{.STAGING}} {{.SLASH}} -mindepth 1 -delete && - : > {{.RUN}}/index-real.txt - # A task does not see the call vars of the task that starts it. Thus, pages gives its - # steps the vars that they read, with the anchor pages. In a run, these are the values - # that pages has. An offline check gives different values. - - task: pages-select - vars: &pages - RUN: '{{.RUN}}' - STAGING: '{{.STAGING}}' - SLASH: '{{.SLASH}}' - HOST: '{{.HOST}}' - INDEX: '{{.INDEX}}' - INDEX_FILES: '{{.INDEX_FILES}}' - UPSTREAM: '{{.UPSTREAM}}' - CONTACT: '{{.CONTACT}}' - PAGE_FOOT: '{{.PAGE_FOOT}}' - - {task: pages-draw, vars: *pages} - - pages-select: - desc: >- - The page state, the stamp and RUN/indexed.new; then RUN/index-gone.txt, the keys to remove, - and RUN/index-make.txt, the directories to draw - internal: true - set: [pipefail] - env: *pagetext - cmds: + if (g !~ /^[^%]*(%s[^%]*)?$/) { print "pages: PAGE_FOOT can have one %s, and %%, not " ENVIRON["PAGE_FOOT"] > "/dev/stderr"; exit 1 } }' + - 'mkdir -p {{.STAGING}} {{.SLASH}} && find {{.STAGING}} {{.SLASH}} -mindepth 1 -delete && : > {{.RUN}}/index-real.txt' # The page state is the state and a line "dir/ TAB TAB" for each directory of # upstream. Thus, a directory with no file also gets a page, and the page of its # parent shows it. @@ -1160,10 +1000,8 @@ tasks: # a page. INDEXED starts with the stamp of the pages that it records. If INDEXED has # no stamp line, the stamps do not agree. - >- - printf '%s\n' {{shellQuote .PAGE_FORMAT}} {{shellQuote .INDEX}} "$INDEX_FILES" "$PAGE_FOOT" - "$UPSTREAM" "$CONTACT" - {{shellQuote .HOST}} | sha256sum | awk '{ print "#pages " substr($1, 1, 16) }' > - {{.RUN}}/index-stamp.txt + printf '%s\n' {{shellQuote .PAGE_FORMAT}} {{shellQuote .INDEX}} "$INDEX_FILES" "$PAGE_FOOT" "$UPSTREAM" "$CONTACT" + {{shellQuote .HOST}} | sha256sum | awk '{ print "#pages " substr($1, 1, 16) }' > {{.RUN}}/index-stamp.txt - awk 'NR == 1 && /^#pages / { next } { print }' {{.RUN}}/indexed.txt > {{.RUN}}/index-old.txt # A page changes when a file in its directory, or a directory below it, is added, # changed or removed. Thus, the verb makes the page of each ancestor of each changed @@ -1177,8 +1015,7 @@ tasks: # the two files. Then make a parent page again only for a new child or a removed # child. - >- - { LC_ALL=C comm -3 {{.RUN}}/index-old.txt {{.RUN}}/page-state.txt | awk -F'\t' '{ print ($1 == "" ? - $2 : $1) }'; + { LC_ALL=C comm -3 {{.RUN}}/index-old.txt {{.RUN}}/page-state.txt | awk -F'\t' '{ print ($1 == "" ? $2 : $1) }'; head -1 {{.RUN}}/indexed.txt | cmp -s - {{.RUN}}/index-stamp.txt || cat {{.RUN}}/page-state.txt; } | {{.ANCESTORS}} | LC_ALL=C sort > {{.RUN}}/index-dirty.txt - '{{.ANCESTORS}} {{.RUN}}/page-state.txt | LC_ALL=C sort > {{.RUN}}/index-dirs.txt' @@ -1192,19 +1029,9 @@ tasks: - >- LC_ALL=C comm -23 {{.RUN}}/index-dirty.txt {{.RUN}}/index-dirs.txt | awk -v i="{{.INDEX}}" '{ print $0 i } $0 != "" { print substr($0, 1, length($0) - 1) }' - | awk -F'\t' 'FILENAME == ARGV[1] { f[$1] = 1; next } !($0 in f)' {{.RUN}}/page-state.txt - > - {{.RUN}}/index-gone.txt + | awk -F'\t' 'FILENAME == ARGV[1] { f[$1] = 1; next } !($0 in f)' {{.RUN}}/page-state.txt - > {{.RUN}}/index-gone.txt - LC_ALL=C comm -12 {{.RUN}}/index-dirty.txt {{.RUN}}/index-dirs.txt > {{.RUN}}/index-make.txt - - pages-draw: - desc: A page in STAGING for each directory of RUN/index-make.txt, and its slashless copy in SLASH - internal: true - set: [pipefail] - env: *pagetext - cmds: - - >- - test ! -s {{.RUN}}/index-make.txt || - awk -v s="{{.STAGING}}/" '{ print s $0 }' {{.RUN}}/index-make.txt | tr '\n' '\0' | xargs -0 mkdir -p + - test ! -s {{.RUN}}/index-make.txt || awk -v s="{{.STAGING}}/" '{ print s $0 }' {{.RUN}}/index-make.txt | tr '\n' '\0' | xargs -0 mkdir -p # SLASH// holds the slashless copies of the pages at that depth. All files in # one tree have the same number of components. Thus, no file is the parent directory # of a different file, and the upload can send the full tree. @@ -1228,48 +1055,12 @@ tasks: - >- test ! -s {{.RUN}}/index-make.txt || awk -F'\t' -v OFS='\t' '{{.PAGE_ENTRIES}}' {{.RUN}}/index-make.txt {{.RUN}}/page-state.txt - | LC_ALL=C sort -t "$(printf '\t')" -k1,1 -k2,2 -k3,3V | awk -F'\t' -v S="{{.STAGING}}/" -v - R="{{.SLASH}}/" -v I="{{.INDEX}}" - -v F="$PAGE_FOOT" -v N="$INDEX_FILES" -v X="{{.RUN}}/index-real.txt" -v U="$UPSTREAM" -v - C="$CONTACT" -v H="{{.HOST}}" - -v css={{shellQuote .PAGE_CSS}} '{{.PAGE_HTML}}' + | LC_ALL=C sort -t "$(printf '\t')" -k1,1 -k2,2 -k3,3V | awk -F'\t' -v S="{{.STAGING}}/" -v R="{{.SLASH}}/" -v I="{{.INDEX}}" + -v X="{{.RUN}}/index-real.txt" -v H="{{.HOST}}" -v css={{shellQuote .PAGE_CSS}} '{{.PAGE_HTML}}' smoke: - desc: >- - Read a key of every type this run published back through the domain, size against the listing - and type against the label; with TL set the tlpdb sha512 against the verified copy; with INDEX - set one redrawn page under both keys; with CANARY set the canary, as a Perl client, against - the bucket's copy; no Content-Encoding on a .tar.gz; warn of paths that never land; fresh; - then the mirror's own checks, smoke-mirror - cmds: - # A task does not see the call vars of the task that starts it. Thus, smoke gives its - # steps the vars that they read, with the anchor smoke. It gives fresh and - # smoke-mirror the three vars that a check reads. In a run, these are the values that - # smoke has. An offline check gives different values. - - task: smoke-sample - vars: &smoke - RUN: '{{.RUN}}' - STAGING: '{{.STAGING}}' - URL: '{{.URL}}' - TL: '{{.TL}}' - INDEX: '{{.INDEX}}' - CANARY: '{{.CANARY}}' - FRESH_KEY: '{{.FRESH_KEY}}' - - {task: smoke-tlpdb, vars: *smoke} - - {task: smoke-page, vars: *smoke} - - {task: smoke-canary, vars: *smoke} - - {task: smoke-encoding, vars: *smoke} - - {task: smoke-unlanded, vars: *smoke} - - {task: fresh, vars: {RUN: '{{.RUN}}', URL: '{{.URL}}'}} - - {task: smoke-mirror, vars: {RUN: '{{.RUN}}', STAGING: '{{.STAGING}}', URL: '{{.URL}}'}} - - smoke-sample: - desc: >- - The first key of each type that label gave this run, and the FRESH_KEY file when the state has - it, read back through the domain; size against the listing, and with http the type against the - label - internal: true - env: {FRESH_KEY: '{{.FRESH_KEY}}'} + desc: Read a key of every type this run published back through the domain, size against the listing and type against the label; with TL set the tlpdb sha512 against the verified copy; with INDEX set one redrawn page under both keys; with CANARY set the canary, as a Perl client, against the bucket's copy; no Content-Encoding on a .tar.gz; warn of paths that never land; fresh; then the mirror's own checks, smoke-mirror + env: {FRESH_KEY: '{{.FRESH_KEY}}', CANARY: '{{.CANARY}}'} cmds: # The sample is the first key of each type that label gave in this run. The command # reads each key again with its encoded URL, and finds its line in the listing with @@ -1286,8 +1077,7 @@ tasks: # # A key in the sample can contain a space. All other keys can also contain a space. - >- - { {{if .FRESH_KEY}}awk -F'\t' -v k="$FRESH_KEY" '$1 == k { print $1 "\t" }' {{.RUN}}/applied.txt - 2>/dev/null || true;{{end}} + { {{if .FRESH_KEY}}awk -F'\t' '$1 == ENVIRON["FRESH_KEY"] { print $1 "\t" }' {{.RUN}}/applied.txt 2>/dev/null || true;{{end}} awk -F'\t' '!seen[$2]++' {{.RUN}}/labeled.txt 2>/dev/null; } | while IFS="$(printf '\t')" read -r k t; do u=$(printf '%s\n' "$k" | {{.ENCODE}} | cut -f2); @@ -1295,31 +1085,15 @@ tasks: h=$({{.HEADERS}} "{{.URL}}/$u"); got=$(printf '%s\n' "$h" | {{.SIZE}}); ct=$(printf '%s\n' "$h" | {{.CTYPE}}); - echo "$k: $got bytes (listing: $want), served as ${ct:-no type} (label: ${t:-none})"; test "$got" = - "$want" || exit 1; + echo "$k: $got bytes (listing: $want), served as ${ct:-no type} (label: ${t:-none})"; test "$got" = "$want" || exit 1; case "{{.URL}}" in http*) test -z "$t" || test "$ct" = "$t" || exit 1 ;; esac; done - - smoke-tlpdb: - desc: >- - With TL set and a tlpdb that prepare verified, the tlpdb sha512 read back through the domain - against the verified copy - internal: true - cmds: # The tlpdb key is in the decision batch, the last batch. Thus, a run that stopped # at MAX_BATCHES did not publish the copy of this hour, and it has nothing to # compare. - >- {{if .TL}}test ! -f {{.RUN}}/tl/tlpkg/texlive.tlpdb.sha512 || test -f {{.RUN}}/chain || - {{.CURL}} {{.URL}}/{{.TL}}/tlpkg/texlive.tlpdb.sha512 | cmp - - {{.RUN}}/tl/tlpkg/texlive.tlpdb.sha512{{else}}echo "no TL subtree: no tlpdb to read back"{{end}} - - smoke-page: - desc: >- - With INDEX set, the last directory of RUN/index-make.txt read back through the domain at its - page key, and with http at its slashless key - internal: true - cmds: + {{.CURL}} {{.URL}}/{{.TL}}/tlpkg/texlive.tlpdb.sha512 | cmp - {{.RUN}}/tl/tlpkg/texlive.tlpdb.sha512{{else}}echo "no TL subtree: no tlpdb to read back"{{end}} # This command reads one of the pages that this run made, at its two keys, with each # key in the URL (no zone rule is necessary). r gives the size and the type of a # key, with the same ranged read of one byte as the sample. @@ -1340,24 +1114,13 @@ tasks: echo "$(printf '%s\n' "$h" | {{.SIZE}}) bytes, type $(printf '%s\n' "$h" | {{.CTYPE}})"; }; s=$(awk -F'\t' -v k="${d}{{.INDEX}}" '$2 == k { print $1 }' {{.RUN}}/index-real.txt); if test -n "$s"; then want=$(r "$s"); - else want=$(wc -c < "{{.STAGING}}/${d}{{.INDEX}}" | tr -d ' '); test -n "$want"; want="$want - bytes"; fi; + else want=$(wc -c < "{{.STAGING}}/${d}{{.INDEX}}" | tr -d ' '); test -n "$want"; want="$want bytes"; fi; got=$(r "${d}{{.INDEX}}"); test -n "$s" || got=${got%%,*}; echo "${d}{{.INDEX}}: $got (${s:-page}: $want)"; test "$got" = "$want"; case "{{.URL}}" in http*) test -z "$d" || { p=${d%/}; - want=$(wc -c < "{{.RUN}}/slash/$(printf '%s\n' "$p" | awk -F/ '{ print NF }')/$p" | tr -d ' '); - test -n "$want"; - got=$(r "$p"); got=${got%%,*}; echo "$p: $got (listing: $want bytes)"; test "$got" = "$want bytes"; - } ;; esac; + want=$(wc -c < "{{.RUN}}/slash/$(printf '%s\n' "$p" | awk -F/ '{ print NF }')/$p" | tr -d ' '); test -n "$want"; + got=$(r "$p"); got=${got%%,*}; echo "$p: $got (listing: $want bytes)"; test "$got" = "$want bytes"; } ;; esac; }{{else}}echo "no INDEX: no directory pages to read back"{{end}} - - smoke-canary: - desc: >- - With CANARY set, in the state and over http, the canary read as a Perl client against the - bucket's copy - internal: true - env: {CANARY: '{{.CANARY}}'} - cmds: # The HTML rewriters of Cloudflare are between R2 and the client. Thus, the check # compares the object with the file that the domain sends. A size check is not # sufficient. Automatic HTTPS Rewrites can make a link one byte longer, while the @@ -1372,19 +1135,12 @@ tasks: # The command does not do this check with file://, which has no bucket to read. It # also does not do it before the state holds the path. - >- - {{if .CANARY}}h="$CANARY"; n=$(awk -F'\t' -v k="$h" '$1 == k { print $2 }' {{.RUN}}/applied.txt - 2>/dev/null); + {{if .CANARY}}h="$CANARY"; n=$(awk -F'\t' '$1 == ENVIRON["CANARY"] { print $2 }' {{.RUN}}/applied.txt 2>/dev/null); case "{{.URL}}" in http*) ;; *) n= ;; esac; if test -z "$n"; then echo "canary skipped ($h is not in the state, or the URL is not http)"; else aws s3 cp {{.AWS_FLAGS}} "{{.S3}}/$h" {{.RUN}}/canary.html && {{.CURL}} -A "libwww-perl/6.67" "{{.URL}}/$h" | cmp - {{.RUN}}/canary.html && - echo "$h: $n bytes, served to a libwww-perl client as the bucket holds them"; fi{{else}}echo "no - CANARY: the edge's HTML rewriters and client filter go unchecked"{{end}} - - smoke-encoding: - desc: Over http, the first non-empty .tar.gz of the state read back with no Content-Encoding - internal: true - cmds: + echo "$h: $n bytes, served to a libwww-perl client as the bucket holds them"; fi{{else}}echo "no CANARY: the edge's HTML rewriters and client filter go unchecked"{{end}} # A Content-Encoding on an object tells each client to decompress the file during # the download. Thus, the client gets a tarball as a tar. The mirror guidelines of # GNU recommend no Content-Encoding. For foo.tar.gz, guess_type in Python gives @@ -1398,19 +1154,10 @@ tasks: - >- g=$(awk -F'\t' '$1 ~ /\.tar\.gz$/ && $2 > 0 { print $1; exit }' {{.RUN}}/applied.txt 2>/dev/null); case "{{.URL}}" in http*) ;; *) g= ;; esac; - if test -z "$g"; then echo "content-encoding check skipped (no non-empty .tar.gz in the state, or - the URL is not http)"; else + if test -z "$g"; then echo "content-encoding check skipped (no non-empty .tar.gz in the state, or the URL is not http)"; else u=$(printf '%s\n' "$g" | {{.ENCODE}} | cut -f2) && h=$({{.HEADERS}} "{{.URL}}/$u") && - if printf '%s' "$h" | grep -qi '^content-encoding:'; then echo "$g: served with a content-encoding" - >&2; exit 1; fi && + if printf '%s' "$h" | grep -qi '^content-encoding:'; then echo "$g: served with a content-encoding" >&2; exit 1; fi && echo "$g: served with no content-encoding"; fi - - smoke-unlanded: - desc: >- - RUN/unlanded.txt, the lines of the delta that the state lacks after the last batch; the first - 20 as warnings on the run - internal: true - cmds: # A path in the listing of upstream that rsync cannot send (exit 23, which retry # accepts) does not go into the bucket. Thus, each run downloads it again, and the # run shows no failure. When the run did all the batches, the lines of the delta @@ -1418,16 +1165,16 @@ tasks: # first 20 is a warning on the Actions run. - >- if test -f {{.RUN}}/changed.txt && test ! -f {{.RUN}}/chain; - then LC_ALL=C comm -23 {{.RUN}}/changed.txt {{.RUN}}/applied.txt > {{.RUN}}/unlanded.txt; else : > - {{.RUN}}/unlanded.txt; fi; - cut -f1 {{.RUN}}/unlanded.txt | head -20 | while IFS= read -r p; do echo "::warning::listed - upstream, never fetched: $p"; done + then LC_ALL=C comm -23 {{.RUN}}/changed.txt {{.RUN}}/applied.txt > {{.RUN}}/unlanded.txt; else : > {{.RUN}}/unlanded.txt; fi; + cut -f1 {{.RUN}}/unlanded.txt | head -20 | while IFS= read -r p; do echo "::warning::listed upstream, never fetched: $p"; done + # A task does not see the call vars of the task that starts it. Thus, smoke gives + # the three vars that a check reads. In a run, they are the values that smoke has. + # An offline check gives different values. + - {task: fresh, vars: {RUN: '{{.RUN}}', URL: '{{.URL}}'}} + - {task: smoke-mirror, vars: {RUN: '{{.RUN}}', STAGING: '{{.STAGING}}', URL: '{{.URL}}'}} fresh: - desc: >- - With FRESH_KEY set and in the state, read it through the domain and fail the run when the - upstream clock in it is FRESH_HOURS (24) old; a source that stalls otherwise leaves every run - green + desc: With FRESH_KEY set and in the state, read it through the domain and fail the run when the upstream clock in it is FRESH_HOURS (24) old; a source that stalls otherwise leaves every run green env: {FRESH_KEY: '{{.FRESH_KEY}}'} cmds: # The run moved each byte that it got. Thus, only the clock that upstream wrote can @@ -1435,23 +1182,18 @@ tasks: # sync while its rsync daemon continues to operate. - >- {{if .FRESH_KEY}}k="$FRESH_KEY"; - if ! awk -F'\t' -v k="$k" '$1 == k { f = 1 } END { exit !f }' {{.RUN}}/applied.txt 2>/dev/null; + if ! awk -F'\t' '$1 == ENVIRON["FRESH_KEY"] { f = 1 } END { exit !f }' {{.RUN}}/applied.txt 2>/dev/null; then echo "freshness skipped ($k is not in the state)"; else - h={{shellQuote .FRESH_H}}; case "$h" in {{.NOT_WHOLE}}) echo "fresh: FRESH_HOURS is a whole number, - not $h" >&2; exit 1 ;; esac; + h={{shellQuote .FRESH_H}}; case "$h" in {{.NOT_WHOLE}}) echo "fresh: FRESH_HOURS is a whole number, not $h" >&2; exit 1 ;; esac; u=$(printf '%s\n' "$k" | {{.ENCODE}} | cut -f2); t=$({{.CURL}} "{{.URL}}/$u" | {{.STAMP}}); case "$t" in *-*) t=$(date -u -d "$t" +%s) ;; esac; test -n "$t"; a=$(( $(date -u +%s) - t )); echo "$a" > {{.RUN}}/fresh.txt; echo "$k: upstream's clock $((a / 3600)) h $((a % 3600 / 60)) min ago, limit {{.FRESH_H}} h"; - test "$a" -le $(( {{.FRESH_H}} * 3600 )) || { echo "::error::$k is $((a / 3600)) h old, past - FRESH_HOURS {{.FRESH_H}}: upstream or SOURCE has stopped moving"; exit 1; }; fi{{else}}echo "no - FRESH_KEY: a source that stops moving goes unnoticed"{{end}} + test "$a" -le $(( {{.FRESH_H}} * 3600 )) || { echo "::error::$k is $((a / 3600)) h old, past FRESH_HOURS {{.FRESH_H}}: upstream or SOURCE has stopped moving"; exit 1; }; fi{{else}}echo "no FRESH_KEY: a source that stops moving goes unnoticed"{{end}} smoke-mirror: - desc: >- - A hook, empty here; a mirror with more to read back than the sample replaces it, and smoke - runs it last + desc: A hook, empty here; a mirror with more to read back than the sample replaces it, and smoke runs it last cmds: [] report-engine: @@ -1462,55 +1204,29 @@ tasks: - | n() { test -f "$1" && wc -l < "$1" || echo 0; } gb() { test -f "$1" && awk -F'\t' '{ s += $2 } END { printf "%.2f", s / 1e9 }' "$1" || echo 0.00; } - r={{.RUN}}; u={{.URL}}; out="${GITHUB_STEP_SUMMARY:-/dev/stdout}" - batches=$(ls $r/batch-*.txt 2>/dev/null | wc -l | awk -v m={{.BATCHES_MAX}} \ - '{ print $1 " batches" ($1 > m ? ", " $1 - m " left for the next run" : "") }') - deleted=$(n $r/deleted.txt); orphaned=$(n $r/orphans.txt) - uploaded=$(awk '/^upload:/ { n++ } END { print n + 0 }' $r/publish.txt 2>/dev/null || echo 0) - types=$(cut -f2 $r/labeled.txt 2>/dev/null | sort -u | wc -l | tr -d ' ') - storage="not measured this run" - test ! -f $r/bucket.txt || storage="$(gb $r/bucket.txt) GB in $(n $r/bucket.txt) objects (reconciled)" + out="${GITHUB_STEP_SUMMARY:-/dev/stdout}" cat >> "$out" </dev/null | wc -l | awk -v m={{.BATCHES_MAX}} '{ print $1 " batches" ($1 > m ? ", " $1 - m " left for the next run" : "") }'); $(n {{.RUN}}/deleted.txt) deleted, $(n {{.RUN}}/orphans.txt) orphaned | + | Published to R2 | $(awk '/^upload:/ { n++ } END { print n + 0 }' {{.RUN}}/publish.txt 2>/dev/null || echo 0) uploaded, $(cut -f2 {{.RUN}}/labeled.txt 2>/dev/null | sort -u | wc -l | tr -d ' ') types | + | State | $(n {{.RUN}}/applied.txt) lines in {{.STATE}} | + | Storage | $(test -f {{.RUN}}/bucket.txt && echo "$(gb {{.RUN}}/bucket.txt) GB in $(n {{.RUN}}/bucket.txt) objects (reconciled)" || echo "not measured this run"){{if .CEILING_GB}} of the {{.CEILING_GB}} GB ceiling{{end}} | EOF # prepare made RUN/tl. Thus, the delta had a path in the signed subtree. - if test -d $r/tl; then - s="the tlpdb did not verify" - test ! -f $r/tl/tlpkg/texlive.tlpdb || - s="tlpdb signed by {{.TL_KEY}}; $(n $r/tl/missing.txt) named containers missing from the bucket" - echo "| Signature | $s |" >> "$out" - fi - # With INDEX set, the pages that index made and the keys that it removed. echo puts one - # space between two arguments. - test -z "{{.INDEX}}" || echo "| Directory pages | $(n $r/index-make.txt) redrawn and sent," \ - "$(n $r/index-same.txt) redrawn and not sent (the bucket had the same bytes)," \ - "$(n $r/index-real.txt) with a copy of an index file as the page key," \ - "$(n $r/index-gone.txt) keys removed |" >> "$out" + test ! -d {{.RUN}}/tl || echo "| Signature | $(test -f {{.RUN}}/tl/tlpkg/texlive.tlpdb && echo "tlpdb signed by {{.TL_KEY}}; $(n {{.RUN}}/tl/missing.txt) named containers missing from the bucket" || echo "the tlpdb did not verify") |" >> "$out" + # With INDEX set, the pages that index made and the keys that it removed. + test -z "{{.INDEX}}" || echo "| Directory pages | $(n {{.RUN}}/index-make.txt) redrawn and sent, $(n {{.RUN}}/index-same.txt) redrawn and not sent (the bucket had the same bytes), $(n {{.RUN}}/index-real.txt) with a copy of an index file as the page key, $(n {{.RUN}}/index-gone.txt) keys removed |" >> "$out" # Rows that show only when they have data. - test ! -s $r/unlanded.txt || echo "| Never landed | $(wc -l < $r/unlanded.txt | tr -d ' ') listed" \ - "upstream that rsync could not send, fetched again every run; the first are warnings on" \ - "the run |" >> "$out" - test ! -f $r/fresh.txt || echo "| Upstream clock |" \ - "$(awk '{ printf "%d h %d min", $1 / 3600, $1 % 3600 / 60 }' $r/fresh.txt) old in $FRESH_KEY," \ - "limit {{.FRESH_H}} h |" >> "$out" + test ! -s {{.RUN}}/unlanded.txt || echo "| Never landed | $(wc -l < {{.RUN}}/unlanded.txt | tr -d ' ') listed upstream that rsync could not send, fetched again every run; the first are warnings on the run |" >> "$out" + test ! -f {{.RUN}}/fresh.txt || echo "| Upstream clock | $(awk '{ printf "%d h %d min", $1 / 3600, $1 % 3600 / 60 }' {{.RUN}}/fresh.txt) old in $FRESH_KEY, limit {{.FRESH_H}} h |" >> "$out" retry: - desc: >- - Run CMD, retrying rsync's transport exit codes (5 10 12 30 35) five times with backoff and - jitter; 23 (a path it could not read, a dangling symlink under -L) and 24 (file vanished - mid-transfer) are successes + desc: Run CMD, retrying rsync's transport exit codes (5 10 12 30 35) five times with backoff and jitter; 23 (a path it could not read, a dangling symlink under -L) and 24 (file vanished mid-transfer) are successes cmds: - | for i in 1 2 3 4 5; do - # An exit in CMD stops only the subshell, not the loop. Because of ||, the errexit - # of Task does not stop the loop. - rc=0; ( {{.CMD}} ) || rc=$? + rc=0; ( {{.CMD}} ) || rc=$? # subshell: an exit in CMD cannot stop the loop. ||: the errexit of Task cannot stop it. case $rc in 0|23|24) exit 0;; 5|10|12|30|35) ;; *) exit $rc;; esac test $i -lt 5 || break - s=$(({{.RETRY_S}} * 2 ** i + RANDOM % (2 * {{.RETRY_S}} + 1))) - echo "rsync exit $rc, retry $i in ${s}s" >&2; sleep $s + s=$(({{.RETRY_S}} * 2 ** i + RANDOM % (2 * {{.RETRY_S}} + 1))); echo "rsync exit $rc, retry $i in ${s}s" >&2; sleep $s done; exit $rc diff --git a/examples/rsync/Taskfile.yml b/examples/rsync/Taskfile.yml index d337545..a820110 100644 --- a/examples/rsync/Taskfile.yml +++ b/examples/rsync/Taskfile.yml @@ -395,6 +395,14 @@ tasks: printf '%s\t1\t-\n' new.txt old.txt timestamp > $r/applied.txt - {task: fresh, vars: {RUN: '{{.F}}/run-fresh', URL: 'file://{{.F}}/run-fresh/srv', FRESH_KEY: new.txt}} - {task: fresh, vars: {RUN: '{{.F}}/run-fresh', URL: 'file://{{.F}}/run-fresh/srv', FRESH_KEY: timestamp}} + # fresh gets FRESH_KEY with no change. A quotation mark, a $, a backslash escape and a + # non-ASCII character in the name do not change a command. fresh writes fresh.txt + # only when it reads the clock. + - | + r={{.F}}/run-fresh; k='a"b$c\nd é.txt'; rm -f $r/fresh.txt + echo $(($(date -u +%s) - 3600)) > "$r/srv/$k"; printf '%s\t1\t-\n' "$k" >> $r/applied.txt + - {task: fresh, vars: {RUN: '{{.F}}/run-fresh', URL: 'file://{{.F}}/run-fresh/srv', FRESH_KEY: 'a"b$c\nd é.txt'}} + - test -s {{.F}}/run-fresh/fresh.txt - '! task fresh-refuses 2>/dev/null' # fresh refuses a FRESH_HOURS with a 0 first. Shell arithmetic reads 010 as 8 hours. - task fresh-hours H=010 2>&1 | grep -q 'FRESH_HOURS is a whole number, not 010' diff --git a/examples/rsync/render.txt b/examples/rsync/render.txt index 207818c..9508493 100644 --- a/examples/rsync/render.txt +++ b/examples/rsync/render.txt @@ -23,13 +23,10 @@ echo "due: RECONCILE=$r RECONCILE_HOURS=$h, last reconcile $ago: $(test -f /work task: [list] rm -f /work/examples/rsync/.run/publish.txt /work/examples/rsync/.run/labeled.txt /work/examples/rsync/.run/orphans.txt /work/examples/rsync/.run/unlanded.txt /work/examples/rsync/.run/fresh.txt /work/examples/rsync/.run/index-same.txt task: [list] mkdir -p /work/examples/rsync/.run /work/examples/rsync/staging task: [retry] for i in 1 2 3 4 5; do - # An exit in CMD stops only the subshell, not the loop. Because of ||, the errexit - # of Task does not stop the loop. - rc=0; ( TZ=UTC rsync --timeout=300 --contimeout=60 --no-h -8 -rL --list-only rsync://rsync.example.org/pub/ > /work/examples/rsync/.run/listing.txt ) || rc=$? + rc=0; ( TZ=UTC rsync --timeout=300 --contimeout=60 --no-h -8 -rL --list-only rsync://rsync.example.org/pub/ > /work/examples/rsync/.run/listing.txt ) || rc=$? # subshell: an exit in CMD cannot stop the loop. ||: the errexit of Task cannot stop it. case $rc in 0|23|24) exit 0;; 5|10|12|30|35) ;; *) exit $rc;; esac test $i -lt 5 || break - s=$((15 * 2 ** i + RANDOM % (2 * 15 + 1))) - echo "rsync exit $rc, retry $i in ${s}s" >&2; sleep $s + s=$((15 * 2 ** i + RANDOM % (2 * 15 + 1))); echo "rsync exit $rc, retry $i in ${s}s" >&2; sleep $s done; exit $rc task: [normalise] awk '/^-/ && match($0, / [0-9][0-9][0-9][0-9]\/[0-9][0-9]\/[0-9][0-9] [0-9][0-9]:[0-9][0-9]:[0-9][0-9] /) { @@ -62,8 +59,7 @@ task: [split] awk -F'\t' '{ s += $2 } END { printf "upstream: %d objects, %.2f G task: [split] mkdir -p /work/examples/rsync/staging && find /work/examples/rsync/staging -mindepth 1 -delete task: [split] df -Pk /work/examples/rsync/staging task: [split] awk -F'\t' -v kb="$(df -Pk /work/examples/rsync/staging | awk 'NR == 2 { print $4 }')" \ - '$2 / 1024 > kb - 1048576 { print "no room for " $1 " (" $2 " bytes)"; bad = 1 } - END { exit bad }' \ + '$2 / 1024 > kb - 1048576 { print "no room for " $1 " (" $2 " bytes)"; bad = 1 } END { exit bad }' \ /work/examples/rsync/.run/changed.txt task: [split] rm -f /work/examples/rsync/.run/batch-*.txt task: [split] awk -F'\t' -v OFS='\t' -v CAP=$((4 * 1000000000)) -v DIR=/work/examples/rsync/.run -v TL="" 'index($1, TL "/tlpkg/") == 1 || $1 !~ /\// { dec[++nd] = $0; next } @@ -77,13 +73,10 @@ function flush( i, f) { if (!nb) return; f = name(++n) task: [prepare] mkdir -p /work/examples/rsync/.run/tl task: [prepare] printf '%s\n' tlpkg/texlive.tlpdb.xz tlpkg/texlive.tlpdb.sha512 tlpkg/texlive.tlpdb.sha512.asc tlpkg/gpg/pubring.gpg > /work/examples/rsync/.run/tl/files.txt task: [retry] for i in 1 2 3 4 5; do - # An exit in CMD stops only the subshell, not the loop. Because of ||, the errexit - # of Task does not stop the loop. - rc=0; ( rsync --timeout=300 --contimeout=60 --no-h -Lt --files-from=/work/examples/rsync/.run/tl/files.txt rsync://rsync.example.org/pub// /work/examples/rsync/.run/tl/ ) || rc=$? + rc=0; ( rsync --timeout=300 --contimeout=60 --no-h -Lt --files-from=/work/examples/rsync/.run/tl/files.txt rsync://rsync.example.org/pub// /work/examples/rsync/.run/tl/ ) || rc=$? # subshell: an exit in CMD cannot stop the loop. ||: the errexit of Task cannot stop it. case $rc in 0|23|24) exit 0;; 5|10|12|30|35) ;; *) exit $rc;; esac test $i -lt 5 || break - s=$((15 * 2 ** i + RANDOM % (2 * 15 + 1))) - echo "rsync exit $rc, retry $i in ${s}s" >&2; sleep $s + s=$((15 * 2 ** i + RANDOM % (2 * 15 + 1))); echo "rsync exit $rc, retry $i in ${s}s" >&2; sleep $s done; exit $rc task: [prepare] xz -dkf /work/examples/rsync/.run/tl/tlpkg/texlive.tlpdb.xz @@ -91,27 +84,23 @@ task: [prepare] cd /work/examples/rsync/.run/tl/tlpkg && shasum -a 512 -c texliv task: [fetch] find /work/examples/rsync/staging -mindepth 1 -delete task: [fetch] cut -f1 > /work/examples/rsync/.run/files.txt task: [retry] for i in 1 2 3 4 5; do - # An exit in CMD stops only the subshell, not the loop. Because of ||, the errexit - # of Task does not stop the loop. - rc=0; ( rsync --timeout=300 --contimeout=60 --no-h -Lt --files-from=/work/examples/rsync/.run/files.txt --ignore-missing-args rsync://rsync.example.org/pub/ /work/examples/rsync/staging/ ) || rc=$? + rc=0; ( rsync --timeout=300 --contimeout=60 --no-h -Lt --files-from=/work/examples/rsync/.run/files.txt --ignore-missing-args rsync://rsync.example.org/pub/ /work/examples/rsync/staging/ ) || rc=$? # subshell: an exit in CMD cannot stop the loop. ||: the errexit of Task cannot stop it. case $rc in 0|23|24) exit 0;; 5|10|12|30|35) ;; *) exit $rc;; esac test $i -lt 5 || break - s=$((15 * 2 ** i + RANDOM % (2 * 15 + 1))) - echo "rsync exit $rc, retry $i in ${s}s" >&2; sleep $s + s=$((15 * 2 ** i + RANDOM % (2 * 15 + 1))); echo "rsync exit $rc, retry $i in ${s}s" >&2; sleep $s done; exit $rc -task: [verify-signed] cd /work/examples/rsync/staging && cut -f1 | grep -E '^/(tlpkg/texlive\.tlpdb|[^/]+)\.sha512$' | while IFS= read -r f; do (cd "$(dirname "$f")" && f=$(basename "$f") && shasum -a 512 -c "$f" && gpgv --status-fd 1 --keyring /work/examples/rsync/.run/tl/tlpkg/gpg/pubring.gpg "$f.asc" "$f" | awk -v k= '/^\[GNUPG:\] GOODSIG /{g=1} /^\[GNUPG:\] VALIDSIG / && $NF == k {v=1} END{exit !(g&&v)}') || exit 1; done -task: [verify-containers] cd /work/examples/rsync/staging && { test "$(grep -c '^/archive/' )" = 0 || { awk -F'\t' -v TL="" 'FILENAME == ARGV[1] { if (index($1, TL "/archive/") == 1) want[$1] = 1; next } +task: [verify] cd /work/examples/rsync/staging && cut -f1 | grep -E '^/(tlpkg/texlive\.tlpdb|[^/]+)\.sha512$' | while IFS= read -r f; do (cd "$(dirname "$f")" && f=$(basename "$f") && shasum -a 512 -c "$f" && gpgv --status-fd 1 --keyring /work/examples/rsync/.run/tl/tlpkg/gpg/pubring.gpg "$f.asc" "$f" | awk -v k= '/^\[GNUPG:\] GOODSIG /{g=1} /^\[GNUPG:\] VALIDSIG / && $NF == k {v=1} END{exit !(g&&v)}') || exit 1; done +task: [verify] cd /work/examples/rsync/staging && { test "$(grep -c '^/archive/' )" = 0 || { awk -F'\t' -v TL="" 'FILENAME == ARGV[1] { if (index($1, TL "/archive/") == 1) want[$1] = 1; next } /^name /{split($0, a, " "); n = a[2]} /^revision /{split($0, a, " "); r = a[2]} /^containerchecksum /{split($0, a, " "); emit(a[2], n)} /^doccontainerchecksum /{split($0, a, " "); emit(a[2], n ".doc")} /^srccontainerchecksum /{split($0, a, " "); emit(a[2], n ".source")} -function emit(sum, base, p, q) { - p = TL "/archive/" base ".tar.xz"; q = TL "/archive/" base ".r" r ".tar.xz" +function emit(sum, base, p, q) { p = TL "/archive/" base ".tar.xz"; q = TL "/archive/" base ".r" r ".tar.xz" if (p in want) { print sum " " p; delete want[p] } if (q in want) { print sum " " q; delete want[q] } } END { for (p in want) { print "no checksum in the tlpdb for " p > "/dev/stderr"; bad = 1 } exit bad }' /work/examples/rsync/.run/tl/tlpkg/texlive.tlpdb > /work/examples/rsync/.run/tl/sums.txt && shasum -a 512 -c --quiet /work/examples/rsync/.run/tl/sums.txt; }; } -task: [verify-decision] cd /work/examples/rsync/staging && { test ! -f /tlpkg/texlive.tlpdb || { cmp /tlpkg/texlive.tlpdb /work/examples/rsync/.run/tl/tlpkg/texlive.tlpdb && xz -dc /tlpkg/texlive.tlpdb.xz | cmp - /tlpkg/texlive.tlpdb && { cut -f1 /work/examples/rsync/.run/applied.txt; cut -f1 ; } | LC_ALL=C sort -u | LC_ALL=C comm -23 - /work/examples/rsync/.run/deleted.txt > /work/examples/rsync/.run/tl/after.txt && awk -v TL="" '/^name /{n=$2} +task: [verify] cd /work/examples/rsync/staging && { test ! -f /tlpkg/texlive.tlpdb || { cmp /tlpkg/texlive.tlpdb /work/examples/rsync/.run/tl/tlpkg/texlive.tlpdb && xz -dc /tlpkg/texlive.tlpdb.xz | cmp - /tlpkg/texlive.tlpdb && { cut -f1 /work/examples/rsync/.run/applied.txt; cut -f1 ; } | LC_ALL=C sort -u | LC_ALL=C comm -23 - /work/examples/rsync/.run/deleted.txt > /work/examples/rsync/.run/tl/after.txt && awk -v TL="" '/^name /{n=$2} /^containerchecksum /{print TL "/archive/" n ".tar.xz"} /^doccontainerchecksum /{print TL "/archive/" n ".doc.tar.xz"} /^srccontainerchecksum /{print TL "/archive/" n ".source.tar.xz"}' /tlpkg/texlive.tlpdb | LC_ALL=C sort | LC_ALL=C comm -23 - /work/examples/rsync/.run/tl/after.txt > /work/examples/rsync/.run/tl/missing.txt && test ! -s /work/examples/rsync/.run/tl/missing.txt; }; } @@ -237,7 +226,7 @@ else: shutil.rmtree(up, ignore_errors=True) PY -task: [publish] test ! -f /work/examples/rsync/staging/"$FRESH_KEY" || aws s3 cp --no-progress --cli-connect-timeout 60 --cli-read-timeout 300 --content-type "$(awk -F'\t' -v k="$FRESH_KEY" '$1 == k { print $2 }' /work/examples/rsync/.run/labels.txt)" /work/examples/rsync/staging/"$FRESH_KEY" s3://example/"$FRESH_KEY" | tee -a /work/examples/rsync/.run/publish.txt +task: [publish] test ! -f /work/examples/rsync/staging/"$FRESH_KEY" || aws s3 cp --no-progress --cli-connect-timeout 60 --cli-read-timeout 300 --content-type "$(awk -F'\t' '$1 == ENVIRON["FRESH_KEY"] { print $2 }' /work/examples/rsync/.run/labels.txt)" /work/examples/rsync/staging/"$FRESH_KEY" s3://example/"$FRESH_KEY" | tee -a /work/examples/rsync/.run/publish.txt task: [merge] (cd /work/examples/rsync/staging && find . -type f | sed 's|^\./||' | LC_ALL=C sort) > /work/examples/rsync/.run/landed.txt task: [merge] awk -F'\t' 'FILENAME == ARGV[1] { drop[$1] = 1; next } !($1 in drop)' /work/examples/rsync/.run/applied.txt | cat - <(LC_ALL=C join -t "$(printf '\t')" /work/examples/rsync/.run/landed.txt ) | LC_ALL=C sort > /work/examples/rsync/.run/applied.new task: [push] xz -T0 -c /work/examples/rsync/.run/applied.new > /work/examples/rsync/.run/applied.new.xz @@ -262,8 +251,7 @@ task: [push] aws s3 cp --no-progress --cli-connect-timeout 60 --cli-read-timeout task: [rebuild] mv /work/examples/rsync/.run/applied.new /work/examples/rsync/.run/applied.txt task: [rebuild] rm -f /work/examples/rsync/.run/rebuild-now task: [reconcile] cat /work/examples/rsync/.run/applied.txt /work/examples/rsync/.run/upstream-dirs.txt | awk -F'\t' '!("" in s) { s[""] = 1; print "" } - { n = split($1, c, "/"); d = "" - for (i = 1; i < n; i++) { d = d c[i] "/"; if (!(d in s)) { s[d] = 1; print d } } }' | awk '$0 != "" { print substr($0, 1, length($0) - 1) }' | LC_ALL=C sort > /work/examples/rsync/.run/index-keys.txt + { n = split($1, c, "/"); d = ""; for (i = 1; i < n; i++) { d = d c[i] "/"; if (!(d in s)) { s[d] = 1; print d } } }' | awk '$0 != "" { print substr($0, 1, length($0) - 1) }' | LC_ALL=C sort > /work/examples/rsync/.run/index-keys.txt task: [reconcile] cut -f1 /work/examples/rsync/.run/bucket.txt | grep -vE '^\.state/' | awk -v i="/example.katoptra.org.directory.index.html" '{ s = "/" $0 } substr(s, length(s) - length(i) + 1) != i' | LC_ALL=C comm -23 - /work/examples/rsync/.run/index-keys.txt | LC_ALL=C comm -23 - /work/examples/rsync/.run/paths.txt > /work/examples/rsync/.run/orphans.txt task: [remove] rm -f /work/examples/rsync/.run/del-*; split -l 1000 -a 4 /work/examples/rsync/.run/orphans.txt /work/examples/rsync/.run/del- task: [remove] for f in /work/examples/rsync/.run/del-*; do awk 'BEGIN { printf "{\"Objects\":[" } { gsub(/\\/, "\\\\"); gsub(/"/, "\\\"") @@ -285,50 +273,42 @@ case $rc in esac task: [pages] awk 'BEGIN { g = ENVIRON["PAGE_FOOT"]; gsub(/%%/, "", g) - if (g !~ /^[^%]*(%s[^%]*)?$/) { - print "pages: PAGE_FOOT can have one %s, and %%, not " ENVIRON["PAGE_FOOT"] > "/dev/stderr" - exit 1 } }' + if (g !~ /^[^%]*(%s[^%]*)?$/) { print "pages: PAGE_FOOT can have one %s, and %%, not " ENVIRON["PAGE_FOOT"] > "/dev/stderr"; exit 1 } }' task: [pages] mkdir -p /work/examples/rsync/staging /work/examples/rsync/.run/slash && find /work/examples/rsync/staging /work/examples/rsync/.run/slash -mindepth 1 -delete && : > /work/examples/rsync/.run/index-real.txt -task: [pages-select] { cat /work/examples/rsync/.run/applied.txt; awk '{ print $0 "\t\t" }' /work/examples/rsync/.run/upstream-dirs.txt; } | LC_ALL=C sort > /work/examples/rsync/.run/page-state.txt -task: [pages-select] printf '%s\n' 3 example.katoptra.org.directory.index.html "$INDEX_FILES" "$PAGE_FOOT" "$UPSTREAM" "$CONTACT" example.katoptra.org | sha256sum | awk '{ print "#pages " substr($1, 1, 16) }' > /work/examples/rsync/.run/index-stamp.txt -task: [pages-select] awk 'NR == 1 && /^#pages / { next } { print }' /work/examples/rsync/.run/indexed.txt > /work/examples/rsync/.run/index-old.txt -task: [pages-select] { LC_ALL=C comm -3 /work/examples/rsync/.run/index-old.txt /work/examples/rsync/.run/page-state.txt | awk -F'\t' '{ print ($1 == "" ? $2 : $1) }'; head -1 /work/examples/rsync/.run/indexed.txt | cmp -s - /work/examples/rsync/.run/index-stamp.txt || cat /work/examples/rsync/.run/page-state.txt; } | awk -F'\t' '!("" in s) { s[""] = 1; print "" } - { n = split($1, c, "/"); d = "" - for (i = 1; i < n; i++) { d = d c[i] "/"; if (!(d in s)) { s[d] = 1; print d } } }' | LC_ALL=C sort > /work/examples/rsync/.run/index-dirty.txt -task: [pages-select] awk -F'\t' '!("" in s) { s[""] = 1; print "" } - { n = split($1, c, "/"); d = "" - for (i = 1; i < n; i++) { d = d c[i] "/"; if (!(d in s)) { s[d] = 1; print d } } }' /work/examples/rsync/.run/page-state.txt | LC_ALL=C sort > /work/examples/rsync/.run/index-dirs.txt -task: [pages-select] cat /work/examples/rsync/.run/index-stamp.txt /work/examples/rsync/.run/page-state.txt > /work/examples/rsync/.run/indexed.new -task: [pages-select] LC_ALL=C comm -23 /work/examples/rsync/.run/index-dirty.txt /work/examples/rsync/.run/index-dirs.txt | awk -v i="example.katoptra.org.directory.index.html" '{ print $0 i } $0 != "" { print substr($0, 1, length($0) - 1) }' | awk -F'\t' 'FILENAME == ARGV[1] { f[$1] = 1; next } !($0 in f)' /work/examples/rsync/.run/page-state.txt - > /work/examples/rsync/.run/index-gone.txt -task: [pages-select] LC_ALL=C comm -12 /work/examples/rsync/.run/index-dirty.txt /work/examples/rsync/.run/index-dirs.txt > /work/examples/rsync/.run/index-make.txt -task: [pages-draw] test ! -s /work/examples/rsync/.run/index-make.txt || awk -v s="/work/examples/rsync/staging/" '{ print s $0 }' /work/examples/rsync/.run/index-make.txt | tr '\n' '\0' | xargs -0 mkdir -p -task: [pages-draw] test ! -s /work/examples/rsync/.run/index-make.txt || awk -v r="/work/examples/rsync/.run/slash/" '$0 != "" { n = split($0, c, "/"); d = "" +task: [pages] { cat /work/examples/rsync/.run/applied.txt; awk '{ print $0 "\t\t" }' /work/examples/rsync/.run/upstream-dirs.txt; } | LC_ALL=C sort > /work/examples/rsync/.run/page-state.txt +task: [pages] printf '%s\n' 3 example.katoptra.org.directory.index.html "$INDEX_FILES" "$PAGE_FOOT" "$UPSTREAM" "$CONTACT" example.katoptra.org | sha256sum | awk '{ print "#pages " substr($1, 1, 16) }' > /work/examples/rsync/.run/index-stamp.txt +task: [pages] awk 'NR == 1 && /^#pages / { next } { print }' /work/examples/rsync/.run/indexed.txt > /work/examples/rsync/.run/index-old.txt +task: [pages] { LC_ALL=C comm -3 /work/examples/rsync/.run/index-old.txt /work/examples/rsync/.run/page-state.txt | awk -F'\t' '{ print ($1 == "" ? $2 : $1) }'; head -1 /work/examples/rsync/.run/indexed.txt | cmp -s - /work/examples/rsync/.run/index-stamp.txt || cat /work/examples/rsync/.run/page-state.txt; } | awk -F'\t' '!("" in s) { s[""] = 1; print "" } + { n = split($1, c, "/"); d = ""; for (i = 1; i < n; i++) { d = d c[i] "/"; if (!(d in s)) { s[d] = 1; print d } } }' | LC_ALL=C sort > /work/examples/rsync/.run/index-dirty.txt +task: [pages] awk -F'\t' '!("" in s) { s[""] = 1; print "" } + { n = split($1, c, "/"); d = ""; for (i = 1; i < n; i++) { d = d c[i] "/"; if (!(d in s)) { s[d] = 1; print d } } }' /work/examples/rsync/.run/page-state.txt | LC_ALL=C sort > /work/examples/rsync/.run/index-dirs.txt +task: [pages] cat /work/examples/rsync/.run/index-stamp.txt /work/examples/rsync/.run/page-state.txt > /work/examples/rsync/.run/indexed.new +task: [pages] LC_ALL=C comm -23 /work/examples/rsync/.run/index-dirty.txt /work/examples/rsync/.run/index-dirs.txt | awk -v i="example.katoptra.org.directory.index.html" '{ print $0 i } $0 != "" { print substr($0, 1, length($0) - 1) }' | awk -F'\t' 'FILENAME == ARGV[1] { f[$1] = 1; next } !($0 in f)' /work/examples/rsync/.run/page-state.txt - > /work/examples/rsync/.run/index-gone.txt +task: [pages] LC_ALL=C comm -12 /work/examples/rsync/.run/index-dirty.txt /work/examples/rsync/.run/index-dirs.txt > /work/examples/rsync/.run/index-make.txt +task: [pages] test ! -s /work/examples/rsync/.run/index-make.txt || awk -v s="/work/examples/rsync/staging/" '{ print s $0 }' /work/examples/rsync/.run/index-make.txt | tr '\n' '\0' | xargs -0 mkdir -p +task: [pages] test ! -s /work/examples/rsync/.run/index-make.txt || awk -v r="/work/examples/rsync/.run/slash/" '$0 != "" { n = split($0, c, "/"); d = "" for (i = 1; i < n - 1; i++) d = d c[i] "/"; print r (n - 1) "/" d }' /work/examples/rsync/.run/index-make.txt \ | LC_ALL=C sort -u | tr '\n' '\0' | xargs -0r mkdir -p -task: [pages-draw] test ! -s /work/examples/rsync/.run/index-make.txt || awk -F'\t' -v OFS='\t' 'FILENAME == ARGV[1] { want[$0] = 1; next } +task: [pages] test ! -s /work/examples/rsync/.run/index-make.txt || awk -F'\t' -v OFS='\t' 'FILENAME == ARGV[1] { want[$0] = 1; next } { n = split($1, c, "/"); d = "" - for (i = 1; i < n; i++) { - if ((d in want) && !((d c[i]) in seen)) { seen[d c[i]] = 1; print d, "d", c[i], "", "" } - d = d c[i] "/" } - if (d in want) print d, (c[n] == "" ? "e" : "f"), c[n], $2, $3 }' /work/examples/rsync/.run/index-make.txt /work/examples/rsync/.run/page-state.txt | LC_ALL=C sort -t "$(printf '\t')" -k1,1 -k2,2 -k3,3V | awk -F'\t' -v S="/work/examples/rsync/staging/" -v R="/work/examples/rsync/.run/slash/" -v I="example.katoptra.org.directory.index.html" -v F="$PAGE_FOOT" -v N="$INDEX_FILES" -v X="/work/examples/rsync/.run/index-real.txt" -v U="$UPSTREAM" -v C="$CONTACT" -v H="example.katoptra.org" -v css=':root{color-scheme:light dark;--bg:#f6f5f2;--fg:#1c1c28;--mut:#6e6e7e;--line:#e2e0da;--hov:#edebe5;--a:#3b45d9}@media (prefers-color-scheme:dark){:root{--bg:#121218;--fg:#f0f0f5;--mut:#8b8b9c;--line:#262630;--hov:#1a1a22;--a:#8b93ff}}body{margin:0 auto;max-width:60rem;padding:2rem 1rem 3rem;background:var(--bg);color:var(--fg);font:15px/1.45 system-ui,-apple-system,sans-serif}h1{font-size:1.35rem;font-weight:650;margin:0;overflow-wrap:anywhere}p{color:var(--mut);margin:.2rem 0 1.25rem;font-size:.9rem}table{width:100%;border-collapse:collapse;font:13.5px/1.4 ui-monospace,SFMono-Regular,Menlo,Consolas,monospace}th{font:600 .7rem/1 system-ui,sans-serif;letter-spacing:.06em;text-transform:uppercase;color:var(--mut);text-align:left;padding:0 .5rem .6rem;white-space:nowrap}td{padding:.4rem .5rem;border-top:1px solid var(--line);vertical-align:baseline}td:first-child{overflow-wrap:anywhere;width:100%}td+td{white-space:nowrap;color:var(--mut);font-variant-numeric:tabular-nums}th:last-child,td:last-child{text-align:right}tr:hover td{background:var(--hov)}a{color:var(--a);text-decoration:none}a:hover{text-decoration:underline}.s a{color:var(--mut)}.k{font-size:.72rem;margin-left:.6rem;padding:.05rem .4rem;border:1px solid var(--line);border-radius:.3rem;color:var(--mut)}.k:hover{color:var(--a);border-color:var(--a);text-decoration:none}footer{margin-top:2rem;padding-top:1rem;border-top:1px solid var(--line);color:var(--mut);font-size:.85rem;overflow-wrap:anywhere}footer a{color:inherit;text-decoration:underline}@media (max-width:36rem){thead{display:none}tr{display:grid;grid-template-columns:1fr auto;border-top:1px solid var(--line)}td{border:0;padding:.1rem .5rem}td:first-child{grid-column:1/-1;padding-top:.45rem}td+td{padding-bottom:.45rem;font-size:.8rem}}' 'BEGIN { for (i = 0; i < 256; i++) ord[sprintf("%c", i)] = i } + for (i = 1; i < n; i++) { if ((d in want) && !((d c[i]) in seen)) { seen[d c[i]] = 1; print d, "d", c[i], "", "" }; d = d c[i] "/" } + if (d in want) print d, (c[n] == "" ? "e" : "f"), c[n], $2, $3 }' /work/examples/rsync/.run/index-make.txt /work/examples/rsync/.run/page-state.txt | LC_ALL=C sort -t "$(printf '\t')" -k1,1 -k2,2 -k3,3V | awk -F'\t' -v S="/work/examples/rsync/staging/" -v R="/work/examples/rsync/.run/slash/" -v I="example.katoptra.org.directory.index.html" -v X="/work/examples/rsync/.run/index-real.txt" -v H="example.katoptra.org" -v css=':root{color-scheme:light dark;--bg:#f6f5f2;--fg:#1c1c28;--mut:#6e6e7e;--line:#e2e0da;--hov:#edebe5;--a:#3b45d9}@media (prefers-color-scheme:dark){:root{--bg:#121218;--fg:#f0f0f5;--mut:#8b8b9c;--line:#262630;--hov:#1a1a22;--a:#8b93ff}}body{margin:0 auto;max-width:60rem;padding:2rem 1rem 3rem;background:var(--bg);color:var(--fg);font:15px/1.45 system-ui,-apple-system,sans-serif}h1{font-size:1.35rem;font-weight:650;margin:0;overflow-wrap:anywhere}p{color:var(--mut);margin:.2rem 0 1.25rem;font-size:.9rem}table{width:100%;border-collapse:collapse;font:13.5px/1.4 ui-monospace,SFMono-Regular,Menlo,Consolas,monospace}th{font:600 .7rem/1 system-ui,sans-serif;letter-spacing:.06em;text-transform:uppercase;color:var(--mut);text-align:left;padding:0 .5rem .6rem;white-space:nowrap}td{padding:.4rem .5rem;border-top:1px solid var(--line);vertical-align:baseline}td:first-child{overflow-wrap:anywhere;width:100%}td+td{white-space:nowrap;color:var(--mut);font-variant-numeric:tabular-nums}th:last-child,td:last-child{text-align:right}tr:hover td{background:var(--hov)}a{color:var(--a);text-decoration:none}a:hover{text-decoration:underline}.s a{color:var(--mut)}.k{font-size:.72rem;margin-left:.6rem;padding:.05rem .4rem;border:1px solid var(--line);border-radius:.3rem;color:var(--mut)}.k:hover{color:var(--a);border-color:var(--a);text-decoration:none}footer{margin-top:2rem;padding-top:1rem;border-top:1px solid var(--line);color:var(--mut);font-size:.85rem;overflow-wrap:anywhere}footer a{color:inherit;text-decoration:underline}@media (max-width:36rem){thead{display:none}tr{display:grid;grid-template-columns:1fr auto;border-top:1px solid var(--line)}td{border:0;padding:.1rem .5rem}td:first-child{grid-column:1/-1;padding-top:.45rem}td+td{padding-bottom:.45rem;font-size:.8rem}}' 'BEGIN { for (i = 0; i < 256; i++) ord[sprintf("%c", i)] = i } function enc(s, out, j, c) { out = ""; for (j = 1; j <= length(s); j++) { c = substr(s, j, 1) out = out ((c ~ /[A-Za-z0-9\/._~-]/) ? c : sprintf("%%%02X", ord[c])) } return out } -BEGIN { n = split(N, nm, " "); for (i = n; i >= 1; i--) rank[nm[i]] = i +BEGIN { F = ENVIRON["PAGE_FOOT"]; N = ENVIRON["INDEX_FILES"]; U = ENVIRON["UPSTREAM"]; C = ENVIRON["CONTACT"] + n = split(N, nm, " "); for (i = n; i >= 1; i--) rank[nm[i]] = i G = "\\.(sig|sign|asc|gpg|dsig|proof|md5|md5sum|sha|sha1|sha224|sha256|sha256sum|sha512|sum|checksums)$" if (C == "" && (j = index(H, ".")) > 1) C = substr(H, 1, j - 1) "@" substr(H, j + 1) } -function esc(s) { gsub(/&/, "\\&", s); gsub(//, "\\>", s) - gsub(/"/, "\\"", s); return s } +function esc(s) { gsub(/&/, "\\&", s); gsub(//, "\\>", s); gsub(/"/, "\\"", s); return s } function grp(n, s, o) { s = sprintf("%.0f", n); o = "" - while (length(s) > 3) { o = "," substr(s, length(s) - 2) o; s = substr(s, 1, length(s) - 3) } - return s o } + while (length(s) > 3) { o = "," substr(s, length(s) - 2) o; s = substr(s, 1, length(s) - 3) } return s o } function iec(n, u, i) { if (n < 1024) return sprintf("%.0f B", n) split("KiB MiB GiB TiB", u, " "); n /= 1024; i = 1; while (n >= 1024 && i < 4) { n /= 1024; i++ } return sprintf(n < 10 ? "%.1f %s" : "%.0f %s", n, u[i]) } function ymd(t) { gsub(/\//, "-", t); return substr(t, 1, 16) } function foot( t) { if (F != "") return "
" sprintf(F, enc(cur)) "
" t = (U == "" ? "" : "Mirror of " esc(U) enc(cur) ".") - if (C != "") t = t (t == "" ? "" : " ") "Report a problem to " \ - "" esc(C) "." + if (C != "") t = t (t == "" ? "" : " ") "Report a problem to " esc(C) "." return (t == "" ? "" : "
" t "
") } function open(d) { cur = d; live = 1; real = ""; nd = 0; nfl = 0; tot = 0; split("", has) } function w(s) { if (real == "") printf("%s", s) > fa; if (cur != "") printf("%s", s) > fb } @@ -336,26 +316,19 @@ function shut( i, r, x, c, e, k, b) { split("", fold); split("", links) for (i = 1; i <= nfl; i++) { r = fname[i]; x = "" while (match(tolower(r), G)) { r = substr(r, 1, RSTART - 1); if (r in has) x = r } if (x != "") { fold[i] = 1 - links[x] = links[x] "" \ - esc(substr(fname[i], length(x) + 1)) "" } } + links[x] = links[x] "" esc(substr(fname[i], length(x) + 1)) "" } } c = (nd ? grp(nd) (nd == 1 ? " directory" : " directories") : "") if (nfl) c = c (c == "" ? "" : ", ") grp(nfl) (nfl == 1 ? " file" : " files") ", " iec(tot) if (c == "") c = "This directory is empty." fa = S cur I; if (real != "") print cur real "\t" cur I > X if (cur != "") { b = substr(cur, 1, length(cur) - 1); k = split(b, e, "/"); fb = R k "/" b } - w("\n" \ - "" \ - "Index of /" esc(cur) "\n") - w("\n\n" \ - "

Index of /" esc(cur) "

" c "

\n") + w("\nIndex of /" esc(cur) "\n") + w("\n\n

Index of /" esc(cur) "

" c "

\n") w("\n") if (cur != "") w("\n") - for (i = 1; i <= nd; i++) - w("\n") + for (i = 1; i <= nd; i++) w("\n") for (i = 1; i <= nfl; i++) if (!(i in fold)) - w("" \ - "\n") + w("\n") w("
NameModified (UTC)Size
../
" esc(dname[i]) "/
" esc(dname[i]) "/
" \ - esc(fname[i]) "" links[fname[i]] "" ymd(ftime[i]) "" iec(fsize[i]) "
" esc(fname[i]) "" links[fname[i]] "" ymd(ftime[i]) "" iec(fsize[i]) "
" foot() "\n") if (real == "") close(fa); if (cur != "") close(fb) } !live || $1 != cur { if (live) shut(); open($1) } @@ -367,13 +340,10 @@ END { if (live) shut() }' task: [index-same] aws s3api list-objects-v2 --bucket example --query 'Contents[].[Key,ETag]' --output text > /work/examples/rsync/.run/bucket-etags.txt || { echo "index-same: the listing failed; index sends each page"; : > /work/examples/rsync/.run/bucket-etags.txt; } task: [index-same] { (cd /work/examples/rsync/staging && find . -type f -exec md5sum {} +) | sed 's# \./#\tp\t#'; for d in /work/examples/rsync/.run/slash/*/; do test -d "$d" || continue; (cd "$d" && find . -type f -exec md5sum {} +) | sed 's# \./#\ts\t#'; done; } > /work/examples/rsync/.run/page-md5.txt task: [index-same] awk -F'\t' -v E="/work/examples/rsync/.run/bucket-etags.txt" -v H="/work/examples/rsync/.run/page-md5.txt" -v I="example.katoptra.org.directory.index.html" -v S="/work/examples/rsync/staging/" -v R="/work/examples/rsync/.run/slash/" -v Y="/work/examples/rsync/.run/index-same.txt" -v M="/work/examples/rsync/.run/index-make.new" 'FILENAME == E { gsub(/"/, "", $2); e[$1] = $2; next } -FILENAME == H { if ($2 == "s") d = $3 "/" - else if (substr($3, length($3) - length(I) + 1) == I) d = substr($3, 1, length($3) - length(I)) - else next +FILENAME == H { if ($2 == "s") d = $3 "/"; else if (substr($3, length($3) - length(I) + 1) == I) d = substr($3, 1, length($3) - length(I)); else next seen[d] = 1; if (e[$3] != $1) diff[d] = 1; next } !($0 in seen) || ($0 in diff) { print > M; next } -{ print > Y; print S $0 I - if ($0 != "") { b = substr($0, 1, length($0) - 1); print R split(b, c, "/") "/" b } }' /work/examples/rsync/.run/bucket-etags.txt /work/examples/rsync/.run/page-md5.txt /work/examples/rsync/.run/index-make.txt | tr '\n' '\0' | xargs -0r rm -f +{ print > Y; print S $0 I; if ($0 != "") { b = substr($0, 1, length($0) - 1); print R split(b, c, "/") "/" b } }' /work/examples/rsync/.run/bucket-etags.txt /work/examples/rsync/.run/page-md5.txt /work/examples/rsync/.run/index-make.txt | tr '\n' '\0' | xargs -0r rm -f task: [index-same] touch /work/examples/rsync/.run/index-same.txt /work/examples/rsync/.run/index-make.new && mv /work/examples/rsync/.run/index-make.new /work/examples/rsync/.run/index-make.txt task: [index] test ! -s /work/examples/rsync/.run/index-make.txt || aws s3 cp --no-progress --cli-connect-timeout 60 --cli-read-timeout 300 --recursive --content-type 'text/html; charset=utf-8' /work/examples/rsync/staging/ s3://example/ task: [index] for d in /work/examples/rsync/.run/slash/*/; do test -d "$d" || continue; aws s3 cp --no-progress --cli-connect-timeout 60 --cli-read-timeout 300 --recursive --content-type 'text/html; charset=utf-8' "$d" s3://example/; done @@ -383,25 +353,25 @@ task: [remove] for f in /work/examples/rsync/.run/del-*; do awk 'BEGIN { printf printf "%s{\"Key\":\"%s\"}", (NR > 1 ? "," : ""), $0 } END { print "],\"Quiet\":true}" }' "$f" > "$f.json" && aws s3api delete-objects --bucket example --delete "file://$f.json" | tee -a /work/examples/rsync/.run/publish.txt | { test "$(grep -c Errors)" = 0; }; done task: [push] xz -T0 -c /work/examples/rsync/.run/indexed.new > /work/examples/rsync/.run/indexed.new.xz task: [push] aws s3 cp --no-progress --cli-connect-timeout 60 --cli-read-timeout 300 /work/examples/rsync/.run/indexed.new.xz s3://example/.state/indexed.txt.xz -task: [smoke-sample] { awk -F'\t' -v k="$FRESH_KEY" '$1 == k { print $1 "\t" }' /work/examples/rsync/.run/applied.txt 2>/dev/null || true; awk -F'\t' '!seen[$2]++' /work/examples/rsync/.run/labeled.txt 2>/dev/null; } | while IFS="$(printf '\t')" read -r k t; do u=$(printf '%s\n' "$k" | awk 'BEGIN { for (i = 0; i < 256; i++) ord[sprintf("%c", i)] = i } +task: [smoke] { awk -F'\t' '$1 == ENVIRON["FRESH_KEY"] { print $1 "\t" }' /work/examples/rsync/.run/applied.txt 2>/dev/null || true; awk -F'\t' '!seen[$2]++' /work/examples/rsync/.run/labeled.txt 2>/dev/null; } | while IFS="$(printf '\t')" read -r k t; do u=$(printf '%s\n' "$k" | awk 'BEGIN { for (i = 0; i < 256; i++) ord[sprintf("%c", i)] = i } function enc(s, out, j, c) { out = ""; for (j = 1; j <= length(s); j++) { c = substr(s, j, 1) out = out ((c ~ /[A-Za-z0-9\/._~-]/) ? c : sprintf("%%%02X", ord[c])) } return out } { print $0 "\t" enc($0) }' | cut -f2); want=$(awk -F'\t' -v k="$k" '$1 == k { print $2 }' /work/examples/rsync/.run/upstream.txt); h=$(curl -fsS --connect-timeout 15 --max-time 60 --retry 6 --retry-connrefused --retry-max-time 600 -r 0-0 -D - -o /dev/null "https://example.katoptra.org/$u"); got=$(printf '%s\n' "$h" | awk 'tolower($1) == "content-range:" { split($3, a, "/"); r = a[2] } tolower($1) == "content-length:" { c = $2 } $2 == "416" { z = 1 } END { sub(/\r/, "", r); sub(/\r/, "", c); print (r != "" ? r : (z ? 0 : c)) }'); ct=$(printf '%s\n' "$h" | awk 'tolower($1) == "content-type:" { sub(/\r$/, ""); sub(/^[^:]*:[ \t]*/, ""); print }'); echo "$k: $got bytes (listing: $want), served as ${ct:-no type} (label: ${t:-none})"; test "$got" = "$want" || exit 1; case "https://example.katoptra.org" in http*) test -z "$t" || test "$ct" = "$t" || exit 1 ;; esac; done -task: [smoke-tlpdb] echo "no TL subtree: no tlpdb to read back" -task: [smoke-page] test ! -s /work/examples/rsync/.run/index-make.txt || { d=$(tail -1 /work/examples/rsync/.run/index-make.txt); r() { h=$(curl -fsS --connect-timeout 15 --max-time 60 --retry 6 --retry-connrefused --retry-max-time 600 -r 0-0 -D - -o /dev/null "https://example.katoptra.org/$(printf '%s\n' "$1" | awk 'BEGIN { for (i = 0; i < 256; i++) ord[sprintf("%c", i)] = i } +task: [smoke] echo "no TL subtree: no tlpdb to read back" +task: [smoke] test ! -s /work/examples/rsync/.run/index-make.txt || { d=$(tail -1 /work/examples/rsync/.run/index-make.txt); r() { h=$(curl -fsS --connect-timeout 15 --max-time 60 --retry 6 --retry-connrefused --retry-max-time 600 -r 0-0 -D - -o /dev/null "https://example.katoptra.org/$(printf '%s\n' "$1" | awk 'BEGIN { for (i = 0; i < 256; i++) ord[sprintf("%c", i)] = i } function enc(s, out, j, c) { out = ""; for (j = 1; j <= length(s); j++) { c = substr(s, j, 1) out = out ((c ~ /[A-Za-z0-9\/._~-]/) ? c : sprintf("%%%02X", ord[c])) } return out } { print $0 "\t" enc($0) }' | cut -f2)"); echo "$(printf '%s\n' "$h" | awk 'tolower($1) == "content-range:" { split($3, a, "/"); r = a[2] } tolower($1) == "content-length:" { c = $2 } $2 == "416" { z = 1 } END { sub(/\r/, "", r); sub(/\r/, "", c); print (r != "" ? r : (z ? 0 : c)) }') bytes, type $(printf '%s\n' "$h" | awk 'tolower($1) == "content-type:" { sub(/\r$/, ""); sub(/^[^:]*:[ \t]*/, ""); print }')"; }; s=$(awk -F'\t' -v k="${d}example.katoptra.org.directory.index.html" '$2 == k { print $1 }' /work/examples/rsync/.run/index-real.txt); if test -n "$s"; then want=$(r "$s"); else want=$(wc -c < "/work/examples/rsync/staging/${d}example.katoptra.org.directory.index.html" | tr -d ' '); test -n "$want"; want="$want bytes"; fi; got=$(r "${d}example.katoptra.org.directory.index.html"); test -n "$s" || got=${got%%,*}; echo "${d}example.katoptra.org.directory.index.html: $got (${s:-page}: $want)"; test "$got" = "$want"; case "https://example.katoptra.org" in http*) test -z "$d" || { p=${d%/}; want=$(wc -c < "/work/examples/rsync/.run/slash/$(printf '%s\n' "$p" | awk -F/ '{ print NF }')/$p" | tr -d ' '); test -n "$want"; got=$(r "$p"); got=${got%%,*}; echo "$p: $got (listing: $want bytes)"; test "$got" = "$want bytes"; } ;; esac; } -task: [smoke-canary] h="$CANARY"; n=$(awk -F'\t' -v k="$h" '$1 == k { print $2 }' /work/examples/rsync/.run/applied.txt 2>/dev/null); case "https://example.katoptra.org" in http*) ;; *) n= ;; esac; if test -z "$n"; then echo "canary skipped ($h is not in the state, or the URL is not http)"; else aws s3 cp --no-progress --cli-connect-timeout 60 --cli-read-timeout 300 "s3://example/$h" /work/examples/rsync/.run/canary.html && curl -fsS --connect-timeout 15 --max-time 60 --retry 6 --retry-connrefused --retry-max-time 600 -A "libwww-perl/6.67" "https://example.katoptra.org/$h" | cmp - /work/examples/rsync/.run/canary.html && echo "$h: $n bytes, served to a libwww-perl client as the bucket holds them"; fi -task: [smoke-encoding] g=$(awk -F'\t' '$1 ~ /\.tar\.gz$/ && $2 > 0 { print $1; exit }' /work/examples/rsync/.run/applied.txt 2>/dev/null); case "https://example.katoptra.org" in http*) ;; *) g= ;; esac; if test -z "$g"; then echo "content-encoding check skipped (no non-empty .tar.gz in the state, or the URL is not http)"; else u=$(printf '%s\n' "$g" | awk 'BEGIN { for (i = 0; i < 256; i++) ord[sprintf("%c", i)] = i } +task: [smoke] h="$CANARY"; n=$(awk -F'\t' '$1 == ENVIRON["CANARY"] { print $2 }' /work/examples/rsync/.run/applied.txt 2>/dev/null); case "https://example.katoptra.org" in http*) ;; *) n= ;; esac; if test -z "$n"; then echo "canary skipped ($h is not in the state, or the URL is not http)"; else aws s3 cp --no-progress --cli-connect-timeout 60 --cli-read-timeout 300 "s3://example/$h" /work/examples/rsync/.run/canary.html && curl -fsS --connect-timeout 15 --max-time 60 --retry 6 --retry-connrefused --retry-max-time 600 -A "libwww-perl/6.67" "https://example.katoptra.org/$h" | cmp - /work/examples/rsync/.run/canary.html && echo "$h: $n bytes, served to a libwww-perl client as the bucket holds them"; fi +task: [smoke] g=$(awk -F'\t' '$1 ~ /\.tar\.gz$/ && $2 > 0 { print $1; exit }' /work/examples/rsync/.run/applied.txt 2>/dev/null); case "https://example.katoptra.org" in http*) ;; *) g= ;; esac; if test -z "$g"; then echo "content-encoding check skipped (no non-empty .tar.gz in the state, or the URL is not http)"; else u=$(printf '%s\n' "$g" | awk 'BEGIN { for (i = 0; i < 256; i++) ord[sprintf("%c", i)] = i } function enc(s, out, j, c) { out = ""; for (j = 1; j <= length(s); j++) { c = substr(s, j, 1) out = out ((c ~ /[A-Za-z0-9\/._~-]/) ? c : sprintf("%%%02X", ord[c])) } return out } { print $0 "\t" enc($0) }' | cut -f2) && h=$(curl -fsS --connect-timeout 15 --max-time 60 --retry 6 --retry-connrefused --retry-max-time 600 -r 0-0 -D - -o /dev/null "https://example.katoptra.org/$u") && if printf '%s' "$h" | grep -qi '^content-encoding:'; then echo "$g: served with a content-encoding" >&2; exit 1; fi && echo "$g: served with no content-encoding"; fi -task: [smoke-unlanded] if test -f /work/examples/rsync/.run/changed.txt && test ! -f /work/examples/rsync/.run/chain; then LC_ALL=C comm -23 /work/examples/rsync/.run/changed.txt /work/examples/rsync/.run/applied.txt > /work/examples/rsync/.run/unlanded.txt; else : > /work/examples/rsync/.run/unlanded.txt; fi; cut -f1 /work/examples/rsync/.run/unlanded.txt | head -20 | while IFS= read -r p; do echo "::warning::listed upstream, never fetched: $p"; done -task: [fresh] k="$FRESH_KEY"; if ! awk -F'\t' -v k="$k" '$1 == k { f = 1 } END { exit !f }' /work/examples/rsync/.run/applied.txt 2>/dev/null; then echo "freshness skipped ($k is not in the state)"; else h=24; case "$h" in ''|0?*|*[!0-9]*) echo "fresh: FRESH_HOURS is a whole number, not $h" >&2; exit 1 ;; esac; u=$(printf '%s\n' "$k" | awk 'BEGIN { for (i = 0; i < 256; i++) ord[sprintf("%c", i)] = i } +task: [smoke] if test -f /work/examples/rsync/.run/changed.txt && test ! -f /work/examples/rsync/.run/chain; then LC_ALL=C comm -23 /work/examples/rsync/.run/changed.txt /work/examples/rsync/.run/applied.txt > /work/examples/rsync/.run/unlanded.txt; else : > /work/examples/rsync/.run/unlanded.txt; fi; cut -f1 /work/examples/rsync/.run/unlanded.txt | head -20 | while IFS= read -r p; do echo "::warning::listed upstream, never fetched: $p"; done +task: [fresh] k="$FRESH_KEY"; if ! awk -F'\t' '$1 == ENVIRON["FRESH_KEY"] { f = 1 } END { exit !f }' /work/examples/rsync/.run/applied.txt 2>/dev/null; then echo "freshness skipped ($k is not in the state)"; else h=24; case "$h" in ''|0?*|*[!0-9]*) echo "fresh: FRESH_HOURS is a whole number, not $h" >&2; exit 1 ;; esac; u=$(printf '%s\n' "$k" | awk 'BEGIN { for (i = 0; i < 256; i++) ord[sprintf("%c", i)] = i } function enc(s, out, j, c) { out = ""; for (j = 1; j <= length(s); j++) { c = substr(s, j, 1) out = out ((c ~ /[A-Za-z0-9\/._~-]/) ? c : sprintf("%%%02X", ord[c])) } return out } { print $0 "\t" enc($0) }' | cut -f2); t=$(curl -fsS --connect-timeout 15 --max-time 60 --retry 6 --retry-connrefused --retry-max-time 600 "https://example.katoptra.org/$u" | awk '!/^#/ { for (i = 1; i <= NF; i++) { if ($i ~ /^[0-9]+$/ && length($i) >= 9) { print $i; exit } diff --git a/toolbox.yml b/toolbox.yml index 7787eb7..8da81ee 100644 --- a/toolbox.yml +++ b/toolbox.yml @@ -16,13 +16,8 @@ # engine include adds are also not available: # - The rsync engine: pipeline, plan-pipeline, list, normalise, state, pull, push, # rebuild, diff and split. Also batches, batch, fetch, publish, label, -# label-trees, merge, checkpoint, remove, delete, reconcile, retry, pages, -# index-same, index-copy, smoke and fresh. Its hooks are prepare, verify, index, -# smoke-mirror and report-engine. The steps of three verbs add these names: -# - verify: verify-signed, verify-containers and verify-decision -# - pages: pages-select and pages-draw -# - smoke: smoke-sample, smoke-tlpdb, smoke-page, smoke-canary, smoke-encoding -# and smoke-unlanded. +# label-trees, merge, checkpoint, remove, delete, reconcile, retry, pages, smoke +# and fresh. Its hooks are prepare, verify, index, smoke-mirror and report-engine. # - The proton engine: pipeline, plan-pipeline, session, session-push, seal, # session-seal, age, pull, push, pd, destination, upload, confirm, list-folder, # trash, empty-trash and empty-trash-pipeline. Its hooks are stage, prune and