Skip to content
algernon28Public

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

Vespera

Curates a local archive into a publication-ready knowledge base.

You point it at a folder of documents and at a handful of examples of what you care about. It walks the archive, extracts the text, throws out the duplicates and the boilerplate, scores what is left against your examples, and hands you the judgement calls it cannot make for you.

It never guesses a threshold and it never edits your archive. Everything it decides, it records; everything it cannot decide, it measures and asks you about.

This file is how to operate the tool. For what each stage does when you run it, and how it resumes after a stop, see docs/running-stage-by-stage.md. For the state of the project — what is built, what is only designed — see AGENTS.md. For why it works the way it does, see docs/adr/.


Step 0 — name your exemplars, before you run anything

Vespera has no idea what you consider relevant, and it will not try to work it out. You supply a seed folder: a handful of documents that are examples of what you want kept. That is the only domain knowledge the tool takes, and everything downstream is measured against it. It also shapes what you get: the topics the finished tree is arranged under are found in what your seed documents say, across all of them, not one for each file.

Put the path in profile.yaml in your working directory, under seedFolder, beside a line saying how you chose them. The working directory is the folder you name, with --db-dir=<path>, the environment variable VESPERA_WORKING_DIR or vespera.working-dir in application.yaml beside the jar, as "Where things live" describes: nothing names one for you, and a command naming none refuses to start. Create it if it is not there yet.

seedFolder:
  value: 'D:\archive\exemplars'
  provenance: "the twelve reports I would keep without reading them again"

Every key in profile.yaml takes this shape: the answer under value, and how you arrived at it under provenance. A bare seedFolder: D:\archive\exemplars is refused, and no command starts until the key is written as above. Put a Windows path in single quotes, or write it with forward slashes: inside double quotes, YAML reads a backslash as the start of an escape.

Do this first. Nothing the tool prints can ask you for it, because by the time anything is printed the first invocation has already happened — and discovering at the second invocation that exemplars were needed all along is the worst way to meet this tool.

The path is five invocations

Getting from a folder of documents to a curated archive takes five invocations. Each one stops where a value is missing that only you can supply. Every stop is deliberate: the tool ends the invocation, keeps everything it learned, and tells you the one thing to do next.

You set You run It does
1 — vespera run <root> walks the archive, extracts the text, takes the census. Stops before deduplication, because the boilerplate floor is read off what this invocation just measured
2 boilerplateDocumentFrequencyFloor vespera run deduplicates, reads your seed folder, scores every survivor against it, places each under one of the topics found in your seed documents, groups them within each topic, and writes you the documents to judge, sixty as Vespera ships. Removes nothing
3 your labels, sixty as Vespera ships, in relevance-labels.yaml vespera label records your labels. They belong to the documents, not to the run, so they survive everything afterwards
4 relevanceScoreFloor vespera run applies your threshold. This is the first invocation that removes anything for being irrelevant
5 arrangementApproved vespera run writes the connecting text over the arrangement you approved, and an overview of each topic over its groups, and leaves the finished tree of HTML pages, a page for each topic and a page for each group beside a copy of each of its documents, and each set of code copied whole, in your working directory or the folder vespera.deliverable-root names

You will be told which value is missing, every time. You do not need this table in front of you and you do not need to know which stage you are at — read the closing line of the output, the last line that does not start Report: , and it names what to set next.

Six is what you get if you set one value per invocation. Both of invocation 2's prerequisites, the floor and seedFolder, are settable as soon as invocation 1 finishes, so setting them together is what makes the path five rather than six.

To stop being asked about each arrangement, write auto into arrangementApproved. Every arrangement is then written over in the invocation that makes it, unread, including the new one a changed value makes, so invocation 4 goes on to write the finished tree. arrangement.html is still written each time. Put why in provenance. Remove the value, or write a name, and the next arrangement is asked for again.

What you have to decide, and where to read it

Vespera writes reports beside the database. Each one measures something; none of them chooses for you. Each has a title, which its page, its tab in the control panel's Reports window and a value's link to it read alike: What the files turned out to be is format-mix.html, Files whose text could not be read extraction-failures.html, Files whose text was read and kept extraction-survivors.html, Text quality distribution confidence-distribution.html, Repeated text across documents document-frequency.html, Word count distribution word-count-distribution.html, Your documents compared with your seed documents seed-corpus-comparison.html, Group sizes cluster-sizes.html, Choosing the relevance cut-off relevance-labelling.html, Topics topics.html, Grouping to approve arrangement.html and Group page sizes cluster-page-sizes.html.

The value Read Which reports
seedFolder your own knowledge of the archive —
boilerplateDocumentFrequencyFloor how many documents share the same passages document-frequency.html
embeddingModel whichever model you can serve locally, if you want one other than hf.co/Qwen/Qwen3-Embedding-8B-GGUF:Q4_K_M —
relevanceScoreFloor what each possible cut would cost you, in documents relevance-labelling.html, cluster-sizes.html, seed-corpus-comparison.html
degenerateOutputConfidenceFloor how well the text extraction went confidence-distribution.html
degenerateOutputWordFloor how many words each kind of file held once its text was extracted, and what each floor would remove word-count-distribution.html
extractionAttempt whether the files the converter did not answer about are worth asking again extraction-failures.html
arrangementApproved whether the groups the tool formed are worth writing over arrangement.html
generationModel whichever model you can serve locally, if you want a different one —
modelsRunOn nothing: the models run on this machine, local, and cloud is not supported yet —
generationContextWindow how much your own machine can read in one go —
deliverableLanguage which language the people reading the pages read: en or it —
clusterPageByteCeiling how large each group's page is when written whole, and how many groups pass each size cluster-page-sizes.html
clusterResolution how finely the documents under each topic divide into groups cluster-sizes.html
logTimestampShareFloor how much of each text file begins with a date or a time format-mix.html
codeNameEndings which text files, or files of no known kind whose bytes read as text, are code by the ending of their name, kept apart from the documents format-mix.html
outOfScopeNameEndings which files, of no known kind or text, you know are not documents by the ending of their name format-mix.html
configurationSettingShareFloor how much of each text file is settings, which files open with a program's configuration, and which interface definitions were kept format-mix.html
codeSetFolders which folders outside the corpus root hold code that is read with the corpus —
foldersNotWalked which folders of the corpus hold nothing you want read —
topics which topics the deliverable is made of, one name a line, where you give them; empty, they are found in your seed documents —
confluenceBaseUrl the address of your Confluence Cloud site, where you publish to it —
confluenceSpaceKey the key of the Confluence space the pages go in —
confluenceFolderId the id of the page or folder the pages go under, the number in its address; name this or its title, not both —
confluenceFolderTitle the title of the page or folder the pages go under; name this or its id, not both —
confluenceEmail the email of your Confluence account —
confluenceRootTitle the title of the top page the others go under, {language} standing for the language's code; empty is Vespera_{language} —
outputs what is drawn: tree for the HTML pages and each plugin by its name, separated by commas; empty is tree, confluence, website —

document-frequency.html shows boilerplateDocumentFrequencyFloor's measurement. For each passage of five words found in two documents or more, stage 3 counts how many documents it is in, out of the documents it measured. The floor is a share between 0 and 1 of the documents measured: a passage is boilerplate, and is left out when documents are compared, when the documents it is found in number at least the floor times the documents measured. A passage is stored as a number, not as its words, so the page shows counts and shares and no text, and names no file.

The page names the content census it was written for. Each row is a share of the documents, rounded down to the hundredth, then how many passages are found in that share, the fewest documents any of them is found in, and how many documents were measured; the third divided by the fourth is the exact share of the least-shared of them.

To choose the floor, pick a row and set the floor to the share it shows: every passage on that row and on the rows after it is then boilerplate, and none on the rows before. Taking the row's third number as k and its fourth as N, k divided by N does the same, written in full or cut off after at least as many decimal places as N has digits: for 7 of 25, 0.28. Every passage found in k documents or more is then boilerplate, and none found in fewer. Do not round k divided by N up: 2 / 3 rounded up to 0.67 leaves out the passages found in 2 of 3 documents.

Stage 3 writes the page each time it measures, and a stage 3 already recorded leaves the page as it is. Stage 3's line in the output of vespera run names the run it used, as Stage 3 (content census) finished under run <id> or Stage 3 (content census) was already recorded under run <id>. If that id is not the one the page names, the invocation went back to a content census recorded earlier, as putting a value back to an earlier one, or running a second archive in the same working directory, does. The page is not that run's. To read that run, use the command below, with the id at the end, after a space.

This prints the same rows as the page, for the run you name. Run it from your working directory, with Python on your machine, while no vespera command is running and no vespera.db-wal lies beside vespera.db (if one does, Where things live says what to do); it reads the database without writing to it and leaves no file beside it:

python -c "import sqlite3,sys;db=sqlite3.connect('file:vespera.db?mode=ro&immutable=1',uri=True);run=sys.argv[1] if sys.argv[1:] else db.execute('select r.id from run r join finished_step s on s.run_id=r.id and s.step=r.stage where r.stage=? order by r.rowid desc limit 1',['content-census']).fetchone()[0];print('run',run);[print(r) for r in db.execute('select 100*f.document_count/c.shingled_document_count/100.0 as share,count(*),min(f.document_count),c.shingled_document_count from shingle_document_frequency f join shingle_corpus_size c on c.run_id=f.run_id and c.shingle_parameter_identity=f.shingle_parameter_identity where f.run_id=? group by share order by share',[run])]"

The first line names the content census it read: the one whose id you put at the end, or, with no id, of the content censuses that finished, the one whose run was created last. Each line after the first is a row as the page shows it, the share written as Python writes it, 1.0 for 1.00; the rule for choosing the floor is the same. If the command stops with TypeError: 'NoneType' object is not subscriptable, no content census has finished in that database. If it stops with unable to open database file, there is no vespera.db where it looked, and it has made none: run it from the directory you run vespera from, or name the moved vespera.db. If it stops with no such table: run, the vespera.db it opened holds no ledger, as an empty file of that name does: check that it is the working directory's. If it prints the run line and nothing else, no run with that id stored any passages.

Fourteen of these are optional and none of them is one of the five stops; all fourteen are here so that you know they exist.

degenerateOutputConfidenceFloor, left unset, removes nothing for extracting badly. degenerateOutputWordFloor, left unset, removes nothing for having too few words: set it, as a whole number, to remove every file whose extracted text holds fewer words than that, whatever kind of file it is, as holding no recognisable text. A word is anything between two spaces, a year or a word in any alphabet included, counted over all of the text, page headers and footers too. word-count-distribution.html shows, for each kind of file, how many of the files kept hold each number of words, and how many of each kind a floor of 2 to 10, 20, 50 or 100 would remove: a one-line note, a shortcut and a picture whose only text is a button's one word go alike. logTimestampShareFloor, left unset, removes no logs: set it, as a share between 0 and 1, to leave out as a log every text file of ten lines or more in which at least that share of the lines read begin with a date or a time. generationModel and generationContextWindow, left unset, write the connecting text with the model and the reading window Vespera ships with — they are the values that already have answers, and setting one only replaces the answer it already had. deliverableLanguage is one of them too: left unset, the model is asked to write each group's heading and connecting text in English, and every page of the tree and of its Confluence rendering says what Vespera says in English. Set it to it and the model writes them in Italian itself, nothing being translated afterwards, and every sentence, heading and title Vespera writes on those pages is in Italian; a title taken from a file's name stays that name. The reports beside the database, the progress page and the last line stay in English. Any value other than en or it, in capitals or not, stops the invocation when it reaches the writing, before any group is asked about or written. Changing it has every group written again, under a run of its own.

clusterPageByteCeiling, left unset, writes every group's page as one file, however large. Every time the tree is drawn, cluster-page-sizes.html beside the other reports shows how large each group's page is written whole, in bytes, its pictures inside it: the largest, the 95th percentile and the median, how many groups pass 100000, 1000000, 10000000 and 100000000 bytes, and the ten largest with how many documents each lists; it names no group and no document. Set it, as a whole number of bytes of 1 or more, to split every page larger than that at the end of a document's entry: the group's first page stays index.html, the next are index-2.html, index-3.html and on, each no larger than the number unless one document's entry alone is, and every page links the first, the previous, the next and the last. The connecting text stays on the first page, and each citation in it links to its document on whichever page lists it. Nothing is asked of the model again, and no stage runs again: the next vespera run that reaches the tree draws it again. A value that is not a whole number of 1 or more is ignored, and the last line says so.

clusterResolution, left unset, divides the documents under each topic into groups as finely as Vespera ships, at 1.0, and cluster-sizes.html shows what that made: a row for each topic and, under a topic of 300 documents or more that Vespera first divided into subtopics, a row for each subtopic, each with its share of the documents grouped. No share is too large and none is a warning: a topic holding most of the documents takes the grouping longer, and nothing else. Set it, as a number above 0, to divide them otherwise: a higher number makes more groups, each smaller. Changing it makes a new scoring run and every run after it, though nothing is embedded again, so the groups are new and you approve the arrangement again. It also draws other documents to judge and writes relevance-labels.yaml over: the labels you recorded are kept, a label typed into the file and not yet recorded goes with the file, so run vespera label first. A value that is not a number above 0 is ignored, and the last line says so.

configurationSettingShareFloor, left unset, treats no file as a configuration file: set it, as a share between 0 and 1, to treat as one every text file of ten lines or more in which at least that share of the lines read are settings (key = value, key: value or a [section] heading, comments not counted), and every XML file that opens with Maven's, logback's, log4j 2's, .NET's or Spring's configuration, whatever its name. One lying in a folder of code the tool keeps together is kept with that code; any other is left out. A guide about configuration, an XML Schema, a WSDL, an OpenAPI or Swagger description and a JSON Schema are kept.

codeSetFolders, left unset, reads nothing outside the corpus root. To have code you keep beside the archive read with it, such as a checkout of a program its documents describe, write the whole path of its folder into codeSetFolders, such as D:\code\billing, several separated by ; on Windows and : elsewhere, with why in provenance, or add it in the control panel's Code sets section, and run again. Each folder is walked after the corpus root, in the same walk, each of its files named by its whole path. Its code is kept apart in code sets of its own, the folder itself being one where no nearer folder holds a build file, named for the folder, and copied whole into the deliverable's code/; its documents are judged as the corpus's are. Nothing is written into it. vespera run refuses, in one line and before anything is walked, a folder that is not there, or one that is, lies inside or lies above the corpus root, the working directory, the seed folder, the deliverable root or a folder listed before it. Changing it makes a new walk, so every run after it is new. A walk stopped partway is continued with the folders it started with, and a change applies from the next walk. A value holding a piece that is not a whole path is ignored, and nothing outside the corpus root is read.

foldersNotWalked, left unset, reads every folder of the corpus. To leave a folder of what you are not keeping unread, such as exports of old flows, write its path from the corpus root into foldersNotWalked, with / between its names, such as archive/old-flows, several separated by ; on Windows and : elsewhere, with why in provenance, or untick it in the tree of the control panel's Folders section and save, and run again. The walk counts such a folder and never goes into it, so nothing below it is read, converted or asked about. Vespera keeps every copy of a document held in several languages, each one scored, grouped and written over as any other document: where the copies of one language lie in a tree of their own, such as docs/zh, name that tree's folder here to leave the language out. Changing it makes a new walk, so every run after it is new. A walk stopped partway is continued with the list it started with, and a change applies from the next walk. A folder listed that is not there leaves nothing out. A value holding a piece that is not a folder below the corpus root, such as a whole path or one with .. or a backslash, is ignored, and every folder is read.

outOfScopeNameEndings, left unset, leaves nothing out for its name. Some files are not documents and hold nothing that says what they are; format-mix.html counts them under What could not be recognised, grouped by how they begin. To leave such files out, write the endings of their names into outOfScopeNameEndings, each beginning with a dot and separated by spaces, with why in provenance, and run again. Case does not matter, and * is not a wildcard: list each ending. Only a file the tool could not recognise, or a text file, is left out this way; a PDF or a Word document whose name ends with a listed ending is kept. For example:

outOfScopeNameEndings:
  value: ".P3A .P3L .P3P .P3S .P3O .P3K .P3T .DGN .LGN .PGN .SGN"
  provenance: "format-mix.html: payment terminal package and signature files, not documents"

extractionAttempt, left unset, is the first attempt at extracting text from your archive. Some files may not have been read because the converter was busy, failing or too slow at the time. extraction-failures.html lists those files beside the ones it could not convert. To have the converter asked about them again, write 2 into extractionAttempt, with why in provenance, and run again. Files it already answered about are not converted again. Everything after extraction is redone, because what was read may have changed: deduplication, scoring, grouping and the connecting text. The groups are new, so you approve the arrangement again before anything is written. Nothing from the first attempt is deleted. Remove the value, or write 1, and the next run is back on the first attempt, with nothing redone. If you had approved the second attempt's arrangement, write the first one's name back into arrangementApproved before the connecting text is written again; the last line of the output names it. Next time, write 3. Do not delete rows from vespera.db to make a stage run again.

The reading window is how much of a group goes into one request. Set it larger and more of each group is read in one go; leave it alone and Vespera uses a size any machine can serve. Groups too large to fit are still written about, from the documents nearest your exemplar, and the finished page says how many of them it was written from.

Every value you set carries a provenance field. In profile.yaml, write down how you arrived at the number. On the control panel you write no provenance: a value you change there is saved with set in the control panel, so an old reason is never left under a new number, and a value you leave as it is is not written, so a provenance you wrote in profile.yaml stays until you change its value in the window. To keep a reason of your own, write it in profile.yaml. The provenance the file states for a value is shown at the end of the tooltip of its title, after Where this value came from:. Nothing checks that you read the report first — what stands between a guess and your archive is what you record there.

Invocation 3 in more detail, because it costs you two hours

Invocation 2 leaves two files beside the database:

  • relevance-labelling.html — how the scores are spread, the bands they fall into, five as Vespera ships, and what cutting at each band boundary would cost you in documents kept and documents lost. Open it in a browser; every document links to the original.
  • relevance-labels.yaml — sixty documents as Vespera ships, twelve from each of its five bands, drawn evenly across the score range (a machine's own configuration may name others, as vespera.relevance.sample.bands and vespera.relevance.sample.per-band), each with relevant: null waiting for a true or a false. A document you have already labelled, against the same seed folder and from the same corpus root, shows your label instead, so a later run's file starts from what you said last time. The file names the seed folder it was written for; if you change seedFolder before running vespera label, it is refused, so run again first.

Label them, then run vespera label. At roughly two minutes a document that is about two hours, and the sixty Vespera ships is a ceiling rather than a target: the same run always draws the same documents, so you can stop and come back.

Your labels outlive the run that drew the documents. They are keyed to the documents, so changing the embedding model later re-scores the archive and re-reads the labels you already set, without having you label anything twice.

Where things live

Everything the tool writes goes in one working directory, never inside your archive, except the deliverable when you name a folder for it and one small lock file per corpus root in a folder of yours (both below), and its configuration files, in the folder the jar lies in (The configuration beside the jar, under Running it):

profile.yaml                     the values you set, and how you arrived at them
profile.yaml.before-repair-<time>  profile.yaml as it stood before the control panel repaired it, kept for you; nothing removes it
not-relevant-folders.yaml        the folders you list as not relevant beneath the tree of the control panel's Folders section; the control panel writes it and sets the labels of relevance-labels.yaml from it, the jar never reads it
vespera.db                       the ledger: every file seen, every verdict, every measurement
vespera.lock                     held by the command running now, or by the control panel while it saves, so a second one is refused
next-invocation.properties       what the next command wants, written as a command ends on its closing line; read by the control panel
effective-values.properties      what the jar uses where the profile names nothing, and where each value came from, written as vespera run or vespera label starts; read by the control panel
logs/vespera.log                 what the console shows, with dates; older days roll in beside it
report.json                      how the command running now, or the last to end, stands, as numbers and words: its steps and legs, charts, runs, counts, gates and refusals, and no document's text; written by vespera run, vespera label and vespera publish, and rewritten every vespera.progress-interval while a command runs
logs/report-*.json               the copy of report.json each command keeps, named by the command's start in UTC; nothing removes it
logs/output-*.log                the whole output of one command, as its console showed it, named as its report's copy; deleted by the first command that begins 14 days after it
report.html                      the page drawn from report.json after each write, through the jar's Mustache template or the one vespera.report.template names by an absolute path: one self-contained page with no script, which reloads at vespera.progress-interval, and which no command opens (ADR-362)
chart-step-<step>.json           the chart of one step, census to generation, written by vespera run beside report.json: the step's counts and, for a step that records verdicts, its verdicts by kind, over the step's own time, in the last vespera run alone; every one is written empty as vespera run begins its steps, frozen with the time it finished once the step has, and marked where the step finished in an earlier invocation; vespera label and vespera publish write none, and a chart-stage-<stage>.json an earlier version wrote is read by nothing
charts/<run id>/chart-step-<step>.json  the chart of a step that records its completion under a run, census, relevance floor and relevance report aside, as it froze in the vespera run in which the step finished, kept under that run; a later vespera run that finds the step finished under the same run draws it again, frozen, with the day and time it finished; never emptied, and nothing removes it
format-mix.html                  what kinds of file the archive holds
format-mix.json                  the same report's numbers and rows as data, written beside its page; report.json copies its headline
extraction-failures.html         the files that could not be read, and why
extraction-survivors.html        the files that were read and kept
confidence-distribution.html     how well the text came out
word-count-distribution.html     how many words each kind of file kept holds
document-frequency.html          how many documents share each passage
seed-corpus-comparison.html      how far your exemplars resemble the archive
relevance-labelling.html         the report you read to choose the threshold
relevance-labelling.json         the same report's numbers and rows as data, written beside its page; report.json copies its headline
relevance-labels.yaml            the documents you label, sixty as Vespera ships
cluster-sizes.html               how the survivors grouped under each topic and each subtopic, and each one's share of them
topics.html                      the topics found in your seed documents: what feeds each, how many documents were placed under it, and the seed documents feeding none
topics.json                      the same report's numbers and rows as data, written beside its page; report.json copies its headline
cluster-page-sizes.html          how large each group's page is, in bytes, written each time the tree is drawn
cluster-page-sizes.json          the same report's numbers and rows as data, written beside its page; report.json copies its headline
arrangement.html                 the groups, named and in order under their topics and subtopics, for you to approve; the topics and their names are shown, not approved
arrangement.json                 the same report's numbers and rows as data, written beside its page; report.json copies its headline
deliverable/<run>/index.html     what was written, group by group, and what produced it, opened when written
deliverable/<run>/documents.csv  every surviving document, with its place in the order and its copy
deliverable/<run>/code.csv       every file of every code set, with its set, its role and its copy
deliverable/<run>/code/<set>/<path>  a checked copy of each code set, its folders kept
deliverable/<run>/<topic>/index.html  one page per topic: its overview, its groups or subtopics, and the documents of other topics that also speak of it
deliverable/<run>/<topic>/<subtopic>/index.html  one page per subtopic, where a topic is divided; its groups' folders lie beside it
deliverable/<run>/<topic>/<group>/index.html  one page per group, its pictures inside it, or its first page where clusterPageByteCeiling splits it; a folder deeper, under its subtopic, where its topic is divided
deliverable/<run>/<topic>/<group>/index-<k>.html  a group's further pages, where clusterPageByteCeiling splits it
deliverable/<run>/<topic>/<group>/<file>      a checked copy of each of the group's documents, under its own name
deliverable/<run>/vespera-deliverable.txt  which working directory and which run drew this folder
deliverable/<run>/vespera-run.db  the record of what this folder was drawn from: every file the walk found, every run behind it and its judgements, and its topics, groups and writing, as one SQLite file; Vespera never reads it back
deliverable/<run>/confluence/pages.json  the Confluence rendering's page list: every page, its title, its body and the files it carries
deliverable/<run>/confluence/<topic>/<group>/page.xml  each page's body in Confluence's storage format, its pictures' files beside it; a folder deeper, under its subtopic, where its topic is divided; the root page's, each topic's and each subtopic's lie above
deliverable/<run>/confluence/drawing.txt  how the last drawing of the Confluence rendering went: drawing, drawn or failed
deliverable/<run>/website/files.json  the website rendering's list: every file of the website, where it comes from in the tree, its size and digest, written last
deliverable/<run>/website/search-index.js  what the website's search looks through: each group's title, label and writing, its topic's and subtopic's names, and its documents' names
deliverable/<run>/website/website.js  the website's script, its search, shipped in the jar
deliverable/<run>/website/website.css  the website's theme, dark whatever the system's setting, shipped in the jar
deliverable/<run>/website/drawing.txt  how the last drawing of the website rendering went: drawing, drawn or failed
website/index.html               the website vespera publish --to website refreshes, to open or to sync, its pages, script, theme, documents.csv and copies beside it
website/vespera-website.json     the files the last vespera publish --to website wrote there, so the next removes only what a publish wrote
website/vespera-deliverable.txt  the one line website, which tells an agent's tools to keep out; it names no path
machine.lock                     not here, but in the folder the corpus-root locks lie in: held by the control panel or the command running now, so a second Vespera on this machine is refused
machine-child.lock               beside machine.lock: held by a command the control panel started, while it runs

The deliverable can be drawn in a folder of your choosing instead: name it with vespera.deliverable-root, in application-local.yaml, as the environment variable VESPERA_DELIVERABLE_ROOT, or with --vespera.deliverable-root=<path>. The folder is then the deliverable itself, index.html at its top, everything listed above under deliverable/<run>/ directly in it, and nothing under the working directory's deliverable/. Left unset, the deliverable goes to deliverable/<run>/ as above. vespera run refuses, in one line and with exit code 1, before it walks anything, a folder that is the corpus root or the seed folder, lies inside either or holds either; a folder that is the working directory's website/ or lies inside it, since your sync of the website would carry the deliverable's code sets; and a folder that already holds anything but a deliverable this working directory drew. It is compared where it really leads, links followed. A folder inside the working directory is fine. With no folder named, the working directory's deliverable/ is checked the same way, so a working directory inside the corpus root or the seed folder is refused. Keeping it out of the rest of your archive is yours to watch: Vespera cannot see where your archive ends. Once a folder holds one run's deliverable, a later run that writes anew needs a new, empty folder: that run stops with one line and exit code 1 before drawing over it, and what it wrote is kept for when you name one. Vespera deletes nothing in the folder.

A vespera.log an earlier version wrote at the top of the working directory, with the files it rolled into, stays there, and so do accounts written to a folder vespera.account-dir named, the invocation-account-*.txt an earlier version wrote in logs, and the progress pages, progress.html, progress-steps.html, progress-samples.html, progress-runs.html and progress.json, it wrote in the working directory: Vespera moves nothing, and reads none of them. Move or delete them yourself.

No command opens a browser, a file or a window, whatever the machine: every vespera run, vespera label and vespera publish names report.html in its first line, Report: <path>, and you open it yourself, and ends its output on a last line that starts Report: too, after its closing line, naming report.json and report.html and saying that nothing was opened and why. Each invocation's report.json says so, in its opened field. report.html is one page that reloads itself every five seconds (vespera.progress-interval), by a <meta http-equiv="refresh"> in its head and no script, while the command runs and after it has ended, so a page you leave open shows the next vespera run, vespera label or vespera publish in the same working directory once it starts, and until then says that the last one has ended. Its status line says when the invocation started and, once it has, when it ended, and when the page was written and at what interval; a page cannot mark itself stale, and the window does. If the working directory is moved or deleted, an open page shows your browser's own error and stops reloading; reload it yourself once report.html is back. Its sections run in this order: the status line; how many steps have finished, the step running and how far it has got, the elapsed time and the database's size, with the survivors and the verdicts of the stage removing occurrences, their rates, and the database's growth; the timeline of the steps; the charts of the database's size, the running step's counts and the verdicts over time; the steps; the gates and the next value; the refusals and the faults; the analysis reports, each with its headline numbers and its link; the runs; the files; and the display and what was opened. A point of a chart shows its value on hover, and each chart's values and the steps stand as tables, each in a fold beneath its drawing. The timeline folds the steps into four parts, deduplication, relevance, clustering and generation, each one bar from its first step's start to its last step's end, with its state; the part the job is in, or where it failed or first stopped at a gate, is open on its steps, and the others are folded, as a reload draws them. Below the timeline, the charts stand three to a row in a window at least 1,666 px wide, two to a row from 1,116 px, and one to a row below that; a chart or a table scrolls sideways only when the window is too narrow to read it. vespera label writes report.json and report.html when it starts and when it ends: the status line, the elapsed time and the database's size, and the table of the labels it recorded. vespera publish writes them when it starts, after each destination ends and when it ends, and the report's publish field holds each destination, how it ended and its closing line; its last line before the report's own is the last destination's. The report names paths and no document's text.

Set it with --db-dir=<path>, which must be written with the =, or with vespera.working-dir in configuration. A vespera.working-dir set in application-local.yaml or as the environment variable VESPERA_WORKING_DIR wins over --db-dir, the control panel's included, so a command given another --db-dir refuses: remove it to name the working directory with --db-dir. Both commands take --db-dir=<path>, so name it on vespera label as well as on vespera run. A command given a --db-dir other than the directory it actually opened refuses and records nothing.

One command at a time uses a working directory. A second command started on it while the first is still running, from a second terminal or from an IDE, refuses at once with one line naming the command that holds it, its process id and when it started, and exits 1; the first carries on. Wait for the first to finish, or stop it, and run the second again. The control panel holds it too, for the moment it saves a value, labels or names into the working directory, and refuses to save while a command holds it. vespera.lock stays in the working directory after every command and is harmless: a command that crashed or was killed releases it as its process ends. Do not delete it while a command is running, because that lets a second one start. If a step fails saying the database file is held by another process, something other than Vespera, such as a database browser, has vespera.db open; close it and run the same command again.

One vespera run at a time walks a corpus root, whatever working directory each names. A second vespera run on a root the first is still walking refuses at once with one line naming the root, the command that holds it, its process id, when it started and its working directory, and exits 1, its report saying it was refused; the first carries on. vespera label is not affected. The lock is a file in .vespera-root-locks in your home folder, one per root, named by a code made from the root's path rather than the path itself, and the line inside it names the root; name another folder with the environment variable VESPERA_CORPUS_ROOT_LOCK_FOLDER: the control panel, and the commands it starts, read that folder from the environment alone, so a vespera.corpus-root-lock-folder set only in a configuration file is not seen by them. Like vespera.lock, a file there is harmless after a command ends, a command that crashed or was killed releases it, and it must not be deleted while a command is running. If that folder cannot be written, vespera run refuses in one line rather than walk unlocked; make it one you can write, or name another.

The same folder holds machine.lock, which lets one Vespera run on this machine at a time (see Starting the control panel), and machine-child.lock, held while a command the control panel started is running. Both are taken before any configuration is read, so they follow the folder VESPERA_CORPUS_ROOT_LOCK_FOLDER names in the environment, and .vespera-root-locks in your home folder where it names none, never one set in a configuration file. Like the others, they are harmless after a command ends and must not be deleted while one runs.

While a command is running, two more files sit beside vespera.db: vespera.db-wal, SQLite's write-ahead log, which holds the most recent changes, and vespera.db-shm, its index. When the command ends they are folded back into vespera.db and deleted, unless something else, such as a database browser, still has the database open. If vespera.db-wal is still there when no command is running, something left it behind: a command cut short, or something else that had the database open when the last command ended, even if it has closed since. Close whatever still has the database open, then run one vespera command, vespera run again for instance, with nothing else open on the database: it opens vespera.db as it starts, and when it ends it folds vespera.db-wal back into vespera.db and deletes both files beside it. vespera run also carries on the run, and once it reaches the documents to label it writes relevance-labels.yaml anew: if you are labelling them, record your labels with vespera label first. To copy the working directory, copy it after the command has ended and vespera.db-wal is gone. If vespera.db-wal is still there, or you have to copy while a command is running, copy all three files together, or the copy is missing the latest changes. Keep the working directory on a disk attached to the machine that runs Vespera, not on a network share: the database relies on shared memory that only works on a local disk.

The database also keeps its temporary files in the working directory, not in the system's temporary folder. It writes them while a step has it sort a great many rows and removes them itself; on Windows they show there under names beginning etilqs_, and they are no part of a copy. They grow with the archive and can run to gigabytes on a large one, so the disk that holds the working directory needs that much free while a command runs.

Commands

vespera run <root> [--db-dir=<path>]     walk a corpus and take it as far as the next missing value
vespera label [file] [--db-dir=<path>]   record the labels you set in the label file
vespera publish [--to <names>] [--invocation <id>] [--db-dir=<path>]  deliver what your last vespera run drew, or the earlier run named, to each destination named: confluence, retrieval, website
vespera serve [--db-dir=<path>]          serve <working directory>/website/ at http://127.0.0.1:<port>/ until the process ends

vespera publish --to confluence,website delivers to each destination in the order written, a name written twice once; bare, it is --to confluence. One that stops does not stop the next: each says how it ended in one line of its own, the last line is the last destination's, and the command ends with exit code 1 if any stopped, so publishing again goes on from there. A name Vespera does not know is refused, exit code 2, before anything is delivered: vespera publish has no destination : it knows confluence, retrieval, website. A destination with no rendering drawn is refused alone, in one line naming it, and so is one whose last drawing did not finish. Where a destination fails in a way Vespera did not foresee, its line names the kind of error and never its message: vespera publish stopped : it threw , whose message is not written. Publish again once that is dealt with.

vespera publish --invocation <id> delivers what an earlier vespera run drew in place of the last one's, the run named by its id as its report and the control panel show it, 20261114T083000Z, or 20261114T083000Z-2 for a second one started the same second; --to chooses its destinations as above. Its deliverable is looked for in the folder that run drew it in, whatever folder is named for the deliverable now. It replaces what a later publish sent, as any publish does: Confluence's pages are updated in place and the ones it does not have go to the space's trash, and the website folder is refreshed to it. An id of another form is refused, exit code 2. Each of these is refused in one line before anything is delivered, exit code 1, ending Name another, or publish without --invocation.: a run whose report is not kept in the working directory's logs, or cannot be read; a command that was not a vespera run; a run that did not reach the deliverable; a deliverable no longer in the folder it was drawn in; and a folder you named that a later run has drawn another deliverable in. A destination that run drew nothing for is refused alone, in one line naming it. Without --invocation, a publish delivers what your last vespera run reached, as before.

--to website copies the website rendering into the working directory's website/ folder, below, and sends nothing anywhere. --to retrieval writes nothing and sends nothing: it says where retrieval/records.jsonl is, the retrieval records drawn where outputs names retrieval, a line for each synthesis doc, topic overview and passage of a document, with its vector, for a store of yours to import. vespera publish to Confluence is the one command that sends anything off the machine, and only the Confluence rendering: it reads the site, the space, the parent and your account's email from VESPERA_CONFLUENCE_* environment variables or, where the shell sets none, from the profile, and an API token from the environment alone, and refuses before sending anything when one is missing. Typed on the command line it writes confluence-env.sh and confluence-env.ps1 into the working directory for you to fill in and run in your shell; started from the control panel it asks for the token in a box and takes the rest from the Profile window. Started from the control panel, a parent's title that names no page or folder of the space, or several, opens a warning asking for another title, which is saved as confluenceFolderTitle in the Profile window and published to; the token is asked for again.

vespera serve serves the website a publish refreshed, as Publishing the website says, and is no part of a run: the control panel never starts it. It listens at the port vespera.serve.port names, 24332 where nothing names one. Set the port with --vespera.serve.port=<port> on the command line, -Dvespera.serve.port=<port> to Java, or the environment variable VESPERA_SERVE_PORT, the first of those three that is set winning, in that order; a blank one counts as not set. A value that is no port number from 1 to 65535, 0 among them, is refused in one line, exit code 1: vespera serve could not listen on port : it is no port number from 1 to 65535. Name another in vespera.serve.port.

vespera serve reads no YAML file, application.yaml and application-local.yaml among them, so a working directory or a port named only in one of them does not reach it. Name the working directory as for any command, with --db-dir=<path>, -Dvespera.working-dir=<path> or VESPERA_WORKING_DIR, and the port as above; with neither named it refuses in one line, exit code 1, before any other refusal of its own, making nothing: vespera serve has nothing to serve: no working directory is named. Name one with --db-dir=, the environment variable VESPERA_WORKING_DIR or the system property vespera.working-dir. A blank -Dvespera.working-dir=, VESPERA_WORKING_DIR, --vespera.working-dir= or --db-dir= counts as not set, as a blank port does.

Every vespera run, vespera label and vespera publish writes report.json in the working directory and a copy of it, logs/report-<start>.json, kept for each command: one JSON file with no document's name or words in it, whose fields docs/report/report.schema.json describes. There is nothing to set for it. vespera.account-dir is no longer read: left in your configuration, it is ignored, and the run logs one warning naming the logs folder, where the copy is written. A command that cannot be started because the working directory is in use, or whose arguments are refused, writes none, and leaves the report of the command that holds the directory as it was. vespera.progress-interval, set in application-local.yaml or with --vespera.progress-interval=<duration>, is how often vespera run rewrites report.json and report.html, and how often an open page reloads: 5s unless you set it.

vespera run takes the archive root as its argument, falling back to vespera.corpus-root in configuration. Given neither, it refuses rather than guessing — a census of the wrong tree reports success.

Running it

You need Java 26 and a Docker daemon. Run every command below from the root of this repository.

Vespera uses Docker Engine, not Docker Desktop:

  • Linux: Docker Engine, installed as the docker service.
  • Windows: a WSL 2 distribution, Ubuntu for example, with Docker Engine installed inside it. You set this up once: install the distribution with wsl --install, then install Docker Engine in it as you would on Linux, with Docker Compose and buildx: Ubuntu's own docker.io package comes without them, so install docker-compose-v2 and docker-buildx beside it (wsl -u root --exec apt-get install -y docker-compose-v2 docker-buildx). Then set systemd=true under [boot] in the distribution's /etc/wsl.conf, once, and run wsl --shutdown so that it takes effect. WSL must also not shut the distribution down for being idle, and Docker with it, in the middle of a run: that is instanceIdleTimeout=-1 under [general] in %UserProfile%\.wslconfig. Where you have no .wslconfig, Vespera writes one holding just that as it starts Docker or the containers, from a command or from the control panel's Start Docker, and says so. Where you have one, Vespera never changes it, and says so if it lacks the setting, which you then add yourself. Either takes effect the next time WSL starts; wsl --shutdown restarts it at once, and stops every WSL distribution and the containers in them. You never need to open a WSL window after that.
  • macOS: Colima, with the docker command-line client and its Docker Compose and buildx plugins (Homebrew's docker-compose and docker-buildx, each followed by the cliPluginsExtraDirs line Homebrew prints).

Build it. This builds the jar Vespera runs from:

./mvnw package

The containers. Vespera needs two services running beside it:

  • Ollama, which serves the models;
  • docling-serve, the document converter.

vespera run and vespera label --auto start them for you when they are not answering, whichever way you run Vespera: the jar, your IDE, the headless and gui profiles below, or the control panel. If Docker itself is not running, they start it first: the docker service on Linux, Docker Engine inside your WSL 2 distribution on Windows, Colima on macOS. The control panel starts nothing as its window opens: its Start Docker button starts Docker and the containers, as these commands do. On Linux, and inside the WSL 2 distribution, that has to work without a password; if it does not, start Docker yourself (sudo systemctl start docker). To use a WSL 2 distribution other than your default one, set VESPERA_CONTAINERS_WSL_DISTRIBUTION to its name. The first start builds the document converter's image, which takes minutes; what Docker prints is in the log as it goes. A container that already exists, running or stopped, is started as it is and never replaced. Closing the control panel stops the containers and leaves Docker running. Its Shut down Docker button, after Stop and Start Docker, stops the containers and then Docker itself, and with it every other container Docker runs, so it asks first. Nothing else in Vespera stops them, and nothing removes a container. The other commands start nothing. If your .env sets SPRING_DOCKER_COMPOSE_ENABLED, that line does nothing, and you can take it out.

If anything stands in the way, the command stops before it walks anything, and its closing line, the last line before the one that starts Report: , says what: no docker command, Docker not starting, Docker Compose or buildx not installed where Docker runs, with the command that installs them, a converter image that neither compose file builds, docker compose failing, a container still not answering three minutes after it was started, a lock file or temporary file it cannot write, or a compose file beside the jar that it cannot read for the converter's image (The configuration beside the jar, below). vespera.containers.wait-at-most changes those three minutes. If you point Vespera at Docling or Ollama on another machine, through vespera.docling.base-url or spring.ai.ollama.base-url, it starts nothing for them. Vespera tells whether a service is on this machine by resolving its address: one that resolves here, as localhost, this machine's own name or network address, 0.0.0.0 or host.docker.internal do, it starts; one that resolves only to another machine, or does not resolve, it starts nothing for. Each service is judged on its own, so with one here and one elsewhere only the one here is started. On Windows the WSL distribution's own address reads as another machine; localhost is the address to use.

You can still start them yourself from compose.yaml, which is the line Vespera runs, without --no-recreate:

docker compose -p vespera up -d --build

-p vespera names the Compose project vespera, whatever your checkout's directory is called. Give it on every docker compose command here, so that each one finds the same containers. Without it, Docker Compose names the project after the directory, and a second checkout starts a second set that fights the first for the same ports.

If your machine has an NVIDIA graphics card that Docker can use, let Ollama and the document converter run on it. Ollama runs its models several times faster there than on the processor, and the converter got through a sample of the archive two to two and a half times faster. On Windows, Docker inside WSL 2 can use one; on Linux, Docker needs NVIDIA's Container Toolkit. Set VESPERA_DOCLING_IMAGE to the card's image, as below: that is what makes Vespera start the containers with a second file, compose.gpu.yaml, named after the first. By hand, the line is:

docker compose -p vespera -f compose.yaml -f compose.gpu.yaml up -d --build

Name both files every time you run up, this time and every time after. An up without compose.gpu.yaml replaces Ollama with one that runs on the processor. The models you gave it are kept, because they live in a volume of their own rather than in the container. It replaces the document converter with the processor build too. stop, exec and down need only -p vespera. Without such a card, leave compose.gpu.yaml out: with it, up stops with an error, and neither Ollama nor the document converter starts. Once a model has answered, docker compose -p vespera exec ollama ollama ps shows under PROCESSOR 100% GPU when the model is wholly on the card, a split such as 30%/70% CPU/GPU when only part of it fits, and 100% CPU when Ollama is not using the card.

With compose.gpu.yaml, the document converter is built from the same Containerfile on a base that can use the card, under a name of its own. Its output differs from the processor build's in a few places, and Vespera records beside every conversion the name of the image that made it. That name is the one you give Vespera, so you have to tell it which image you started. Set VESPERA_DOCLING_IMAGE in the shell you run vespera from, before you run it. In PowerShell:

$env:VESPERA_DOCLING_IMAGE = 'vespera/docling-serve-cu128-libreoffice:v1.32.0-docling-parse-7.17.0-r2'

In a POSIX shell:

export VESPERA_DOCLING_IMAGE='vespera/docling-serve-cu128-libreoffice:v1.32.0-docling-parse-7.17.0-r2'

If you run Vespera from the IDE instead, add this line to the file .env at the root of this repository, creating it if it is not there. The Local SpringApp and Local Vespera Label run configurations in .run/ read it, and a command of java -jar does not; the control panel starts its fields from the .env of the folder you start it from (below):

VESPERA_DOCLING_IMAGE=vespera/docling-serve-cu128-libreoffice:v1.32.0-docling-parse-7.17.0-r2

The converter reports which image it runs, and Vespera checks that against the name you gave it before it converts anything. If you forget, or if you go back to compose.yaml alone and leave the variable set, the two do not match. The command then stops before converting a single file, and its last line names both images: the one the converter runs and the one Vespera was told. Set the variable to the image you meant to run, or start the containers again with the files that build the other one, and run the same command again. If you go back to compose.yaml alone, unset the variable and take the line out of .env again.

The first run after you switch to the card converts every file again, once, because conversions recorded under the processor build's name are not reused under the card's. On the card that is still quicker than finishing on the processor. Every step after the conversion runs again too. The labels you already set in the label file are kept, but the label file is written anew, and you are asked to approve the arrangement again. If you switch back, the processor build finds its own conversions where it left them.

To check that the converter is using the card, run this. It prints True when it is:

docker compose -p vespera exec docling-serve python -c "import torch;print(torch.cuda.is_available())"

To check that Vespera recorded the card's image, run this from your working directory, once a run has converted something, with Python on your machine, while no vespera command is running and no vespera.db-wal lies beside vespera.db (if one does, Where things live says what to do). It prints what the newest conversion run in the ledger was keyed on, and the image is the part after image=. After your first run on the card, that is vespera/docling-serve-cu128-libreoffice:

python -c "import sqlite3;print(sqlite3.connect('file:vespera.db?mode=ro&immutable=1',uri=True).execute('select config_consumed from run where stage=? order by rowid desc limit 1',['extraction']).fetchone())"

The document converter's image is not pulled. It is built on your machine from docker/docling-serve, because it adds LibreOffice to the published image, so that .doc and .ppt files convert. --build builds it the first time and rebuilds it if its Containerfile has changed. The first build is the slow one. After that, Docker reuses what it built.

The services listen on ports 11434 and 5001, which is where Vespera looks for them. If something else on your machine already holds one of those ports, the start fails and names the port.

Leave the containers up for all five invocations. They can be days apart. If they are down when you run the next one, it starts them again.

If a container stops on its own, Docker starts it again, and it does the same after your machine restarts, as long as Docker itself starts. When the document converter goes away while documents are being converted, Vespera waits up to three minutes for it to come back and then asks again about the document it was converting, so a restart of the converter does not stop the command. A document that makes the converter go away twice, or that the converter answers with an error, is marked, skipped and listed in extraction-failures.html, and the command carries on with the rest. If five documents in a row make it go away twice, or are turned down for the converter's own reasons, Vespera does not yet know whether the documents or the converter are to blame, so it checks the converter with a small document of its own. If the converter reads that one, the five were the documents' own doing: each is marked, skipped and listed in extraction-failures.html, and the command carries on. Only if the converter cannot read Vespera's own document either does the command stop, and it says so. One of your exemplars failing that way does stop the command, and the message names the file. If the converter is not back within three minutes the command stops and says so, and you can run the same command again once it is back. A container you stop with docker compose -p vespera stop stays stopped, even across a restart of your machine, until the next vespera run or vespera label --auto starts it. A command that reaches the document converter while it is still starting waits up to three minutes for it to answer, and says so in one line, rather than stopping.

Ollama's models. Ollama serves only the models it has been given. vespera run gives it the three it needs, each when it first needs it, if Ollama does not serve it yet:

  • The embedding model, at scoring in invocation 2. That is hf.co/Qwen/Qwen3-Embedding-8B-GGUF:Q4_K_M, unless you set embeddingModel: Qwen3-Embedding-8B, named by its file on Hugging Face, 4.7 GB.
  • The model that names the topics, at the head of the arrangement, in the invocation that makes it. That is qwen3:8b, unless vespera.naming.model names another; it has no profile key, and the control panel shows it and does not set it.
  • The model the connecting text is written with, at the writing in invocation 5. That is qwen3:8b, unless you set generationModel.

Each can be a model of Ollama's own library, such as bge-m3:latest, or a model file on Hugging Face, named as Ollama names one, such as hf.co/Qwen/Qwen3-Embedding-8B-GGUF:Q8_0. A model named another way is another model to Vespera, even where its file is the same: qwen3-embedding:8b holds the file the embedding model's name holds, and the documents are embedded and scored again under it. Vespera pulls only a model Ollama does not serve: one it already has is never pulled again, so it cannot change under your archive. While it pulls, a line in the log and the bar on the progress page say how many megabytes it has. A pull of several gigabytes takes as long as your connection needs; Vespera gives up only when the pull has gone ten minutes without getting any further, and vespera.models.pull-stalled-at-most changes those ten minutes.

If a pull fails, the command stops before it sends anything to the model, and its line names the model and quotes what Ollama said: a name it does not know, a network it cannot reach, a drive that is full. Fix what that says and run the same command again. A pull that stops part-way, for that reason or because you stopped the command, leaves no half-model behind: Ollama serves the model only once all of it has arrived, and the next pull carries on from what Ollama kept.

To pull them yourself instead, set VESPERA_MODELS_PULL=false. A model Ollama does not serve then stops the command at the step that needs it, with a line naming it. These are the lines:

docker compose -p vespera exec ollama ollama pull hf.co/Qwen/Qwen3-Embedding-8B-GGUF:Q4_K_M
docker compose -p vespera exec ollama ollama pull qwen3:8b

The models run on this machine, the naming model too: modelsRunOn says so, local where the profile states nothing, and Vespera never judges where a model runs from its name or from Ollama. cloud, or any other value, stops vespera run and vespera label --auto at their start, before anything is walked. vespera label --auto pulls nothing. It labels the documents with qwen3:8b, whatever generationModel says, and vespera run pulls that model only at invocation 5 and only where generationModel is unset; so before you first use it, pull qwen3:8b with the second line above.

Pulling the embedding model again can change it, because a name can be published again with other contents. Vespera then treats it as another model: the next vespera run embeds and scores every document again, the relevance threshold removes nothing until the sample is labelled again from the new label file, and the arrangement has to be approved again. Pull it again only when you mean to, and not while vespera run is computing vectors: if the model changes under that step, the command stops there with a line saying so, and the next vespera run embeds again, storing the vectors that are missing under the model as it is now.

Run it. A jar you build is target/vespera-0-SNAPSHOT.jar: only a release, cut by pushing a vMAJOR.MINOR.PATCH tag, carries a version, and its jar, vespera-<version>.jar, is attached to that release's page on GitHub. vespera --version prints it. Each vespera in this file is this command:

java -jar target/vespera-0-SNAPSHOT.jar

So vespera run <root> is java -jar target/vespera-0-SNAPSHOT.jar run <root>, and vespera label is java -jar target/vespera-0-SNAPSHOT.jar label. Name the working directory as "Where things live" describes, with --db-dir=<path> for one: vespera run, vespera label and vespera publish naming none, like the jar started with no argument and no display, refuse in one line, exit code 1, before they make, write or open anything: Vespera is not started: no working directory is named. Name one with --db-dir=, the environment variable VESPERA_WORKING_DIR, or vespera.working-dir in application.yaml beside the jar. An application.yaml beside the jar that an earlier version wrote, whose line still names .vespera under the directory you run the command from, goes on naming it until you change the line; the application.yaml.new beside it holds the shipped line. --help and --version need no working directory. Started with no argument at all, the jar opens the control panel instead (below). To be sure it never does, give --headless as the first argument: java -jar target/vespera-0-SNAPSHOT.jar --headless run <root> is vespera run <root>, and --headless alone prints the commands' usage. Only the first argument is read so; a --headless anywhere else is refused as an option the command does not know.

The configuration beside the jar. The jar keeps its configuration in the folder it lies in, target/ for the jar ./mvnw package builds: application.yaml, compose.yaml, compose.gpu.yaml, and the docker/ folder the converter's image is built from. Its first start writes each of them there from its shipped copy, and every later start writes again any that is missing, so deleting one is how you take the shipped copy back. Change application.yaml there to change a value for good. Vespera starts the containers from the compose files there, not from this repository's; the lines above that start them by hand use the repository's, which are the same until you change either. A value for this machine alone goes in application-local.yaml, beside them, which stands over application.yaml. It need not be there, and the jar never writes it. ./mvnw clean empties target/, your changes there with it, so to keep them, copy the jar to a folder of its own and start it there.

A folder this user cannot write refuses the start, before anything else is made, in one line on standard error, exit code 1. The control panel's start is refused in the same line, and no window opens:

Vespera cannot write <name> in the folder beside the jar, <folder> (<reason>); make that folder one this user can write, and start Vespera again.

A file already there is never written over, by this jar or by a newer one. When a newer jar's shipped copy differs from your file, the jar starts with your file as it is, writes its shipped copy beside it as <name>.new, for you to compare, and says so in a line naming the keys that differ, never their values. A file of docker/, or a YAML file that does not load, is named with no key. vespera logs the line at WARN, so it is in the log and on the console; the control panel says it in its log as the window opens. A key the newer jar needs and your file lacks is among those named. A jar built again in target/ without clean is such a newer jar.

<name> beside the jar differs from this jar's own at <key>, <key>; this jar's is written beside it as <name>.new

To read the configuration from somewhere else, name it with --spring.config.location=<file>, or with the environment variable SPRING_CONFIG_LOCATION. Spring then reads what that names in place of both files beside the jar. The jar still writes any missing file beside itself, and still starts the containers from the compose files there.

Run from the IDE, the folder beside the jar is target/, which a clean build empties. So the run configurations in .run/ also read application-local.yaml at the root of this repository, over the files in target/: keep yours there. .gitignore keeps it out of a commit.

Or build and run in one command, headless. The headless profile builds the jar, skipping the tests, and runs one command of it with the Java that runs Maven, from the root of this repository, with --headless before the command, so it never opens the control panel. Name the command and its arguments in vespera.args, quoted as one value; given none, the jar prints the commands' usage:

./mvnw -Pheadless "-Dvespera.args=run <root>"

On Windows it is .\mvnw.cmd -Pheadless "-Dvespera.args=run D:\corpus", the quotes kept.

Stop the containers when you are finished. Vespera never stops them:

docker compose -p vespera stop

This keeps the models you gave Ollama. docker compose -p vespera down removes the containers and keeps the models too, because they live in a volume of their own. docker compose -p vespera down -v removes that volume as well, and the models with it. After that, vespera run pulls them again when it needs them, or the same ollama pull lines as above give them back.

When compose.yaml has changed since you started the containers, for example after you update your checkout, docker compose -p vespera up -d --build, or the up line above that names both files if you started them with both, replaces the container of each service whose part of compose.yaml or compose.gpu.yaml changed. Ollama's models stay. docker compose -p vespera exec ollama ollama list shows which it has. Nothing in your working directory is lost.

Starting the control panel

One Vespera runs on a machine at a time. While a control panel is open, or a vespera run, vespera label or vespera publish is running, a second control panel does not open and those commands started anywhere else stop at once, before they write anything, with a line beginning another Vespera is running on this machine that names the one running. This keeps one from stopping the containers another is using. The commands the control panel starts itself are not affected, and neither is vespera serve.

The control panel is a window of the jar itself, which runs the same commands one at a time, each as a command of the same jar started with the Java that started the window. Start the jar with no argument, or double-click it, and the window opens; from the root of this repository:

java -jar target/vespera-0-SNAPSHOT.jar

Or build the jar and open the control panel in one command, with the gui profile, which skips the tests and starts the jar with no argument with the Java that runs Maven. --headless, or the headless profile, stays the way to run a command without it (above):

./mvnw -Pgui

On Windows it is .\mvnw.cmd -Pgui. Ctrl+C ends Maven, the control panel and the invocation it started.

The commands it runs start the containers as above. As its window opens the control panel starts nothing: it checks, in a list in its header, whether the Docker daemon, Docling and Ollama answer and whether Ollama has the embedding model and the generative model in force, again every 5 seconds and at once after anything that changes them, each row reading OK, KO, or a dash where it is not asked. It asks Docling and Ollama at the addresses the Services section shows at that moment, so an address you correct there is checked within 5 seconds, with nothing to start first; a field left blank stands for what the environment, .env or application.yaml names. Start Docker, before Shut down Docker, starts Docker and the containers on this machine, for the addresses and the converter's image the Services section shows, waits for them, and then downloads a model Ollama lacks where vespera.models.pull is ticked in Command options; it is off while everything here answers. While it runs, the output, cleared as you press it, shows what it is doing as it does it: Docker's engine starting, docker compose up building and pulling, the wait for the containers, and last the line it ends on; Shut down Docker shows its two stops the same way, and the next Run's output takes their place. Clicking a model's KO downloads the missing models in a panel at the window's top right, one at a time: Hide keeps the download going, the row then reading KO ↓ and how far it has got, which brings the panel back when clicked; Cancel stops it, keeping what was downloaded for the next one; Retry follows a lost connection; Close once all are done. Run waits until the Docker daemon, Docling and Ollama show OK, and, with vespera.models.pull unticked, until the models are there; Auto Label waits for Ollama and the generative model. What a button, a field or a row is for is said when you rest the pointer on it, not in the window. It listens on no port and opens no browser of its own, so no other program on this machine can reach it, and it asks for no password or key.

The jar opens the window on the kind of system it was built on, since a build packs JavaFX, which draws it, for that system alone; every command runs anywhere. Where it cannot open a window, with no display, or as a jar built on another system, it answers as its command line, printing the commands' usage after one line saying why:

the control panel cannot open a window here, so the jar answers as its command line: <reason>

The first start of the window on a machine keeps JavaFX's native libraries under .openjfx/cache in your home folder. A start of the jar that opens the window makes no .vespera in the folder you start it from. Started from no jar, as from an IDE's class folders, it prints one line naming the class path, and exits 1: start it from the jar ./mvnw package builds.

What the window shows. Down the left, eleven sections, one open at a time: Profile, whose Open profile opens the Profile window (below); Preflight, a row each, marked OK or KO, for whether the corpus root and the working directory are there, with each one's free space, who holds either, as vespera run (pid 44072) since 23:19, said once where one command holds both, the holder's whole line on mouse over, and, where Docling runs another image than the one Run would use, both images; Docling's and Ollama's answers are the header's alone; Folders, the corpus root, the list of your working directories, the output folder and, beneath them, the tree of the folders to read and the folders not relevant (below); Services, the converter's image, offered as GPU or CPU, the converter's and Ollama's addresses and the label model; Seed folder; Topics, the topics your pages are grouped under, by name, a row each with its Remove: type a name in the box and press Add, which stays disabled, saying why beneath the box, while it is empty or holds a name already listed, as Pagamenti is already a topic, two names being one where they differ only in capitals, accents or the spaces around them, and Save writes the list into profile.yaml's topics, a list emptied and saved having the topics found in your seed documents again, while a list vespera run would refuse, holding an empty line or a name twice, is drawn as it is, each such row in red with the reason; Labels, the two label commands: Save my labels (vespera label), which reads relevance-labels.yaml in the working directory and saves the labels set there for the next Run, and Auto Label (vespera label --auto), which has the local model read the documents and set their labels and the cut-off, never replacing a label you set; Code sets; Outputs, what is drawn, a box each for HTML pages, Confluence, Website and Retrieval records, the first three ticked where profile.yaml's outputs is unset, unticking HTML pages unticking and disabling Confluence and Website, which are drawn from them, and Save writing the names ticked into outputs; Command options; and Reports. A section still missing a value says needs a value in its title. On the right, Run, Stop, Start Docker and Shut down Docker, Publish once a rendering is drawn, with a box beside it for each destination whose rendering is drawn, and each value the next invocation wants with its Save, always those the working directory chosen asks for, read again as the window opens, as you choose another working directory and as a command there ends. Run, Save my labels and Auto Label are disabled while the control panel's command runs or another holds the working directory, and Run and Auto Label also wait for the header's check, above. Below them, the command's whole output across the window, read from its file in logs once its report names it, any part of which is selected and copied with Ctrl+C, and beneath it, behind a bar you drag to share the height between them, the job's steps, a bar each with its stage beside it, the census shown as 0, the relevance floor as 5e and the relevance report as 5, with a Stop on the running step's row saying what a stop there costs; the steps scroll on their own, and the output opens 300 px high. While a command runs, its report's status line stands in the header under the database's size, marked stale where the report has not been written for three of its intervals. Beneath the output, behind a divider you can move, a small panel says how the last command of the working directory chosen ended and why, whichever way it ended: failed or refused, with the command's own lines, its faults, the step it stopped at, its exit code and the time; a defect of the control panel; Stopped by you at the step it ran; or ended, with its closing line and how long it took. One that ended before the window opened is dated, Last invocation, Mon 16 Nov 18:02, and one whose report never ended, stopped or crashed before its last write, says Ended without its last write and the step it was at; a working directory never used shows none, and nothing shows while a command runs. Where that command wrote a report of its own, the panel also lists the gates it stopped at with the value each wants, the next value and the files it wrote. Open profile stands in it where its line names the Profile window, and Copy details always. Drag the bar at the sidebar's right edge to widen or narrow the sidebar. The steps fold into four parts, as on the progress page: deduplication, relevance, clustering and generation, each one bar of how far it has got, with its state. The part the job is in, the part where it failed, or the part where it first stopped at a gate is open on its steps, and the others are folded. A part you open or fold stays that way for as long as the window is open. A running step or part says how far it has got, as running 37%, rounded down, where it has a count to go by. A step still running when the command ends without its last report, as after Stop, reads stopped, or failed after any other ending, with no bar moving and no Stop of its own. Each bar of the steps, and each folded part's bar, carries how far it has got as a whole percent, centred on it; a step that runs with no count shows … and a step that failed with none shows —. An open part shows its name alone, with no bar and no percent. Beneath an open part's steps, a chart for each of its steps, three to a row at the window's opening size, drawn from the chart-step-<step>.json files of the working directory: each plots the step's counts over its own time and, where the step records verdicts, its verdicts by kind; a chart of two lines or more names them in a legend above it, a point shows its value under the pointer, and the latest point of each line is marked (ADR-365). A step that has finished has its chart frozen, its lines dimmed and the time it finished in its title, with the day before it where that is not today. Each vespera run empties every chart as its job begins, so the charts show that invocation's work: a step it does not reach stays empty, a step it finds already finished under the same run draws again, frozen, the chart it drew in the invocation in which it finished, kept under charts/<run id>/, or says finished in an earlier invocation where no chart of it was kept, and a step that said no count says no count reported. After a restart of Vespera or of the window, the charts of the last vespera run are drawn again as they were. A chart is at least 320 by 240 px, and a narrower window puts fewer to a row. The database's size is a line of the header, read every second (ADR-382, ADR-432).

Where each field starts. A field starts from the first of these that names it: the value you last started a command with, the environment variable of its property's name (VESPERA_DOCLING_IMAGE for vespera.docling.image), the .env of the folder you started the jar from, for the converter's image the GPU image, and application.yaml in the folder beside the jar; the line beneath it says which. Only a value you set in the window, or one you started with before, is remembered for the next start. A value is passed to the command where it is remembered or where the command would otherwise read another, so one left as the environment or application.yaml gave it leaves your application-local.yaml in force. The corpus root and the working directory are always passed. The first row of the list of working directories is the folder something names as the working directory, as it names the jar's, shown only where something does, and choosing it passes no value of yours: the first that names a folder of the system property vespera.working-dir, the variable VESPERA_WORKING_DIR (or VESPERA_WORKINGDIR, vespera_workingdir, vespera_working_dir), and the vespera.working-dir line of application-local.yaml, then of application.yaml, beside the jar, read as the jar reads it, so that the shipped line's db-dir is the system property db-dir or the variable DB_DIR (or DBDIR, dbdir, db_dir). Its words after its name say which: (from the system property vespera.working-dir), (from the environment variable ), (from the system property db-dir), or (from application.yaml beside the jar), or application-local.yaml, where the line's own text names it. Where none names a folder there is no such row, and no .vespera is used in its place. The row chosen is the folder the window reads and the one Run, Save my labels, Auto Label and Publish pass, as --db-dir and as the system property vespera.working-dir, and the command started inherits none of those eight variables, so the folder passed is the folder it opens.

Command options holds four settings the jar ships, and no label file: Save my labels saves the labels of the working directory's relevance-labels.yaml, which the Folders section sets by folder, and a label file kept elsewhere is recorded with vespera label <file> on the command line. The four settings are passed on the command line only where they differ from what the command would read without them, and remembered by none: vespera.progress-interval, vespera.containers.wait-at-most, vespera.models.pull and vespera.models.pull-stalled-at-most. Back to the defaults puts them back. The section shows the command line each button starts, as it would be typed after java -jar <jar>.

Reports lists every report the jar writes in the working directory, by its name, as format-mix; one not written yet is grey. A value's row in the Profile window links each of its reports by the report's title (above), and one not written yet stands there as its title in grey, followed by (not written yet). A report written there opens inside the control panel, never in a browser: in the Reports window, a window of its own beside the main one, which the first report you open opens, each report a tab there that you can close, with a Reload that reads it again; opening a report again selects its tab and brings the window back. Wherever its link is pressed, in this section, on a value's row or inside another report, a report opens there. Closing the last tab closes the window, and closing the window closes its tabs; it opens again where and as large as you left it, until the control panel closes. The window belongs to the working directory, and closes when you name another. A link inside a report to another report opens that report's tab. A link to any other file, a lead document arrangement.html links where it lies in your archive among them, is handed to your system, which opens it with the program it gives that kind of file; and a link to a report not written yet, or to an address that is not a file of this computer, opens nothing, the line at the top saying why. After any such link the report is read again from its file, back at its top.

The Profile window is a window of its own, moved and sized apart from the control panel's, and one form of every value of profile.yaml, a row of one line each: a ▸, the value's title with its key in brackets beneath it, and a control of its kind. Rest the pointer on the title for the full explanation of the value and, where the file states one, its provenance (above). Press ▸, or reach it with Tab and press Space, to open what the value does, the value in use where the profile names none, what a change redoes and the reports that inform it; a row stays open while you work in the same working directory. A list value, codeNameEndings or outOfScopeNameEndings, shows each item as a tag: type one or more in the box after the last, split by spaces or commas, and press Enter to add them; press a tag's ×, or reach it with Tab and press Space, to remove it, or press Backspace in the empty box to remove the last. What is still typed in the box when you press Save is saved with the tags. Save writes what changed, each with set in the control panel, and Reset puts the form back to the file. Closing the Profile window, or the control panel, with a change not saved asks whether to Save, Discard or Cancel, and so does anything that loads the profile again; naming another working directory asks Save or Discard only, and a Save there writes into the directory the form was loaded from; choosing Profile again brings the open window forward, where it was left. The Seed folder section and the values the next invocation wants are drawn as the same rows. The tree in the Folders section, beneath its three fields, is the corpus root's folders, every one ticked at first. Untick a folder holding nothing you want read and press Save: it is written into the profile's foldersNotWalked, so the next walk counts it and never goes into it, and every run after it is new; tick every folder again and save, and the value is taken out. A folder's subfolders are listed when you open it, never before, and no file is listed. Beneath the tree are the folders not relevant, kept in not-relevant-folders.yaml: type a folder from the corpus root, or Browse to it, and press +, or select one and press Remove, and every document of relevance-labels.yaml is labelled at once, not relevant at or below a folder listed and relevant elsewhere, over the labels it held. A folder not relevant is still read and judged; only its documents in relevance-labels.yaml are labelled.

Beside each folder field is a Browse button, which opens your system's folder dialog, and the field says what the control panel finds there: not chosen, found with its free space, to be created, or no such folder. A code set's folder is added with Browse and +, and only a folder outside the corpus root is added, to codeSetFolders: one under the corpus root is refused, since the code sets under it are found by their build files and named by their folders. Nothing is written by Browse: a launch value is written as a command starts, and any other as you save its form. The working directory has to be there before a command starts, and the command is refused with one line if it is not: the first value you save in the Profile window makes it, a folder only you can write in, and nothing else on the control panel does. The folder named for the deliverable need not be, since the command creates it. When Auto Label labels nothing, because there is no label file or one holds labels not yet recorded, the command says why in one line.

Working directories. The Folders section lists every working directory you have started a command in, the one this window started in last on top, after the one something names, where something does, and the one you choose is the one every part of the window reads and every command is started in; the list is kept beside the values you last started with. Each row shows the folder's name, the corpus root its last vespera run walked, the last command's date and how it ended, the furthest part of the steps that run reached and the value wanted next, all read from the folder's report.json and its copies in logs, without opening its database; a folder another process is using says in use by another process since the time it started, and choosing it watches that command, with Run disabled. New… proposes a folder beside the one chosen, or in your home folder where none is chosen, named for the corpus root's folder and today's date, as Letters_20261114, with -2, -3 and on where that name is taken today; edit it if you like, and the first value you save in the Profile window makes it. Browse adds a folder of your choosing after the row something names, chooses it and keeps it in the list at once; Vespera makes no folder for it, though the system's dialog can make one, and an empty folder is a working directory like any. Remove from list takes a row off the list and deletes nothing. Where no working directory is chosen, the list reads No working directory yet. Choose a folder with Browse, or name a new one with New…., the panel beneath the output says No working directory is chosen. Choose one from the list, or press Choose a folder… to pick a folder. with a Choose a folder… button that does what Browse does, and Run, Save my labels, Auto Label and Publish start nothing, saying so in one line, as Run is not started: no working directory is chosen. The first time the window opens, a .vespera folder holding vespera.db in the folder you started it from, which earlier versions used where nothing named a working directory, is added to the list once, with a line saying so, and chosen where no other row is; the window never looks again, so a folder you remove stays off the list. To rename a working directory, rename its folder while no command is using it, and Browse to it; a deliverable folder you named for it then needs to be an empty one again, since its marker names the old path. Run started on a working directory named for another corpus root than the one in the field says so in one line, and starts. Each working directory is resumed under its own profile.yaml, and the ones you do not choose stay as they are. Publish sends what that working directory's last vespera run drew: where that run stopped before the deliverable, the publish sends nothing and says so in one line, though an earlier run's rendering is still there.

The control panel runs until you close its window. Closing it while its command runs asks once, Stop and close or Keep the window; Stop and close ends the command as Stop would. Ending it with Ctrl+C, or by closing the terminal it runs in, also ends the invocation it started, if one is running; a forced kill of the control panel, such as kill -9, taskkill /F or End task in Task Manager, does not, and the control panel started again shows that invocation holding the working directory.

If you moved the corpus-root locks to another folder, the control panel sees that folder only through the environment variable VESPERA_CORPUS_ROOT_LOCK_FOLDER, set in the shell you start it from, which the commands it starts inherit. A folder you name with vespera.corpus-root-lock-folder in a configuration file or on the command line is one it does not see.

Where the run ends

Approving the arrangement is the last thing you are asked for. The fifth invocation writes the connecting text over each group, then a short overview of each topic over its groups' text, and leaves you a tree of HTML pages in your working directory: index.html, a listing of every surviving document, documents.csv, and beside them one folder per topic, numbered in the arrangement's order and named from the topic's name, holding the topic's page and one folder per group, each named and placed the way you approved them. A topic Vespera divided into subtopics holds one folder per subtopic instead, each holding the subtopic's page and its groups' folders. documents.csv gives each document its topic, its subtopic where its topic is divided, its group and their places in the order. A group's folder holds its page, index.html, one file that carries every picture it shows inside it, a PNG or JPEG image among its documents shown there too, and a copy of each of its documents under the document's own file name, made safe to open on Windows and cut to 100 UTF-16 code units, which is 100 characters but for an emoji or another character outside Unicode's Basic Multilingual Plane, each counting as two, the cut keeping an extension of up to 16, and cut further where the name with .part after it, which is what a copy is called while it is being made, would pass 255 bytes in UTF-8, the most a name can hold on the usual file systems of Linux and macOS. Where two documents of one group would share a name, case aside, and whichever way an accented letter is written, which is how Windows and macOS compare names, the one with the lowest occurrence_id in documents.csv keeps it and each of the others takes ~ and its own occurrence_id before its extension, as report~42.pdf; a document named as the page, as the page with .part after it, which is what the page is called while it is being written, or as another's copy with .part after it takes that suffix whatever its occurrence_id. Each copy is made byte for byte from your archive and checked against the digest taken when the document was converted; one changed or gone since then, one the drive had no room left for, one whose name this computer will not take, or one whose name, with every suffix it took, would still pass 255 bytes with .part after it, is not copied, and its entry and its row in documents.csv say why. Every invocation that reaches the tree draws it again, without asking the model anything, and makes the copies still missing. Before the first copy a line counts every document listed and the bytes they hold, though some of their copies may stand already, and a copy found standing moves the count of documents copied on, so a tree drawn again reaches 100%. It opens index.html in your default browser once the tree is written, and its last line names it: The deliverable is at <path>/index.html, with N groups left unwritten and M documents not copied. A tree an earlier version drew, with its copies in a folder documents/, is drawn again in this layout and every document copied again; documents/ stays, and the last line adds that it holds an earlier layout's copies and can be deleted by hand. Any other folder at the tree's top or in a topic's or a subtopic's folder that this drawing did not write is counted there, as N folders left by an earlier drawing can be deleted by hand, and not named; so is any file in a group's folder that is neither its page nor one of its copies, such as a picture an earlier version wrote beside the page, as N folders and M files left by an earlier drawing can be deleted by hand, either count left out where it is 0. After the last copy, where any copy stands, made this time or kept from before, at an absolute path longer than 259 characters, which some programs on Windows cannot open, a warning line of its own says how many and names none: name a folder with a shorter path in vespera.deliverable-root. With none, there is no such line.

Each code set is copied whole to code/<set>/ at the tree's top, its folders below the set's folder kept, each name made safe as a document's copy is; index.html lists the sets, and code.csv holds a row for each file. A set is named for its folder, and where two names meet, case aside, each but the first takes ~ and the lowest occurrence_id among its files. Where a set's folder in code/ is not the one the tree's earlier code.csv gave it, a change of case alone aside, the drawing says so in one line, A code set's folder in code/ is now ‹now›, no longer ‹earlier›, which is left as it is., and the earlier folder is counted on the last line as left behind. A code-set-names.yaml left in the working directory by an earlier version is no longer read, and can be deleted by hand. A file of code is checked against the size and modification time recorded when the archive was walked; one that differs is changed and not copied. Where the run has a code set, the last line adds , and K files of code not copied.

A group's page opens with the heading written for it, carries the prose with every citation resolved into a link to that document's numbered place in the list below, and then lists the group entire — including the documents the writing never mentioned. Each entry in that list links to its document's copy inside the tree, so a sentence you doubt is two clicks from the document behind it, and the tree can be carried elsewhere and opened with every link still working. Where a group was written from part of its documents, the page says so and names both numbers. Where an answer was turned down, the group keeps its place and its page says plainly that nothing was written over it, rather than leaving a gap you have to notice. Once every group left unwritten is one whose answer was turned down the last two times it was asked, nothing else is missing, and writing was not stopped by five answers turned down in a row, the work is recorded as done: each such group's page says it stands without writing, running the same command again asks nothing more about it, and the pages can be published. Where clusterPageByteCeiling is set and a group's page would be larger, the group's folder holds index.html and index-2.html on, each listing the next of its documents, numbered on from the page before, with links to the first, previous, next and last page above and below the list; the index links the first. A page an earlier drawing wrote past the last one is counted on the last line with the other files left behind, and is not deleted.

A topic's page carries its name, then its overview, in which every sentence cites the groups it is drawn from by their numbers in the topic, each number a link to that group's page; where the overview covers only some of the topic's groups, a line beneath it says how many. Where no overview could be written, a sentence in its place says why. Below it stands the table of the topic's groups, numbered, with their documents and titles, or, for a topic divided into subtopics, the table of its subtopics, each linking to its page, whose table lists that subtopic's groups. A group keeps the one number in its topic, whichever subtopic it lies in. Last, under See also, the topic's page lists the documents placed under another topic that also speak of this one, each linking to the first page of its group: a document is copied once, under its own topic. index.html heads each topic with its name, linking to its page, and lists its groups beneath it, or its subtopics and theirs.

Beside the HTML pages the same drawing writes the Confluence rendering under confluence/: the same groups, prose, citations, entries and pictures, as page bodies in Confluence's storage format, one page.xml for the root page, one for each topic, its overview above its table of groups or subtopics and its See also below, one for each subtopic and one for each group (page-<k>.xml for a group's further pages), each picture a file of its own beside its group's body, and pages.json listing every page, its parent, its title, its body and the files it carries with their sizes and digests. Each title ends with its place and the code of the language the deliverable is drawn in, as Payroll (01.02 · en) for a group, Payroll (01.02 page 2 · en) for a group's second page, Payments (01 · en) for a topic's page, Salaries (01.2 · en) for a subtopic's, Payroll (01.2.03 · en) for a group in a subtopic, and Pagamenti (01.02 pagina 2 · it) in Italian, and the root page is Vespera_en or Vespera_it; no title and no page carries the run's id, which the HTML pages still name. drawing.txt beside pages.json says drawing while the rendering is drawn, drawn once it is finished, and failed where it could not be. A Confluence rendering that cannot be drawn stops alone: the HTML tree is drawn whole, the invocation ends as it would have, and its last line adds the confluence rendering could not be drawn (), so a publish to it is refused until a drawing finishes. Nothing is sent: the rendering waits for vespera publish, below.

The same drawing writes the website rendering under website/: the same groups, writing, citations, entries and pictures, split at the same documents as the HTML pages, with index.html at its top, search-index.js, website.js, website.css, documents.csv and files.json, and drawing.txt saying how its drawing went. Its copies are not written there again: files.json names each where it stands in the tree. It is what vespera publish --to website reads, not a website to open, since its pages link copies that stand beside them only once a publish has copied them. A website rendering that cannot be drawn stops alone, as the Confluence rendering does, and the last line says so the same way.

Those files are where the run stops. No stage sends them anywhere; the one thing that does is vespera publish, and only when you run it (ADR-308). Your archive is untouched throughout: each document is opened only to be read and copied, and nothing is written there (ADR-274). What you do with any of it is yours.

Publishing to Confluence

vespera publish, or vespera publish --to confluence, sends the Confluence rendering your last vespera run in that working directory drew to your Confluence Cloud site, as pages under one page or folder you name in one space you can write to, with each group's copies and pictures as files on its page. It sends nothing else: not the HTML pages, not the database, not a log. It reads no document's text and runs no model. It is Cloud only.

A rendering whose drawing did not finish is not sent. Where that run's confluence/drawing.txt says drawing or failed, the publish sends nothing, touches no page, and says vespera publish sent nothing to confluence: the last drawing of its rendering did not finish. Run vespera run to draw it again, and publish again. An earlier run's rendering is not sent in its place. A rendering drawn before drawing.txt existed is sent as it always was.

The token. Make a classic API token for your Atlassian account at id.atlassian.com → Security → API tokens. A token lasts at most a year, and your organisation's admin may turn tokens off. Vespera reads it, with your account's email, from the environment and from nowhere else, and writes neither to any file or log.

The five values, and the root page's title where you want another, set in the shell you publish from, for each publish, or, for the first four and the title, in the profile (below):

Variable What it holds
VESPERA_CONFLUENCE_BASE_URL your site's address, https://<your-site>.atlassian.net
VESPERA_CONFLUENCE_SPACE_KEY the key of the space the pages go in
VESPERA_CONFLUENCE_FOLDER_ID or VESPERA_CONFLUENCE_FOLDER_TITLE the page or folder they go under, by its id (the number in its address) or by its title; name one, not both
VESPERA_CONFLUENCE_EMAIL your account's email
VESPERA_CONFLUENCE_ROOT_TITLE the title of the top page the others go under, {language} standing for the language's code, read from the environment alone, the shell's value coming before the profile's confluenceRootTitle; empty is Vespera_{language}
VESPERA_CONFLUENCE_TOKEN the API token

VESPERA_CONFLUENCE_USER, VESPERA_CONFLUENCE_PARENT_ID and VESPERA_CONFLUENCE_PARENT_TITLE are no longer read: set the names above instead. A confluence-env.sh or confluence-env.ps1 already in your working directory that sets the old names is not written over; edit it, or delete it and the next publish missing a value writes it afresh.

In the profile. The site, the space, the parent, the email and the root page's title are also keys of the profile, confluenceBaseUrl, confluenceSpaceKey, confluenceFolderId or confluenceFolderTitle, confluenceEmail and confluenceRootTitle, which the Profile window of the control panel edits like any other. A value the shell sets, not blank, comes before the profile's; the parent is taken whole from one side, so a shell naming it by title leaves the profile's id unread. The token is in no key: it is read from VESPERA_CONFLUENCE_TOKEN alone. A profile that is missing or does not load gives no value, and the shell alone is read.

From the control panel. Publish to Confluence asks for the API token each time in a box that shows nothing of what is typed, and passes it to that publish alone: it is written to no file, the profile and the window's own state among them. Ticking only the website asks for none. A publish the control panel starts reads none of the VESPERA_CONFLUENCE_* variables of the window's environment, so the profile is its one source of the four values, and where one is missing it says which key to set in the Profile window and writes no template.

On the command line, a publish missing any of them sends nothing, names what is missing, and writes confluence-env.sh and confluence-env.ps1 into the working directory where they are not there already. Fill in the site, the space and the parent in one, then run it in your shell, . ./confluence-env.sh or . .\confluence-env.ps1: it asks for the email and the token each time and writes neither down. Then:

vespera publish --db-dir=<path>

An address that is not https, or that carries a user, a query or a path past /wiki, is refused before anything is sent, and nothing is sent to any other host. A parent named by title must be the one page or folder of the space so titled.

What it does. It goes through the pages one at a time, parents first: it creates a page that does not stand yet, sends its files, then its body, and only then goes on to the next page, so a page never shows a file it has not been sent, save one the site refused as too large. A link to a page not created yet shows as a link to create it until that page is reached, and as a link to it from then on. Before it writes anything, it checks that no page in the space already carries one of the rendering's titles without being a page of this tree. Every page carries the label vespera-en or vespera-it, its language's, so you can find one language's pages by it.

One tree for each language, updated in place. Under the page or folder you name there is one tree for each language the deliverable is drawn in, Vespera_en and Vespera_it by default side by side, and every publish brings the tree of its rendering's language to the rendering it sends, whichever run drew it, and touches no page of the other language's tree: the root page, each topic's, each subtopic's and each group's page are matched by their place, 01, 01.2, 01.02, 01.2.03 or 01.02 page 2. A page whose body has not changed is not sent again, a changed one is sent as its next version, retitled where its title changed, and a file already on its page at the same size and digest is skipped, a changed one replaced. A group that has another place in a later run is drawn on the page of its new place, so a page keeps its address, its history and its comments with its place, not with a group. Once every page of the rendering is sent, the pages of the tree the rendering no longer has, from a group that is gone or a group with fewer pages, are moved to the space's trash, deepest first, where a space admin can restore them; nothing is deleted for good. A page with anything still under it, a page someone added or a page of the tree left standing for that reason, is left as it stands, so nothing put under the tree goes to the trash with it, and so is one the site refuses to delete, as it does where your account may not delete pages in the space: its line says so and the closing line counts it, the publish carries on, and the next publish tries again. A page in the space that carries one of the rendering's titles but is not a page of this tree stops the publish before it writes anything, and is left as it stands; so one space holds one tree of each language. A language's tree must stand directly under the page or folder you name for its publish: if it stands elsewhere the publish stops, naming its root page's title and where it stands. Where the site asks it to slow down, it waits as long as the site says and sends again.

A tree an earlier version published, titled Vespera, or Vespera and 12 characters of its run's id, is taken as a language's tree by the first publish in that language that finds it directly under the page or folder you name and no tree of its language there. Its language is the one its root page's heading names, Deliverable for English and Base di conoscenza for Italian: a publish in the other language leaves it as it stands, says so in a line, and makes a tree of its own. Taken, its pages, those standing under its root page, keep their addresses, are retitled with the language's code and updated in place, their files are not sent again, and a line before the first page says so. A root page whose heading names neither stops the publish before it writes anything, since its language cannot be told: put the heading back, or move the page away. So do two such trees of one language under it, naming both: move one to the trash and publish again.

What it tells you. First, before it sends anything, the tree it sends to, Vespera_en or Vespera_it by default, or the title confluenceRootTitle / VESPERA_CONFLUENCE_ROOT_TITLE gives, how many pages and files it is about to publish and how many bytes the files hold. Then, as each page starts, page 3 of 74: with its title and its files and bytes; a line for each file, file 41 of 3509: with its name and size, as it is sent, found unchanged or refused; and once the page's body is sent, a line saying whether the page was created, updated or unchanged and how many of its files were sent. Then, where any page of the tree is to go, how many, and removal 2 of 5: with its title and whether it was moved to the trash or left as it stands, and why. Last, a closing line naming the tree and counting the pages and files, and the pages removed and left. The page and file numbers tell you how far it has got even once the first line has scrolled away. A copy or a picture's file the site refuses as too large is skipped, its entry still naming it: its line says so, and the closing line counts those and their bytes; no file lists them, so keep the output if you want the list, and the next publish tries them again. A confluence-not-carried.csv an earlier version wrote in the working directory is removed by the next publish, which says so on a line after its first; where it cannot be removed, the line says so and the publish carries on. Any other refusal stops the publish with a line saying what the site answered and, once it had reached the pages, which page it stopped at: the pages before it are complete, that page may lack some of its files or its body, and the pages after it are not created yet, or stand as an earlier publish left them. What was sent stays, and publishing again goes on from there.

What Confluence does with it, as measured on a Cloud site: a title holds at most 255 characters and is unique within a space, which is why each carries its place; no page body or file up to 10 MiB was refused; and Confluence rewrites a body as it saves it, so a page read back is never byte for byte the body sent, which is why a publish compares what it sent, never what it reads back.

The control panel offers Publish beside Run and Stop once a rendering is drawn, with a box beside it for each destination whose rendering is drawn, confluence, retrieval and website, in that order, Confluence's ticked and the others not. Pressed, it starts vespera publish --to with the destinations ticked; with none ticked it starts nothing and says so in one line. The boxes are no launch value and are not remembered from one start of the control panel to the next. The control panel never starts vespera serve. Its child inherits the five values from the shell the control panel was started in, and the control panel neither asks for nor holds any of them.

Earlier deliverables. Beneath the list of working directories, the Folders section lists the deliverables the chosen working directory's runs drew that are still there, newest first: one row for each, dated by the last vespera run that reached it, with its corpus root, the folder it lies in and what it drew, the one your last run reached marked (latest). A deliverable whose folder is gone, or whose named folder a later run has drawn another in, is not listed; No earlier deliverable of this working directory still stands. says when none is. Nothing is chosen as the list is read, again whenever another working directory is chosen and when a command ends, so Publish delivers what your last run reached. Choose a row and Publish offers a box for each destination that deliverable drew, and starts vespera publish --invocation <id> --to with the ones ticked; with no row chosen the boxes are those of the latest. Publishing a row other than the latest to Confluence or the website asks once first, The deliverable of replaces what is published to <Confluence, the website, or Confluence and the website>, and what it does not hold is taken off there., Publish or Cancel, before the API token is asked for; Cancel starts nothing. Clear the choice goes back to the latest.

Publishing the website

vespera publish --to website copies the website rendering your last vespera run drew into one folder, website/ in the working directory, whatever folder the deliverable is drawn in, and ends. The folder's name holds no run, so a link to it, or your own sync of it to a share or another machine, points at one place through every run. It sends nothing off the machine and reads none of the Confluence values: reaching the website from elsewhere is your sync's or your proxy's.

What it holds. index.html, listing every topic, subtopic and group; a page for each topic and each subtopic, as in the HTML pages; a page for each group, index-<k>.html for its further pages where clusterPageByteCeiling splits it, split at the same documents as the HTML pages, so each is the HTML page and a few hundred bytes more; the copy of each document a page lists, beside it; and documents.csv. Every page has a search box and a link back to the contents, is drawn dark, whatever your system's setting, and takes one column in a narrow window. The search looks through each group's title, label and writing, the names of its topic and subtopic, and the names of its documents, in your browser: it reads no document's text and asks no server, so it works on a page opened from a file and on a copy with no network. Its script and theme are Vespera's own, shipped in the jar, and no page loads anything from the internet. Its fixed words, Search, Nothing found, Contents and the rest, are in the deliverable language. The code sets and code.csv are not on it, because a configuration file can hold a password and a sync would carry it; they stay in the deliverable.

What a publish changes. It writes only the files whose content is not already there, copies first and index.html last, each beside its place and then moved in, and removes the files its own earlier publishes wrote that the rendering no longer has, with a folder that leaves empty. Nothing else in the folder is touched: a file it wrote that you have since changed is kept and counted, and a file of yours stays, unless it has a name the website uses, which is replaced. Its last line is vespera publish refreshed the website at : N files written, N standing, N removed, N changed by hand and kept. Open /index.html, or run vespera serve. A file of the rendering whose size or content is not what files.json says stops it, before that file is moved in: draw it again with vespera run. A publish stopped says vespera publish stopped the website: . N of N files written. Publish again to go on from there., the reason naming the kind of error where a file could not be read or written; publishing again goes on from there and leaves nothing behind it cannot remove. vespera-website.json keeps the list of what it wrote, and vespera-deliverable.txt the one line website; neither names a path.

The website takes, on the working directory's drive, a second copy of every document it lists.

Serving it. vespera serve [--db-dir=<path>] serves the folder to a browser on this machine alone: it listens on 127.0.0.1, which nothing changes, at the port Commands says, with no login, prints serving at http://127.0.0.1:/ until this process ends, opens no browser, and serves until you end it with Ctrl+C or by closing its terminal. It refuses in one line, exit code 1, making nothing: a working directory that is not there, vespera serve has nothing to serve: is no folder that is there.; a website folder holding no vespera-website.json, vespera serve has nothing to serve: holds no published website. Run vespera publish --to website first.; and a port another program listens at, vespera serve could not listen on port : it is in use. Name another in vespera.serve.port. It takes no lock and opens no database, so vespera publish --to website can refresh the website while it serves, each file arriving whole, and a vespera run or another publish is not refused for it. It answers only GET and HEAD addressed to 127.0.0.1:<port> or localhost:<port>, and refuses a request another website open in the same browser sends. A folder is answered by its index.html, never by a list of what it holds, and vespera-deliverable.txt, vespera-website.json and a file being written are not served. Every copy, and anything of yours in the folder, is served sandboxed, so a document that is a web page runs no script there; the website's own pages, script, theme and listing are not. It writes no log of what is asked. Anything on this machine that can reach 127.0.0.1 can read the website while it runs.

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages