feat: Web console: replace Ace with CodeMirror 6

[vogievetsky]

Note from the human

When the console was started I used Ace because it was the only editor that could support all the features the console needed. Ace is really a pain to use, the documentation is basically non existent. Everything the console does with Ace is basically a hack. I have always dreamed of switching to one of the more popular editors but it always felt like a mountain of effort. Well Claude Opus 5.5 did it, and tidied up all the code to remove all the Ace-isms that were all over the place, and it fixed some bugs in highlighting and completions. Otherwise everything is pixel perfect (except for a few nothingburger edge cases). You can see a migration plan file that I checked in then deleted if you want to follow the reasoning. I reviewed the results after each commit.

We are living in the future.

This branch also migrates the console build to mise as volta is deprecated and bumps http-cache-semantics to avoid a reported npm issue

---

AI Summary

The web console's code editors moved from Ace (ace-builds + react-ace) to CodeMirror 6. All the editors now share one CodeEditor component: the workbench, the explore view's SQL inputs, JSON inputs, spec and value dialogs, history and explain. Two Lezer grammars, one for DruidSQL and one for Hjson, drive the highlighting and the completions.

The goal was to keep the console looking and behaving the same. On top of that, the code was cleaned up until it reads as if the console had used CodeMirror from the start, with no Ace-shaped leftovers. Where the behavior changes, it's either a bug fix or listed below.

What changed

Editor (src/components/code-editor/)

• CodeEditor: a controlled component (value / onChange). The props that change at runtime are applied through CodeMirror compartments: language, read-only, line numbers, padding, transparent background and placeholder. ref gives the CodeMirror EditorView.
• stateCacheId / forgetEditorState: they keep the undo history and selection between mounts, using EditorState.toJSON. They replace AceEditorStateCache.
• Theme (code-editor-theme.ts): reproduces the previous look (solarized_dark colors with the console's overrides).
• Tooltip host: the autocomplete list and its doc panel render into a shared container at the start of <body>, so dialogs and popovers don't clip them.
• Find/replace panel: built from Blueprint components (search-panel.tsx) instead of CodeMirror's default panel.
• Inline parse errors: showEditorError (error-mark.ts) underlines a parse error in the text and shows the message as a data-tooltip on hover. JsonInput and the workbench (for JSON queries) use it.
• Docs: a README for the component, and CLEANUP.md, which records what was cleaned up and which Ace behaviors were kept on purpose.

Languages (src/editor-languages/)

• dsql.grammar: DruidSQL tokens and nested parentheses.
  • An external specializer decides which identifiers are keywords, functions, constants or types.
  • getDsqlLanguage(availableSqlFunctions) makes a language that also knows the functions the cluster reports.
  • This replaces the global initAceDsqlMode.
• hjson.grammar: a real Hjson grammar: objects, properties, arrays, the root object without braces, strings with escapes, and comments. An external tokenizer handles quoteless keys and quoteless strings.
• dsql(options) / hjson({ jsonCompletions }): each returns a LanguageSupport that brings its own completions and bracket closing.
• Generated parsers: script/build-grammars.mjs builds them at build time. They are gitignored, like lib/sql-docs.ts, and run as part of script/build and npm run test-unit.
• Keywords file: lib/keywords.ts moved to src/editor-languages/dsql-keywords.ts.

Completions (src/editor-completions/)

• Shape: completions are CodeMirror Completions with a structured doc (name, syntax, description), not docHTML. The SQL docs are now "doc markdown" rendered as DOM, with no HTML strings; snarkdown was removed.
• Context comes from the syntax tree, not from scanning text:
  • comments and literals are detected from the tree;
  • the keyword before the cursor is found through the tree, including one on an earlier line or past a comment;
  • the SQL references and literals are collected from tree nodes;
  • the Hjson context (hjson-context.ts, which replaces the 400-line scanner in utils/) now sees the properties after the cursor too.
• Word characters per language: DruidSQL treats $ as a word character. Hjson treats $ and - as word characters.

Positions

• 0-based → 1-based: the 0-based RowColumn is replaced by the 1-based LineColumn everywhere, which matches how Druid, Hjson and CodeMirror number lines.
• Renamed helpers:
  • offsetToRowColumn → offsetToLineColumn
  • DruidError.startRowColumn → DruidError.startLineColumn, and the same for end
  • getRowColumnFromIssue → getLineColumnFromIssue
  • QuerySlice.startRowColumn → QuerySlice.startLineColumn, and the same for end
  • extractRowColumnFromHjsonError → extractLineColumnFromHjsonError
• Moving the cursor: focusEditorAt(view, position) replaces the goToPosition imperative handles.

Workbench

• Run-query gutter buttons: these are now a CodeMirror extension (sub-query-markers.ts).
  • A state field finds the queries in the text.
  • Each marker is a real DOM run button with a "Run this query" tooltip, replacing the CSS pseudo-element drawing.
  • Hovering a button highlights its query.

User-visible changes

• Find panel: a Blueprint find/replace bar at the bottom of the editor replaces Ace's floating search box. It has match case, regexp and by-word options, and a replace row unless the editor is read-only.
• Hjson highlighting is correct:
  • Values containing colons (timestamps, host:port, URLs) are no longer partly shown as keys. That affected 97 of the 452 valid JSON examples in the Druid docs.
  • Quoteless values are highlighted as whole strings.
  • A word that is being typed inside an object is shown as a key.
  • An unclosed quoted string ends at the end of its line.
• SQL highlighting:
  • An unclosed '… or "… is colored as a string or identifier up to the end of the line.
  • A sign is an operator: a-1 is a, -, 1.
• Parse errors: they are underlined in the editor, with the message on hover.
• Completions:
  • A function with several signatures is listed once, with every signature in the doc panel.
  • Docs like ARRAY<STRING> are no longer swallowed as HTML.
  • The word being typed is not suggested back.
  • { is not auto-closed in SQL.
  • In SQL, - no longer joins words, so SELECT a-co completes co.
  • Hjson completion rules can depend on properties written below the cursor.
• Undo: CodeMirror groups typing into undo steps a little differently from Ace.

Dependencies and licenses

• Added: @codemirror/{autocomplete,commands,language,search,state,view} and @lezer/{common,highlight,lr}. @lezer/generator is a dev dependency, used only at build time.
• Removed: ace-builds, react-ace and snarkdown.
• Licenses: licenses.yaml and licenses/bin were updated. The CodeMirror packages and their dependencies (style-mod, w3c-keyname, crelt, @marijn/find-cluster-break) are all MIT. The ace-builds, react-ace and fast-equals entries were removed.

Testing

• Unit tests: npm run test-unit passes, with 850 Jest tests.
• New specs cover:
  • the grammars (node names, unterminated tokens, Hjson edge cases, highlight styles);
  • the Hjson context;
  • SQL and Hjson completions, the completion source and its word characters;
  • doc rendering, the error mark, the sub-query markers;
  • CodeEditor itself (find/replace, state caching).
• Snapshots: the snapshot serializer replaces each editor with a comment giving its language, value, placeholder and read-only state. In the updated snapshots only the editor wrapper markup changed.
• Corpus check: the grammars were compared with the old Ace rules on the Druid docs examples. SQL matched in 447 of 448. Valid JSON is now always correct, where the old rules got 97 of 452 wrong.
• Manual check in a running console:
  • workbench highlighting, completions and docs;
  • run-query markers;
  • undo history surviving tab switches;
  • find/replace;
  • JSON error underline and tooltip;
  • explore-view SQL inputs.

---

Prompt for migrating a fork or feature branch that still uses Ace

You are updating a branch of the Apache Druid web console (web-console/) that was written against the old Ace
editor (ace-builds + react-ace) so that it works on top of master, which now uses CodeMirror 6. Keep the branch's
own features and behavior; only change how they talk to the editor.

Before changing anything, read web-console/src/components/code-editor/README.md (the CodeEditor component, languages,
completions, positions, styling, testing) and web-console/AGENTS.md (commands, generated files, conventions).

Setup: Node is pinned in web-console/.node-version and installed with mise (do not change the global Node). After
merging/rebasing, run `npm install` and `npm run compile` (it generates lib/sql-docs.ts and the Lezer parsers
src/editor-languages/*.parser*.ts, which are gitignored; typecheck fails without them).

Old -> new mapping:

Editor component
- `import AceEditor from 'react-ace'` -> `CodeEditor` from src/components (code-editor/code-editor.tsx).
  - mode="dsql" / mode="hjson" -> language={dsql(...)} / language={hjson(...)} from src/editor-languages/.
  - showGutter -> showLineNumbers; focus -> autoFocus; width/height -> style or a className with CSS;
    theme, name, setOptions, editorProps -> remove (the theme is built in). Extra behavior -> CodeMirror extensions
    via the `extensions` prop (see src/views/workbench-view/flexible-query-input/sub-query-markers.ts for an example).
  - `onLoad`/editor instance access -> `ref` (gives the CodeMirror EditorView).
- Imports of src/bootstrap/ace(.scss) -> remove. `.ace_*` CSS -> `.cm-*` classes or code-editor-theme.ts.
- AceEditorStateCache -> the `stateCacheId` prop, and `forgetEditorState(id)` when the state is no longer needed.

Languages and completions
- src/ace-modes/* -> src/editor-languages/ (dsql.ts, hjson.ts, Lezer grammars). `initAceDsqlMode(fns)` is gone:
  pass `availableSqlFunctions` to `dsql({...})` (get them with `useAvailableSqlFunctions()`).
- src/ace-completions/* -> src/editor-completions/. Completions come with the language:
  `dsql({ columnMetadata, columns, availableSqlFunctions, skipAggregates })`, `hjson({ jsonCompletions })`. Remove any
  custom Ace completers / getCompletions props.
- Ace completion fields -> EditorCompletion (CodeMirror's Completion): value -> label, caption -> displayLabel,
  meta -> detail, score -> boost (range -99..99), docHTML/makeDocHtml -> doc: { name, syntax?, description?,
  descriptionMarkdown? } (plain data, no HTML).
- getSqlCompletions/getHjsonCompletions are now builders `(context: CompletionContext, word: CompletionWord, options)`;
  getSqlLiterals/getPossibleSqlReferences take an EditorState (with dsql()) instead of text.
- getHjsonContext moved from src/utils to src/editor-completions/hjson-context.ts: `getHjsonContext(state, pos)`, no
  `isEditingComment`, `currentKey` only when editing a value.
- lib/keywords.ts -> src/editor-languages/dsql-keywords.ts.

Positions (0-based -> 1-based!)
- RowColumn { row, column } (0-based) -> LineColumn { line, column } (1-based). Convert any arithmetic: drop the
  `- 1`s that converted Druid/Hjson positions to Ace rows, and use `line - 1` where a 0-based line count is needed
  (e.g. `changePrefixLines(slice.startLineColumn.line - 1)`).
- offsetToRowColumn -> offsetToLineColumn; extractRowColumnFromHjsonError -> extractLineColumnFromHjsonError;
  DruidError.startRowColumn/endRowColumn (and extractStart/EndRowColumn) -> startLineColumn/endLineColumn;
  WorkbenchQuery.getRowColumnFromIssue -> getLineColumnFromIssue; QuerySlice.startRowColumn/endRowColumn ->
  startLineColumn/endLineColumn; QueryErrorPane's moveCursorTo takes a LineColumn.
- goToPosition(rowColumn) on FlexibleQueryInputHandle / SqlInputHandle -> the ref is the EditorView:
  `focusEditorAt(view, lineColumn)`. To mark an error in the text: `showEditorError(view, { position, message })`.

Component props
- SqlInput: editorHeight -> height, showGutter -> showLineNumbers, ref -> EditorView.
- FlexibleQueryInput: showGutter -> showLineNumbers, editorStateId -> stateCacheId,
  leaveBackground -> transparentBackground (inverted: leaveBackground={true} becomes transparentBackground={false};
  the default is still transparent), ref -> EditorView, new readOnly.
- JsonInput: focus -> autoFocus, width removed (size it with CSS), new readOnly.

Tests
- Ace API in specs (editor.getSession(), setValue, etc.) -> `EditorView.findFromDOM(el.querySelector('.cm-editor'))`
  and `view.dispatch({ changes })` (wrap in act() when React state changes).
- Completion specs: `completionContextAt(dsql(), 'SELECT * FROM t WHERE |')` from src/test-utils/completion-context.ts.
- DOM snapshots show each editor as a comment like `Code editor, language: dsql, value: "..."`. After updating
  snapshots (`npx jest -u <files>`), check that only editor markup changed.

Process
1. Merge master, resolve conflicts using the mapping above.
2. Grep the branch for leftovers until none remain: `ace`, `Ace`, `react-ace`, `ace-builds`, `RowColumn`, `rowColumn`,
   `.row`, `showGutter`, `editorHeight`, `editorStateId`, `leaveBackground`, `docHTML`, `caption`, `meta:`, `score:`,
   `goToPosition`, `initAceDsqlMode`, `ace-modes`, `ace-completions`, `lib/keywords`.
3. Run `npm run test-unit` (generators, typecheck, eslint, stylelint, prettier, jest) and fix everything.
4. Start the dev server (check whether port 18081 is already in use first) and check the branch's features in the
   browser, especially anything that positions the cursor or reads positions (off-by-one errors are the most likely
   mistake).
5. Summarize what was migrated, anything that could not be mapped one to one, and any behavior differences.

[FrankChen021]

🟢 Approval recommended

The body-level tooltip container avoids clipping, but the autocomplete and documentation styles remain scoped to the editor wrapper, so these popups lose the intended console styling. This is a presentation issue and does not affect query execution, so it can be addressed after merging.

Reviewed 100 of 100 changed files.

Validation: git diff --check 1265b47c9a51252295a4aed42d95c459b00bf2a3...ef4a7b16e0ad63a1139a5342d284c0c1e4fd1c22 passed. Static review only; no tests or builds run.

Severity	Findings
P0	0
P1	0
P2	0
P3	1
Total	1

---

This is an automated review by Codex GPT-5.6-Luna(max)

After addressing the findings or replying to the comments, you can request another review from me to trigger a new automated review.

[FrankChen021]

[P3] Keep tooltip styles on the external host

Finding: This moves CodeMirror tooltips outside the editor element, but the custom tooltip rules in code-editor-theme.ts are emitted by EditorView.theme and scoped beneath the editor's generated theme class. The body-level host is not beneath that class, so the autocomplete list and documentation pane lose the console's intended colors, sizing, and pre-wrapped multiline signatures.

Suggestion: Add equivalent styles scoped to the tooltip host, such as through a base or global theme, so the body-level tooltip container receives the custom editor styling.