feat(deepnote): load integrations from .deepnote.env.yaml and .env#440
Draft
tkislan wants to merge 112 commits into
Draft
feat(deepnote): load integrations from .deepnote.env.yaml and .env#440tkislan wants to merge 112 commits into
tkislan wants to merge 112 commits into
Conversation
…d) + add project-id resolver
Chunk 1 of single-notebook migration (§4 partial, §5). No behaviour change.
- Manager caches originals in a nested Map<projectId, Map<notebookId, project>>
so sibling files sharing a project.id no longer clobber each other.
- New API: getOriginalProject(projectId, notebookId) exact/no-fallback,
getAnyProjectEntry(projectId), storeOriginalProject/updateOriginalProject
(3-arg), updateProjectIntegrations iterates all entries.
- Update IDeepnoteNotebookManager and IPlatformDeepnoteNotebookManager; repoint
all project-level read-only callers to getAnyProjectEntry.
- Add canonical readDeepnoteProjectFile and resolveProjectIdFor{File,Notebook}.
- Selection state and init-run tracking intentionally kept (removed in later chunks).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01URccsVKXeNKZqqPi89L4ro
…rop selection machinery Chunk 2 of single-notebook migration (§1 + Cleanup). - deserializeNotebook renders the first non-init notebook (findDefaultNotebook), falling back to the only/init notebook; never composes init. - serializeNotebook resolves the target from document metadata alone (projectId + notebookId required) and looks it up with the exact getOriginalProject, throwing clear errors instead of falling back to a wrong sibling. - detectContentChanges collapses to a single-notebook comparison. - Remove the ?notebook=<id> selection machinery: findCurrentNotebookId, the manager's selection state + interface methods, the explorer's query-param opens and selectNotebookForProject calls, and the tree item's custom resourceUri. - Explorer no longer depends on IDeepnoteNotebookManager. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01URccsVKXeNKZqqPi89L4ro
…k siblings Chunk 3 of single-notebook migration (§0, §2, §3). - Add allocateSiblingUri: the single filesystem-aware, collision-safe sibling filename allocator (bumps -2/-3 before .deepnote, honors an in-batch reserved set, bounded retries). - Add a notebook file factory (buildSingleNotebookFile / buildSiblingNotebookFileUri) for creating sibling single-notebook files (wired into the explorer in a later chunk). - Add DeepnoteMultiNotebookSplitter: on opening a multi-notebook .deepnote file, offer to split it into one new single-notebook file per notebook. The action flushes the editor if dirty, writes all children, migrates the environment selection, then closes the tab and deletes the original to trash. A child-write failure leaves the original intact (write-before-delete). - Wire the splitter into activation with an optional (desktop-only) environment mapper; add a refresh() passthrough on the explorer. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01URccsVKXeNKZqqPi89L4ro
Chunk 4 of single-notebook migration (§6). - Add DeepnoteProjectMetadataPropagator (desktop): given a project id and a project-level mutator, enumerate every sibling .deepnote file on disk (open or closed), apply the change, and write it back. Skips no-op writes, refreshes the manager cache for open siblings, and collects per-file failures instead of aborting. Fires an onFileWritten hook so the file watcher treats each write as a self-write (no reload/save storm). - Route integration updates and project rename through the propagator so closed siblings stay consistent; web falls back to the cache-only / single-file paths. - Expose getOriginalProject/updateOriginalProject on the platform manager interface. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01URccsVKXeNKZqqPi89L4ro
…us bar Chunk 5 of single-notebook migration (§7). - Tree is grouped: ProjectGroup (by project id) -> ProjectFile -> Notebook. A single-notebook file is a leaf labelled with its notebook; legacy multi-notebook files stay collapsible. The init notebook is excluded from counts everywhere. - Refresh is grouping-safe: refreshNotebook evicts every sibling cache entry for a project id and all refreshes fire a full-tree change (no per-item fires). - Commands are project-scoped vs notebook-scoped; new/duplicate/add-notebook create sibling files via the factory (never appended), delete removes the file for a single-notebook file, and notebook names are unique within a project group. - Add a status bar item showing the active Deepnote notebook with a "Copy Active Deepnote Notebook Details" command. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01URccsVKXeNKZqqPi89L4ro
Chunk 6 of single-notebook migration (§8). - Key the server starter maps, the config handle, and the kernel auto-selector by notebook.uri.toString() - the same identity the kernel and controller use - so a notebook's server is 1:1 with its kernel. Sibling notebooks of one project no longer share a server; the working directory and SQL env are taken from each notebook's own file. - Fix environment deletion: stop every server using the environment (including closed notebooks whose server is still running) before removing the mappings, driven from the notebook->environment mapper. Drop the dead environmentServers map that was never populated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01URccsVKXeNKZqqPi89L4ro
…le reader Chunk 7a of single-notebook migration (§9). - Write snapshots with notebook-scoped filenames via @deepnote/convert (generateSnapshotFilename / parseSnapshotFilename), replacing the local slug and filename regex. - readSnapshot resolves snapshots path-free (it runs at deserialize, which has no URI): glob by project id, rank the notebook-scoped match first and keep legacy project-scoped snapshots as a fallback, and skip an empty-output "latest" (save race) or a corrupt file while walking candidates. Legacy snapshots are read, never migrated or deleted. - Defer the execution snapshot save until outputs settle (quiet window with a max wait) and cancel it on re-execute / close. - Use convert's computeSnapshotHash on the save path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01URccsVKXeNKZqqPi89L4ro
Chunk 7b of single-notebook migration (§10). - The init runner now subscribes to kernel start and restart events and runs the init notebook found in its own sibling .deepnote file (matched by project id + initNotebookId via isValidSiblingInitCandidate), instead of looking it up in the main file's notebooks. - Track "init has run" per kernel in a WeakSet<IKernel>: a fresh kernel runs init once, and an in-place restart (which fires onDidRestartKernel) re-runs it so the kernel is re-initialized before the next user cell. A missing sibling is logged and skipped without permanently marking the project. - Remove the manager's persistent init-run tracking and the selector's init staging; the runner owns init triggering. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01URccsVKXeNKZqqPi89L4ro
- void the fire-and-forget onExecutionComplete call (no-floating-promises), matching the existing void performSnapshotSave pattern. - Use American "behavior" in a comment. - Add test-only technical words (basenames, initmain, Résumé, unparseable) to cspell. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01URccsVKXeNKZqqPi89L4ro
…of duplicating it The mocha ESM loader wholesale-mocked @deepnote/convert and reimplemented its pure helpers (resolveSnapshotNotebookId, splitByNotebooks, isValidSiblingInitCandidate, snapshot filename generate/parse, hashing, etc.). That duplicated upstream logic with no drift detection: if convert changed, the mock silently kept the old behavior and tests stayed green against a fiction. - Remove the @deepnote/convert interception from build/mocha-esm-loader.js so unit tests exercise the real package's pure functions (and now track its actual API). - Mock only the one genuinely side-effecting export, convertIpynbFilesToDeepnoteFile (real node:fs I/O), via esmock in the explorer import suites where it is used. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01URccsVKXeNKZqqPi89L4ro
…r paths From the Codex review of the PR (F1/F3/F4/F5), all independently verified: - F1 (P1): snapshot save fetched the cached project with getAnyProjectEntry(projectId), which can return the wrong sibling when multiple single-notebook siblings of one project are open, silently skipping the snapshot write. Use the exact getOriginalProject(projectId, notebookId) lookup instead. - F3: collectNotebookNamesForProject globbed **/*.deepnote without skipping snapshot sidecars, so stale snapshot notebook names polluted the name-uniqueness set. Filter snapshot files (matching the tree provider and propagator). - F4: detectContentChanges compared notebooks[0]; for a legacy [init, main] file the edited notebook is not at index 0, so edits were missed and modifiedAt preserved. Match the notebook by id. - F5: the deferred-save timer fired performSnapshotSave as a floating promise; wrap the save body in try/catch/finally so a build/write failure is logged (not an unhandled rejection) and execution state is always cleared. Adds regression tests for F1 (exact lookup), F3 (snapshot exclusion), and F4 (match by id). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01URccsVKXeNKZqqPi89L4ro
…el init runs Addresses round-2 code-review findings G2 and G3 (both verified P2). - G2: deepnoteFileChangeWatcher's snapshot block-id recovery used the project-only getAnyProjectEntry(projectId), which can return a different open sibling's cached project (siblings share project.id), leaving originalBlocks undefined and silently skipping recovered outputs. Use the exact getOriginalProject(projectId, notebookId) — the same fix already applied to snapshotService (F1), here in the watcher path that was missed. - G3: moving init execution to the event-driven runner dropped the notebook-close cancellation that the kernel auto-selector used to provide, so closing a notebook mid-init left the remaining init blocks executing against a closed notebook. Tie the init run to a CancellationTokenSource cancelled on notebook close and dispose it in a finally. Adds regression tests for both (each fails on the pre-fix code). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01URccsVKXeNKZqqPi89L4ro
…ort/delete Removes two pieces of functionality; also folds in the branch's in-progress updates this work was layered on top of (they could not be isolated, as the removals are interleaved with and built on top of that WIP). Removed - project-metadata propagator: - Delete DeepnoteProjectMetadataPropagator and its types, drop the DI binding, and unwire it everywhere (activation, file-change watcher self-write hook, integration webview, explorer rename). Project-level fields are no longer fanned out across sibling files: each notebook owns its own integrations, and project-name drift is accepted for now. Drop the now-dead updateOriginalProject manager method and two stale comments. Removed - project-level explorer commands: - Delete the exportProject and deleteProject commands (constants, command-arg type, registrations, package.json command defs + sidebar menus, nls titles, and their unit tests). Per-notebook export remains via the existing exportNotebook command (first non-init notebook of the file). Also includes the branch's pending updates the above was built on: dependency bumps (incl. @deepnote/convert 4.0), the getOriginalProject -> getProjectForNotebook manager rename and getAnyProjectEntry removal, and assorted snapshot/serializer/kernel adjustments. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P2CLu8UmD8ceGNyv8u96pQ
Brings in #432 (Cloud SQL integration support). Resolved the package.json and package-lock.json conflicts by keeping this branch's newer @deepnote/* versions (blocks 4.6.0, convert 4.0.0, runtime-core 0.4.0); @deepnote/database-integrations is 1.5.0 on both sides, and the Cloud SQL source from #432 merged cleanly. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P2CLu8UmD8ceGNyv8u96pQ
@deepnote/blocks@4.6.0+ renders `text-cell-bullet` blocks with `indent_level >= 1` using leading spaces (two per level) before the bullet marker. stripMarkdown's bullet regex only matches at column 0, so the leading indentation must be trimmed first for the plain-text cell value to round-trip correctly. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P2CLu8UmD8ceGNyv8u96pQ
…es re-exports snapshotFiles.ts re-exported six snapshot-filename helpers from @deepnote/convert. Remove the re-export block and import the helpers directly from @deepnote/convert at each use site (snapshotService.ts and the snapshotFiles unit test). snapshotFiles.ts now keeps only its local helpers (SNAPSHOT_FILE_SUFFIX, isSnapshotFile, extractProjectIdFromSnapshotUri) plus the single internal use of parseSnapshotFilename. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P2CLu8UmD8ceGNyv8u96pQ
Refactor the buildSnapshotPath method to accept an object as an argument, improving readability and maintainability. Update all relevant calls to this method throughout the snapshotService and its unit tests to match the new signature. This change enhances the clarity of parameter usage and reduces the risk of errors when passing arguments.
Trim the single-notebook test suites by removing duplicate and tautological tests and collapsing/merging several others, shrinking the PR's test additions by ~640 lines with no loss of real coverage. Cuts target only tests this branch added: - exact-(projectId, notebookId)-lookup restatements duplicated across the watcher, serializer, snapshot, and manager suites - wrapper tests already covered by the delegate's own tests (addNotebookToProject, sibling-file allocation, project-id resolution) - tautologies over trivial template/getter functions (serverUtils) - framework-registration smoke tests (status bar) Merges keep the one meaningful assertion and drop the duplicate scaffolding (e.g. legacy-delete no-op folded into the existing delete test; two init builders parametrized into one). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P2CLu8UmD8ceGNyv8u96pQ
When splitting a legacy multi-notebook .deepnote file into single-notebook
siblings, rename the original to `<name>.deepnote.legacy` instead of moving it
to the OS trash. The `.legacy` suffix takes it out of the extension's view (it
no longer matches `*.deepnote`) while keeping it on disk next to the split
results, so the user can restore it by removing the suffix.
Unlike `workspace.fs.delete({ useTrash: true })`, this is deterministic and does
not depend on an OS trash backend (which can be absent on headless Linux).
Collisions bump the name to `.legacy-2`, `.legacy-3`, … and the rename still
happens only after every child is durably written (write-before-retire).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P2CLu8UmD8ceGNyv8u96pQ
Add an ExTester end-to-end test that drives the real VS Code UI through the on-open split of a legacy multi-notebook .deepnote file: it asserts the split prompt, the one-file-per-notebook result, the retained `.legacy` backup, that each sibling opens without re-prompting, and that content plus the project integration fan out into every split file. Add a `createScreenshotter(this)` helper that captures step screenshots into a per-spec directory derived from the running test file (`test/e2e/screenshots/<spec>/`), plus the `sales-analytics.deepnote` fixture. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P2CLu8UmD8ceGNyv8u96pQ
…ites Add ExTester end-to-end suites covering: - opening a plain single-notebook file (opens directly, no split prompt, the status bar shows the notebook name); - splitting a multi-notebook file that declares an init notebook (the init notebook becomes its own single-notebook sibling; each main sibling still references it via initNotebookId); - the init-notebook runner: the sibling init notebook runs hidden in a main notebook's kernel so its definitions are available, and re-runs after a kernel restart. Add the quick-notes and etl-pipeline fixtures (including the pre-split extract/init siblings), and disable the kernel-restart confirmation in the E2E settings so the restart test can drive it non-interactively. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P2CLu8UmD8ceGNyv8u96pQ
Add an ExTester end-to-end test asserting the Deepnote Explorer groups sibling
.deepnote files by project: three files sharing one project.id collapse into a
single "Marketing" group ("3 files") whose leaves are the three notebooks, while
a file from a different project appears as its own group. Reads the tree by
diffing visible leaves before/after expanding (avoids the page-object library's
flaky CustomTreeItem.getChildItems). Adds the three marketing fixtures.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P2CLu8UmD8ceGNyv8u96pQ
When several E2E suites run in one ExTester session (as in CI, via the `*.e2e.test.js` glob), every workspace-folder open after the first failed with "Failed to open folder after 5 attempts" in the suite's `before all` hook — only the alphabetically-first suite passed. Root cause: the simple "Open Folder" dialog (files.simpleDialog.enable) navigates one directory level *toward* the typed path per OK click and only accepts the folder once the browser is AT it. The helper clicked OK once then re-opened the dialog each attempt, which reset navigation back to the default directory — for the 2nd+ open that default is the previous, now-deleted workspace, so the dialog fell back to "/" and never converged on the target. Fix: click OK repeatedly within a single dialog until the pre-open workbench element detaches (reload = folder accepted), instead of re-opening per attempt; and set `window.openFoldersInNewWindow: "off"` so "Open Folder" reuses the current window, keeping that reload detectable. Verified with four suites (16 tests) opening four folders in one session. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P2CLu8UmD8ceGNyv8u96pQ
Add an ExTester end-to-end test for the Deepnote status-bar item: it shows the active notebook's name (with the "Copy Active Deepnote Notebook Details" tooltip), hides when a non-notebook editor is focused, and — on click — copies the notebook details to the clipboard with a confirmation toast. The clipboard is verified by pasting into a scratch text file and reading it back. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P2CLu8UmD8ceGNyv8u96pQ
Add an ExTester end-to-end test for the notebook-management commands that create and rename sibling .deepnote files from the Deepnote explorer: New Notebook, Add Notebook (project-group context menu), Duplicate Notebook, and Rename Notebook — each verified by the resulting notebook name inside the sibling files plus the confirmation toast. Delete Notebook is included as a pending test: its context-menu -> native confirmation-modal interaction is unreliable to drive under ExTester (documented inline), so it is left as a manual check. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P2CLu8UmD8ceGNyv8u96pQ
Add an ExTester end-to-end test for the Deepnote integrations UI: opening "Manage Integrations" for a notebook whose project declares an integration lists it (the "Sales BigQuery" integration on the sales-analytics-revenue fixture), while a plain notebook (quick-notes) shows no such integration. Adds the sales-analytics-revenue fixture (a single-notebook split of the Sales Analytics project carrying the BigQuery integration + its SQL cell). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P2CLu8UmD8ceGNyv8u96pQ
Add an ExTester end-to-end test for renaming a Deepnote project from the Explorer: none of the three "Marketing" siblings is opened, then the project group is renamed to "Growth" via its context menu; the new name is asserted to fan out to every sibling .deepnote file on disk (and the old name gone), with the Explorer group relabelled and a confirmation toast. Extract the shared Deepnote tree helpers (getDeepnoteExplorerSection, readDeepnoteTreeRows, findDeepnoteGroup/Leaf, selectDeepnoteContextMenu) into test/e2e/helpers/deepnoteTree.ts for reuse across the tree-driven suites. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P2CLu8UmD8ceGNyv8u96pQ
Add an ExTester end-to-end test for the edge case where a .deepnote file's only notebook is its init notebook. Opening bootstrap-only.deepnote renders that notebook as a fallback (status bar shows "Bootstrap"), does not raise the split prompt (it is a single-notebook file), and the Explorer shows it with "0 notebooks" (the init notebook is excluded from the count). Adds the bootstrap-only fixture. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P2CLu8UmD8ceGNyv8u96pQ
Add createDeepnoteFile/Project/Notebook/Block/WorkspaceFolder builders to deepnoteTestHelpers, each taking a Partial<> of the real type (no casts, no bespoke options type), and replace the per-file fixture builders that had drifted across the deepnote unit tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AGod1izfU9yA8ZQfxD5yeZ
Saves an open, dirty notebook before a disk read-modify-write so the file watcher's reload cannot clobber unsaved cell edits. Shared so the rename and integration-write paths can both guard their writes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AGod1izfU9yA8ZQfxD5yeZ
…t loss renameProject and renameNotebook read each file from disk and rewrote it without saving an open, dirty document first; the watcher then reloaded the stale disk content over the live cells and saved it, destroying unsaved edits. Flush (save) the open document first and abort if the save is declined, re-reading from disk so the rewrite preserves live content. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AGod1izfU9yA8ZQfxD5yeZ
updateProjectIntegrations only touched cached (open) siblings, so editing integrations from one notebook silently changed open siblings but left closed ones stale. Write the new integration list into every sibling .deepnote file on disk (open or closed), rewriting only project.integrations and flushing any open dirty sibling first so cells are not clobbered. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AGod1izfU9yA8ZQfxD5yeZ
…ately The split's cleanup only covered the child-write phase; failures during env migration, tab close, backup allocation, or rename left generated sibling files and partial env mappings behind, and a post-rename failure still told the user the original was "left unchanged". Track applied steps and unwind them in reverse on any failure, and derive the message from whether the rename happened and rollback fully restored. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AGod1izfU9yA8ZQfxD5yeZ
…le cap readSnapshot globbed all of a project's snapshots and capped the result at 200 before filtering for the requested notebook, so once siblings pushed a project past 200 files the notebook's own latest could be truncated away. Search the notebook-scoped snapshots first (capped on their own), then fall back to legacy project-wide files. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AGod1izfU9yA8ZQfxD5yeZ
The multi-notebook split prompt now states that bundling notebooks in one .deepnote file is a legacy layout being replaced by one file per notebook, instead of just describing the file. Sync the unit test's PROMPT_MESSAGE constant and the e2e suites' SPLIT_PROMPT regex (no longer keyed on the dropped word "contains") and content assertions to the new copy. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AGod1izfU9yA8ZQfxD5yeZ
…ding filenames
Two issues on the explorer's "Add Notebook" (and the sibling New/Duplicate
flows) for a project that already spans multiple notebooks/files:
- Filenames accumulated other notebooks' names. `buildSiblingNotebookFileUri`
derived the new file's stem from `group.files[0]`'s filename, which is itself
a split sibling (`{base}-{slug}`). A freshly added `{base}-{slug}.deepnote`
sorts before the source (`-` < `.`), becomes the next `files[0]`, and each add
appended another slug (`marketing-a-b-c.deepnote`). Now the stem comes from the
project name (`slugifyProjectName(project.name)`, matching the snapshot-filename
convention) and never reads a sibling filename, so it can't compound.
- Adding a notebook collapsed expanded groups. The add flows called the full
`refresh()`, which resets the initial-scan flag and tears the whole tree down
to a loading node before rebuilding. Switched them to the scoped
`refreshNotebook(projectId)` (already used by rename/delete), which re-scans and
picks up the new file without the teardown.
Tests: project-name-based naming and the scoped-refresh behavior are covered by
new DeepnoteExplorerView / factory unit tests; the notebookCommands E2E suite
asserts the exact base filenames.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AGod1izfU9yA8ZQfxD5yeZ
DeepnoteExplorerView built its own `new DeepnoteTreeDataProvider(logger)` in
the constructor, so a unit test asserting on the provider had to force-cast
`(explorerView as unknown as {...}).treeDataProvider` to reach the private field.
Pass the provider in through the constructor instead (built and handed over by
deepnoteActivationService.ts, which already instantiates the explorer manually).
The parameter is typed as the concrete DeepnoteTreeDataProvider, not the base
`TreeDataProvider<DeepnoteTreeItem>`: the explorer calls refresh/refreshNotebook/
refreshProject/findTreeItem and disposes the provider, none of which are on that
interface. The addNotebookToProject test now injects a ts-mockito
`mock<DeepnoteTreeDataProvider>()` and asserts with `verify(...)`, dropping the
force-cast; the other instantiation sites pass a real provider to preserve
behavior.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AGod1izfU9yA8ZQfxD5yeZ
Use "Filenames" instead of "Basenames" so the comment passes cspell. Co-authored-by: Cursor <cursoragent@cursor.com>
The project-rename e2e only exercised closed siblings, so the guard that saves open, dirty notebooks before the rewrite (otherwise the file-watcher reload clobbers the live cell edits) was unprotected. Extend the existing suite: open one sibling, type an unsaved marker into its code cell, then Rename Project, and assert the marker survived on disk alongside the new project name. Verified as a real regression guard — with the pre-rewrite flush removed, this new case fails (marker lost) while the existing three still pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AGod1izfU9yA8ZQfxD5yeZ
… files The `notebook` codicon renders as a clock face at tree size (verified against the codicon set in VS Code 1.111 — it's a real glyph, `\ebaf`, not a fallback), so every `.deepnote` file showed a clock in the Explorer. Replace it: - Single-notebook files now use `file-code`, matching the notebook children they represent — every "notebook you open" looks the same. - Legacy multi-notebook files use `book` (a container of notebooks), and their tooltip gains a "(legacy)" suffix to mark the retired one-file-many-notebooks layout. Icons rendered and confirmed in a headless VS Code 1.111 (folder / book / file-code, no clock); tree-item and provider unit-test icon and tooltip assertions updated accordingly. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AGod1izfU9yA8ZQfxD5yeZ
F1: gate the integrations env-file watcher on server existence (IDeepnoteServerStarter.isServerRunningForFile), not kernel.startedAtLeastOnce. The toolkit server captures env at spawn — before the first cell executes — so an env edit in the open→first-run window previously produced no restart prompt and the first cell ran with stale values. F2: dispose the notebook's kernel in restartServerForNotebook. areKernelConnectionsEqual compares only id/environmentName/notebookName for Deepnote kernels, so a same-env respawn (only baseUrl changes) looks "unchanged" and updateConnection does not dispose the kernel — leaving it bound to the killed server (reconnect spinner + "connection failed" on nearly every restart). Disposing it forces a fresh kernel against the new server and suppresses the spurious error. F3: run each notebook's restart atomically. The cancellable withProgress token was forwarded into both stopServer and startServer; cancelling after the process was killed rejected and stranded the notebook on a dead server. Cancellation is still honored between notebooks, but once a restart starts it runs to completion. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CPTs6CHNncauUuTGkwpNtH
Remove the promptSeq/supersede logic and the isRestarting guard from IntegrationsEnvFileWatcher. Restarting is a user action — a stale or missed prompt is simply re-triggered by the next save or a manual restart — so overlapping prompts/restarts don't warrant special handling. This also removes the source of the cross-project supersession and change-during-restart edge cases. The server-existence gate, atomic restart, debounce, and per-.deepnote dedup are unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CPTs6CHNncauUuTGkwpNtH
Bring the feature branch up to date with the base (12 commits: explorer/tree refactors, per-sibling integration persistence, document-flush guard, snapshot scoping, and e2e coverage). No conflicts — disjoint file sets. Merged tree compiles and the full unit suite passes (2512 passing, 0 failing). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CPTs6CHNncauUuTGkwpNtH
…nToken mock Condense the integrations injection E2E's header to two lines and the folder-open note to one (the isolation rationale stays in the const comments). Replace the `as unknown as CancellationToken` force casts in the watcher unit test with a fully-typed `mock<CancellationToken>()` helper. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CPTs6CHNncauUuTGkwpNtH
Extension-side infrastructure for cloud-style live integration-env injection
(no server restart):
- IntegrationsEnvVarsEndpoint: a loopback HTTP server (127.0.0.1) serving
GET /userpod-api/:projectId/integrations/environment-variables, returning a
project's integration env vars as [{name,value}] from
SqlIntegrationEnvironmentVariablesProvider — what the toolkit's
set_integration_env() fetches. Localhost-only, no auth.
- IntegrationEnvLiveRefresher: runs `import deepnote_toolkit;
deepnote_toolkit.set_integration_env()` silently (executeHidden) in each
affected running kernel and shows one dismissible notification, only when the
snippet ran without error outputs.
Registered node-only. Not yet wired to the server-starter (B2) or the
watcher / restart handler (B3).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CPTs6CHNncauUuTGkwpNtH
At server spawn, when the loopback endpoint is listening and the .deepnote file resolves to a project id, set the toolkit config env so it fetches integration env from our endpoint in "direct mode": DEEPNOTE_RUNTIME__ENV_INTEGRATION_ENABLED / __RUNNING_IN_DETACHED_MODE / __WEBAPP_URL, plus legacy DEEPNOTE_PROJECT_ID (maps to runtime.project_id and satisfies set_notebook_path's has_env check). Detached mode avoids dev_mode's /work redirect. Guarded all-or-nothing on baseUrl + project id, so when the endpoint isn't ready the feature stays off and the pre-existing spawn-time SQL_* injection remains the env source. That injection is unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CPTs6CHNncauUuTGkwpNtH
Replace the toolkit-server restart on integration/env-file change with a live in-kernel refresh via IIntegrationEnvLiveRefresher (runs set_integration_env(), which re-fetches from the local endpoint and updates os.environ without a restart), plus a dismissible "environment updated" notification. - IntegrationsEnvFileWatcher: on a debounced .deepnote.env.yaml/.env change, live-refresh the affected notebooks' kernels (dir-then-root; no prompt). - IntegrationEnvRefreshHandler (renamed from IntegrationKernelRestartHandler, which no longer restarts): on a SecretStorage integration change, refresh open Deepnote notebooks. Node-only now (the live-refresh mechanism has no web equivalent), so it is removed from the web service registry. Delete the restart machinery this obsoletes: restartServerForNotebook (IDeepnoteKernelAutoSelector) and isServerRunningForFile (IDeepnoteServerStarter), including their impls and tests. Full unit suite: 2521 passing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CPTs6CHNncauUuTGkwpNtH
…add live-refresh e2e Verifying the live set_integration_env() loop against the real pinned toolkit (2.1.1) surfaced a crash: its get_project_auth_headers() dereferences project_secret with no null-check, so detached mode fails unless a secret is set. Inject DEEPNOTE_RUNTIME__PROJECT_SECRET (a generated token), and — since the loopback endpoint serves integration credentials — have the endpoint validate that bearer token (401 otherwise). Extend integrationsEnvFileInjection.e2e.test.ts to prove the no-restart live refresh: after the initial run, rewrite .env and assert a re-run of the SAME kernel reads the new value (applied via the watcher's set_integration_env()). Verified end-to-end against toolkit 2.1.1: direct-mode URL matches the endpoint route, initial fetch+inject, live refresh on re-fetch, and unset-on-remove all pass. Full unit suite: 2522 passing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CPTs6CHNncauUuTGkwpNtH
Remove comments that restate readable code; keep only the non-obvious "why" as single lines — the toolkit 2.1.1 project_secret quirk + detached-mode guard, the endpoint's bearer-token validation, the `_dntk`-not-guaranteed import, and the "public for testing" note. Comments-only change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CPTs6CHNncauUuTGkwpNtH
…ndpoints The loopback server hosts the toolkit's userpod-api surface, of which the integration environment-variables route is currently the only endpoint. Rename it to a host-oriented name so future userpod-api endpoints have a home, and drop the http.Server force-cast in stop() now that @types/node declares closeAllConnections() directly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CPTs6CHNncauUuTGkwpNtH
Resolve six verified findings from the multi-reviewer pass on PR #440: - F2: attach a persistent 'error' listener to the userpod-api server so a post-listen error is no longer an unhandled EventEmitter throw; on failure, prompt the user to Restart the service or Reload Window. - F4: drop the redundant `as AddressInfo` assertion (the typeof guard narrows it) and the now-unused import. - F5: compare the bearer token with crypto.timingSafeEqual (length-guarded) instead of a short-circuiting !==. - F6: skip notebooks whose `deepnote.integrations.envFile.enabled` is false in the env-file watcher, mirroring the config provider's gate. - F9: refresh kernels in parallel via Promise.all so one hung kernel can't starve the others. - F14: extract loadFileConfigs() and mergeIntegrationConfigs() from the ~150-line getEnvironmentVariables() (behavior unchanged). Adds targeted unit tests for the F2 recovery path, the F5 same-length-token case, the F6 enabled/disabled gate, and F9 parallelism. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CPTs6CHNncauUuTGkwpNtH
Brings in the squash-merged PR #429 ("one notebook per .deepnote file"). This branch forked from an earlier tk/single-notebook, so #429's later work surfaced as conflicts in shared files. Resolved by taking #429's newer versions of the shared single-notebook code (persistProjectIntegrations refactor with activeFileUri, snapshot candidate ranking, abortable tab-close) while preserving this PR's env-file integration additions and unioning divergent imports. Compiles clean; full unit suite green (2543 passing). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CPTs6CHNncauUuTGkwpNtH
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #440 +/- ##
===========================
===========================
🚀 New features to boost your workflow:
|
- Move the IUserpodApiEndpoints symbol + interface to the platform types layer so src/kernels (deepnoteServerStarter) can import it without crossing the kernels->notebooks import boundary, matching how the sibling ISqlIntegrationEnvVarsProvider is placed. (import/no-restricted-paths) - Add a file-level eslint-disable for local-rules/dont-use-process to the integrationsFileConfigProvider .node.unit.test, which legitimately exercises a process.env override of a .env value (same convention as other .node.unit.test files). - Add userpod, Userpod, dedup, unstarted to the cspell dictionary. Compiles clean; lint, spell-check, and format all pass; full unit suite green (2543 passing). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CPTs6CHNncauUuTGkwpNtH
…them Drop the cspell entries for "dedup" and "unstarted" and reword the two usages instead: "no dedup" -> "no deduplication" (a word cspell already accepts) and "an unstarted kernel" -> "a kernel that has not started". cspell keeps only userpod/Userpod. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CPTs6CHNncauUuTGkwpNtH
…int-disabling the test Replace the file-level eslint-disable (local-rules/dont-use-process) with a small overridable seam: the provider reads the environment through a protected getProcessEnvironment(), so the test supplies a controlled env via a subclass and never touches the real process.env. This removes the eslint-disable plus two test smells — mutating the global process.env and cleaning it up in a try/finally — and makes every test in the file hermetic (no leakage from the real CI environment). Compiles clean; lint/spell/format pass; full unit suite green (2543 passing). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CPTs6CHNncauUuTGkwpNtH
…egrations
- F1: per-project endpoint token + filter/merge notebooks that share a project
id, so a kernel can't read another (or a sibling's) project credentials
- F2: only live-refresh when a .deepnote.env.yaml exists, and use a transient
status-bar message instead of a toast, so unrelated .env saves are silent
- F3: await endpoint readiness before injecting integration env; warn on skip
- F6: surface .deepnote.env.yaml validation issues as diagnostics
- F9: extract the shared isFederatedAuthMetadata guard into integrationTypes
- F10: drop the unreachable DATAFRAME_SQL_INTEGRATION_ID condition + unused import
- F13: share merged (SecretStorage + file) configs with the integration
detector, SQL cell status bar, and SQL LSP so UI/LSP agree with execution
- F16: bounded retry instead of a fixed 5s sleep in the live-refresh e2e
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CPTs6CHNncauUuTGkwpNtH
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
@coderabbitai ignore
Summary
Adds CLI-parity
.deepnote.env.yaml+.envintegration loading to the extension as a complementary source alongside VSCode SecretStorage, and applies integration env to running kernels the same way Deepnote cloud does — via the toolkit's liveset_integration_env(), with no server restart.What it does
1. File-based integration config (loader + merge)
IntegrationsFileConfigProvider: reads a.deepnote.env.yaml(dir-then-root), resolvesenv:refs against.env(dotenv) andprocess.env(real env wins), never throws. Reuses@deepnote/database-integrations'parseIntegrations; replicates only the Node fs/dotenv shell.SqlIntegrationEnvironmentVariablesProvidermerges file configs over SecretStorage (file wins on id conflict; file-only additive; federated skip + DuckDB unchanged).dotenvdependency and thedeepnote.integrations.envFile.enabledsetting (default true).2. Live env injection (cloud-parity, no restart)
Rather than restarting the toolkit server when integration env changes, the extension mirrors the cloud "no restart" path:
IntegrationsEnvVarsEndpoint: a loopback HTTP endpoint (127.0.0.1, no auth) servingGET /userpod-api/:projectId/integrations/environment-variables→[{name,value}]from the provider. This is exactly what the toolkit'sset_integration_env()fetches.DEEPNOTE_RUNTIME__ENV_INTEGRATION_ENABLED/__RUNNING_IN_DETACHED_MODE/__WEBAPP_URL+DEEPNOTE_PROJECT_ID— so it fetches integration env at kernel start. Guarded to degrade to the existing spawn-time injection when the endpoint/project id isn't available.IntegrationEnvLiveRefresher: on an env-file change (IntegrationsEnvFileWatcher, debounced) or a SecretStorage integration change (IntegrationEnvRefreshHandler), runsdeepnote_toolkit.set_integration_env()silently (executeHidden) in the affected running kernels — re-fetch + liveos.environupdate — and shows one dismissible "environment updated" notification.The spawn-time SQL_* injection is retained as the initial-load safety net. Node-only feature; web is unchanged.
Test plan
integrationsEnvFileInjection.e2e.test.tsproves the.deepnote.env.yaml→env:→ dotenv → integration → kernel path via spawn-time injection. Runs under ExTester in CI.set_integration_env()loop — the toolkit fetching the local endpoint and updating a running kernel'sos.environon change.Follow-up (separate
deepnote/deepnoterepo)@deepnote/database-integrationscould add a Node-only subpath export (/node) for the fs/dotenv shell and aDatabaseIntegrationConfig-typed federated guard, to remove the small pieces the extension/CLI currently replicate (plan drafted separately).🤖 Generated with Claude Code
https://claude.ai/code/session_01CPTs6CHNncauUuTGkwpNtH