Expose structured observation for async jobs

[abrady]

Summary

• accept an optional per-job reportFile for inline, file-backed, registered, and batched async scripts
• ingest JSONL progress, log, and output events without requiring Daz's main thread on Studio 6
• expose the final observation on Studio 4 after main-thread execution returns, without constructing QScriptEngine on an HTTP worker
• expose a bounded observation record through request status, result, and list responses
• add matching report_file= support to the synchronous and asynchronous dazpy clients
• prevent path aliases from letting a report file overwrite its submitted script file

Dependency chain

• Rebased onto current master / dazpy 2.9.2, including dazpy script-call batching + find_skeleton race fix #39's script-call batching and find_skeleton() retry.
• Built on Add async script-file jobs #36, which is already merged and supplies asynchronous script-file jobs.
• This is the first of two prerequisites for Make Daz MCP compact and dazpy-backed daz-mcp-server#9.
• Unify dazpy protocol operations #38 is stacked on this PR and supplies the unified dazpy protocol surface required by MCP [Refactoring] Phase 7: Settings Management System #9.
• MCP [Refactoring] Phase 7: Settings Management System #9 should not be accepted against a nightly until both Expose structured observation for async jobs #37 and Unify dazpy protocol operations #38 are present.
• The current nightly-70 tag points at the dazpy script-call batching + find_skeleton race fix #39 merge and does not contain this change.

Validation

• 738 passed, 30 subtests passed across tests/test_dazpy.py and tests/test_dazpy_aio.py
• exact CI unit command passed: 734 tests
• two CI source ratchets prevent worker endpoints from bypassing the SDK guard and require final ingestion before the main-thread terminal-state guard
• Studio 6 Release plugin compiled successfully with INSTALL_TO_DAZ=OFF; no DLL was installed and Daz was not restarted
• git diff --check and Python compileall passed

An SDK4 SDK is not present in the local build environment. The SDK4 boundary is enforced by the compile-time branch, a debug main-thread assertion around the QtScript parser path, the source ratchets above, and the documented Studio 4 final-only observation contract.

Live acceptance of the rebased DLL is intentionally deferred until the whole dependency stack is available in a published nightly.

[ghirpara]

Code review

Found 1 issue:

1. ingestReportLocked() is called from HTTP worker threads and, on SDK4 (Qt 4.8) builds, transitively constructs a QScriptEngine off the main thread, violating this codebase's own documented threading constraint

include/JsonStd.h documents that the SDK4 JsonStd::parseObject() overload constructs a QScriptEngine (a QObject) and states: "this must only be called on the Qt main thread." CLAUDE.md reiterates this: "Studio 4 must keep submission on the main thread because JsonStd::parseObject() uses QScriptEngine there... Neither path may touch GUI objects, DzScript, the DAZ API, or shared mutable state from a worker."

The new AsyncRequestManager::ingestReportLocked() calls JsonStd::parseObject() with no #if DAZ_SDK_MAJOR_VERSION guard:

daz-script-server/src/AsyncRequestManager.cpp

Lines 248 to 256 in 7b5c3f5

	if (line.isEmpty()) continue;
	QVariantMap event;
	std::string parseError;
	if (JsonStd::parseObject(line, event, parseError)) {
	applyReportEventLocked(req, event);
	} else {
	QVariantMap warning;
	warning.insert("level", "warning");

It's invoked unconditionally from getStatusJson(), getResultJson(), and listJson():

daz-script-server/src/AsyncRequestManager.cpp

Lines 318 to 324 in 7b5c3f5

	QMutexLocker locker(&m_mutex);
	if (!m_requests.contains(qid))
	return {404, "{\"success\":false,\"error\":\"Request not found\"}"};
	AsyncRequest& req = m_requests[qid];
	ingestReportLocked(req);
	std::string status = statusToString(req.status);

Those methods are called directly from the HTTP handlers with no Qt::BlockingQueuedConnection main-thread hop:

daz-script-server/src/RequestHandlers.cpp

Lines 298 to 303 in 7b5c3f5

	AsyncStatusHandler::AsyncStatusHandler(DzScriptServerPane* pane) : m_pPane(pane) {}
	void AsyncStatusHandler::handle(HttpContext& ctx)
	{
	std::pair<int, std::string> result = m_pPane->getAsyncStatusJson(ctx.urlMatch);
	ctx.respond(result.first, result.second);

So on SDK4 builds, any client polling /requests/:id/status, /requests/:id/result, or GET /requests for a request with reportFile set constructs a QScriptEngine on a raw httplib worker thread — the exact class of hazard flagged as a blocker on PR #36. Suggest gating report ingestion to the main thread (or to SDK6-only) the same way the async enqueue handlers already split on DAZ_SDK_MAJOR_VERSION.

🤖 Generated with Claude Code

_{- If this code review was useful, please react with 👍. Otherwise, react with 👎.}

[ghirpara]

Looks like there were a few issues that we probably need to look at before merging, but the one posted in the review seems to be the one that is the main blocker for this PR, since it looks like this reintroduces something we saw in #36 (Qt work is ending up on worker threads, this time through obs ingestion instead of enqueuing).

[abrady]

Thanks — agreed, this was a real SDK4 blocker. Fixed in 5ca387f.

• Worker-facing status/result/list paths now call an SDK-gated helper: Studio 6 retains live QJsonDocument ingestion, while Studio 4 deliberately performs no report parsing on raw httplib workers.
• markCompleted() performs final ingestion on the main thread for every SDK, before the existing terminal-state guard, so a late return after cancellation or stale-job failure can still publish its final observation safely.
• The Studio 4 contract is now explicit in the README, OpenAPI description, changelog, and threading architecture: final observation after execution, no report-driven live progress while the main thread is occupied.
• A debug assertion protects the QtScript parser path, and two CI source ratchets prevent worker methods from bypassing the SDK guard.

Validation: 738 client tests + 30 subtests, the exact CI unit command (734 tests), Python compileall, git diff --check, and a Studio 6 Release compile with INSTALL_TO_DAZ=OFF. There is no SDK4 SDK in this local build environment, so I have called that out explicitly in the PR validation rather than claiming an SDK4 compile.

You mentioned a few issues, with this as the main blocker. I only see this one detailed publicly; if there are other unpublished findings, please send them over and I'll address them before merge.

[ghirpara]

@abrady The next reported issue was this:

" AsyncRequestManager uses a single QMutex m_mutex to guard its whole shared state — the map of tracked requests, the pending queue, everything. Every operation that touches async request state takes this same lock: submit(),
dequeueNext(), markRunning(), markCompleted(), cancelJson(), and now getStatusJson(), getResultJson(), listJson(). Because it's one mutex, only one of these can run at a time, across every HTTP worker thread and the main
thread's own async processing loop."

Took a look and it potentially could cause issues someone uses the mutex under heavy load. I think the reason it gets flagged is because the ingest report is doing chunked disk I/O which is going to be slower than other memory/cpu bound things that use the lock.

I'd be okay with deferring fixing it while I investigate it more and can address it in a separate PR. I'll be out of pocket for the next couple of weeks so I want to try and get the tightly scoped and significant issues addressed before I go.