Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -198,7 +198,8 @@ jobs in FIFO order.
**GET /job?id=N** - Poll job status. **GET /job?id=N&result=1** fetches the
result as multipart/mixed, one pair per track: a JSON replay request part
(the request carrying the semantic stream, the score and the seeds of the
track) then the audio part (MP3 or WAV, selected by `output_format` in the
track, plus `"truncated": true` when a stage hit its budget before its end
token) then the audio part (MP3 or WAV, selected by `output_format` in the
request). `lm_batch_size` songs times `synth_batch_size` noise variations
come out song-major.
**POST /job?id=N&cancel=1** cancels a running job.
Expand Down
11 changes: 7 additions & 4 deletions docs/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -172,9 +172,11 @@ free guidance, grown on demand; `--max-seq` and `--max-batch` are the
levers that trade context and batch for VRAM. Measured peak above idle
on a 65 s song in Q8_0 under STRICT: 5.8 GB at the full context, 3.8 GB
at `--max-seq 8192`, the compute buffers of the three graphs making the
difference with the weights plus the cache, the VAE tiles first. A cache smaller than the
default budgets needs a `duration` on the request, the generator refuses
a prefix plus budget that would not fit.
difference with the weights plus the cache, the VAE tiles first. On a cache
smaller than the default budgets the generator shrinks a stage's budget to
the room the prefix leaves (`[AR] Frame budget clamped to N by the KV
cache`), the stream ends there marked truncated; it refuses only when that
room is below the stage's `min_tokens`.

The cost of STRICT is one reload of each half per song from the page
cache, about a second on the pod, `--keep-loaded` removes it.
Expand Down Expand Up @@ -838,7 +840,8 @@ GET /job?id=N Poll job status
GET /job?id=N&result=1 Fetch job result
multipart/mixed, boundary yue2-batch-boundary: per track, song-major,
one application/json replay request part (the request carrying the
semantic stream, the score and the seeds of that track), one
semantic stream, the score and the seeds of that track, plus
"truncated": true when a stage hit its budget before its end token), one
audio/mpeg or audio/wav part and one application/octet-stream part with
the latents the audio was decoded from (raw f32 [T, 64], the .vae
layout); for a VAE job, application/octet-stream latents from an encode
Expand Down
21 changes: 16 additions & 5 deletions src/generate.h
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@
#include "sampling.h"
#include "timer.h"

#include <algorithm>
#include <cstdio>
#include <cstring>
#include <vector>
Expand Down Expand Up @@ -61,7 +62,7 @@ static bool yue2_generate(Qwen3LM * lm,
const std::vector<std::vector<int>> & prefixes,
const std::vector<std::vector<int>> & negatives,
float cfg_scale,
const Yue2Sampling & s,
const Yue2Sampling & requested,
int64_t seed,
Yue2Phase phase,
std::vector<Yue2Generation> * out,
Expand All @@ -74,14 +75,24 @@ static bool yue2_generate(Qwen3LM * lm,
fprintf(stderr, "[AR] FATAL: guidance %.3f needs an unconditional prefix per sequence\n", (double) cfg_scale);
return false;
}
// Every set holds its prefix, the budget and the end token
// Every set holds its prefix, the budget and the end token. A budget the
// context cannot hold shrinks to the room left, and the stream ends
// truncated there; only a room below the stage's minimum stops it.
Yue2Sampling s = requested;
for (int i = 0; i < B; i++) {
size_t longest = guided && negatives[i].size() > prefixes[i].size() ? negatives[i].size() : prefixes[i].size();
if ((int) longest + s.max_tokens + 1 > context) {
fprintf(stderr, "[AR] FATAL: prefix %zu + budget %d + end exceeds context %d\n", longest, s.max_tokens,
context);
int room = context - (int) longest - 1;
if (room >= s.max_tokens) {
continue;
}
if (room < std::max(1, s.min_tokens)) {
fprintf(stderr, "[AR] FATAL: prefix %zu leaves %d of context %d, the stage needs at least %d\n", longest,
room > 0 ? room : 0, context, std::max(1, s.min_tokens));
return false;
}
fprintf(stderr, "[AR] %s budget clamped to %d by the KV cache (prompt %zu tokens)\n",
phase == YUE2_PHASE_ABC ? "Score" : "Frame", room, longest);
s.max_tokens = room;
}

const int N = guided ? 2 * B : B;
Expand Down
22 changes: 21 additions & 1 deletion tools/yue-server.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -452,6 +452,26 @@ static std::string json_string(const char * key, const std::string & value) {
return out;
}

// The JSON part of a track: its replay request, and "truncated" when a stage
// hit its budget before the end token. A resubmit ignores the extra key.
static std::string track_json(const Yue2Request & replay, bool truncated) {
std::string json = request_to_json(&replay);
if (!truncated) {
return json;
}
yyjson_doc * read = yyjson_read(json.c_str(), json.size(), 0);
yyjson_mut_doc * doc = yyjson_doc_mut_copy(read, NULL);
yyjson_doc_free(read);
yyjson_mut_obj_add_bool(doc, yyjson_mut_doc_get_root(doc), "truncated", true);
char * text = yyjson_mut_write(doc, 0, NULL);
std::string out = text ? text : json;
if (text) {
free(text);
}
yyjson_mut_doc_free(doc);
return out;
}

static void handle_props(const httplib::Request &, httplib::Response & res) {
Yue2Request d;
request_init(&d);
Expand Down Expand Up @@ -686,7 +706,7 @@ static void run_job(std::shared_ptr<Job> job, Yue2Request request) {
}
Yue2Request replay = request_replay(request, song.score.empty() ? request.abc : song.score,
pipeline_format_tokens(song.tokens), (int) t / M, (int) t % M);
request_parts.push_back(request_to_json(&replay));
request_parts.push_back(track_json(replay, song.truncated));
latents.push_back(std::move(song.latents));
}

Expand Down