Skip to content

jinja: unary minus before a variable fails to parse #51

Description

@am17an

Upstream: ggml-org#29233

Dynamic negative slice bounds such as items[:-n] fail to parse, while negative integer literals work.

Reproduced on spark-a934.local, Linux aarch64, upstream 58367713a6935c0810103378144008df32e3d5db, CPU Release llama-common. No model required.

#include "jinja/lexer.h"
#include "jinja/parser.h"
#include <iostream>
int main() {
    for (const char * s : {"{{items[:-n]|string}}", "{{items[:-1]|string}}"}) {
        try {
            jinja::lexer lexer;
            auto tokens = lexer.tokenize(s);
            auto program = jinja::parse_from_tokens(tokens);
            std::cout << "parsed\n";
        } catch (const std::exception & e) {
            std::cout << e.what() << '\n';
        }
    }
}

From a checkout with llama-common built in build/bin:

g++ -std=c++17 -Icommon repro.cpp -Lbuild/bin -Wl,-rpath,"$PWD/build/bin" -lllama-common -o repro
./repro

Observed: Unexpected token: - of type 24 for the variable-bound template; parsed for the literal control. Failure is at parse time, before variable evaluation. Introducing commit not bisected.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions