Welcome to NGRAMS Discussions

[mtrenkmann]

Hi there 👋

I'm pleased to announce the launch of NGRAMS, a project I've been working on for a long time.
Here on GitHub Discussions you can connect and have conversations about the service and related topics.

Feel free to

• Ask questions,
• Request features,
• Share ideas,
• Give general feedback.

[hebozhe]

Will there be an API?

[mtrenkmann]

👉 https://docs.ngrams.dev

[hebozhe]

Example I slapped together using Python:

"""
Take initial ngrams from already retained sentences shorter than the ngram limit, 
with their punctuations, and provide further extensions to generate 2-to-4-grams.
"""
import json
from os import getcwd
import re
import sys
from typing import TypedDict
from requests import Session, Response
from subprocess import Popen


cwd: str = getcwd()
sys.path.append(cwd)
from torbrowsing.torbrowsing import create_tor_process, create_tor_rq_session
from text_correction.text_correction import LangCode


TSV_PATH: str = f"{cwd}/data/ngdev/tsv"
tor_proc: Popen[bytes] = create_tor_process()
tor_sess: Session = create_tor_rq_session(with_proc=tor_proc)


class NgramsDevJSONTokens(TypedDict):
    text: str
    type: str
    inserted: bool


class NgramsDevJSONNgrams(TypedDict):
    id: str
    absTotalMatchCount: int
    relTotalMatchCount: float
    tokens: list[NgramsDevJSONTokens]


class NgramsDevJSON(TypedDict):
    queryTokens: list[dict[str, str]]
    ngrams: list[NgramsDevJSONNgrams]
    nextPageToken: str
    nextPageLink: str


def get_qry(qry: str, lang: LangCode, count: int) -> tuple[str, ...]:
    global tor_proc, tor_sess
    lang_corp: dict[LangCode, str] = {
        "en": "eng",
        "de": "ger",
        "fr": "fre",
        "es": "spa",
        "ru": "rus",
        "zh": "chi_sim",
    }
    search_params: dict[str, str] = {
        "query": qry,
        "flags": "cses",
    }
    ngd_url: str = f"https://api.ngrams.dev/{lang_corp[lang]}/search"
    try:
        res_tup: tuple[str, ...] = tuple()
        while ngd_url:
            ngd_resp: Response = tor_sess.get(url=ngd_url, params=search_params)
            ngd_json: NgramsDevJSON = json.loads(ngd_resp.text)
            # print(ngd_json)
            for dex in range(len(ngd_json.get("ngrams", []))):
                ngd_ngrams: NgramsDevJSONNgrams = ngd_json["ngrams"][dex]
                ngd_tokens: list[NgramsDevJSONTokens] = ngd_ngrams["tokens"]
                toks_list: list[str] = [
                    tdv
                    for td in ngd_tokens
                    for tdk, tdv in td.items()
                    if tdk == "text" and isinstance(tdv, str)
                ]
                res_tup += (" ".join(toks_list),)
            ngd_url = ngd_json.get("nextPageLink", "")
            # print(res_tup, len(res_tup))
            if len(res_tup) >= count:
                return res_tup
        return res_tup
    except ConnectionError:
        tor_proc.kill()
        tor_proc = create_tor_process()
        tor_sess = create_tor_rq_session(with_proc=tor_proc)
        return get_qry(qry, lang, count)


if __name__ == "__main__":
    print(get_qry(qry="No ** .", lang="en", count=1000))
    tor_proc.kill()

The ...JSON... classes are there to keep mypy/Pylance happy. Obviously, this snippet of code is missing a lot of pieces, such as the code I use to write results to TSV files in the event of future lookups, to reduce future server pings.

[gabiUfsc]

what about brazilian portuguese, is there any project?

[mtrenkmann]

I don't know of any project for Portuguese ngrams.

NGRAMS indexes part of the Google Books Ngram Dataset that does not contain Portuguese language.

[gabiUfsc]

Great, something to work on. Tks Gabriela Graudenz, (48)991279336 - (48) 996860705 Em ter., 12 de mai. de 2026 10:22, Martin Trenkmann < ***@***.***> escreveu:

…

I don't know of any project for Portuguese ngrams. NGRAMS indexes part of the Google Books Ngram Dataset <https://storage.googleapis.com/books/ngrams/books/datasetsv3.html> that does *not* contain Portuguese language. — Reply to this email directly, view it on GitHub <#1 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/B4KKEFTN346NE3TUWJ4WHY342MQS3AVCNFSM6AAAAACY2V7N6WVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTMOBZGI4TGNQ> . You are receiving this because you commented.Message ID: ***@***.***>