AI and data stack advisories

Severe, 6 weeks2431Projects328

2431 severe, 6 weeks · 328 projects

NLTKGHSA-q4c8-9gwf-255x

NLTK: resource exhaustion

NLTK

CVE-2026-129231 · Published Nov 19, 2026 · updated Nov 20, 2026

Medium5.3
Fix: upgrade to 3.10.4 or later
GitHub advisory

Summary

SeekableUnicodeStreamReader.readline in nltk/data.py assembles a line by repeatedly reading a block, appending it to a growing chars buffer, and recomputing chars.splitlines(True) over the entire buffer on every loop iteration. The read block size doubles only until it caps at 8000 characters, so for a single unterminated line of N characters the loop runs about N/8000 iterations and iteration k re-scans roughly 8000*k characters, giving O(N^2) total work. The default size=None used on the line-oriented corpus-reader read path means the early break never fires before end of file. An attacker who supplies a corpus file containing one very long line with no newline can force quadratic CPU consumption in the reader's process. This is the same growing-buffer re-scan algorithmic-complexity class the project already treats as advisory-worthy in sibling corpus readers.

Root Cause

The readline loop violates the expectation that reading a line costs time linear in the line length. On each pass it executes chars += new_chars; lines = chars.splitlines(True), scanning the whole accumulated buffer again rather than only the newly read block. Because readsize is capped at 8000 (nltk/data.py:1997-1998), the number of iterations grows linearly with the line length while each iteration's scan cost also grows linearly, producing quadratic total cost. The loop...

Affected versions

PackageAffectedFixed in
nltk
PyPI
< 3.10.43.10.4
Details and references

## Summary `SeekableUnicodeStreamReader.readline` in `nltk/data.py` assembles a line by repeatedly reading a block, appending it to a growing `chars` buffer, and recomputing `chars.splitlines(True)` over the entire buffer on every loop iteration. The read block size doubles only until it caps at 8000 characters, so for a single unterminated line of N characters the loop runs about N/8000 iterations and iteration k re-scans roughly 8000*k characters, giving O(N^2) total work. The default `size=None` used on the line-oriented corpus-reader read path means the early break never fires before end of file. An attacker who supplies a corpus file containing one very long line with no newline can force quadratic CPU consumption in the reader's process. This is the same growing-buffer re-scan algorithmic-complexity class the project already treats as advisory-worthy in sibling corpus readers. ## Root Cause The `readline` loop violates the expectation that reading a line costs time linear in the line length. On each pass it executes `chars += new_chars; lines = chars.splitlines(True)`, scanning the whole accumulated buffer again rather than only the newly read block. Because `readsize` is capped at 8000 (`nltk/data.py:1997-1998`), the number of iterations grows linearly with the line length while each iteration's scan cost also grows linearly, producing quadratic total cost. The loop exits only on end of file or when `size is not None`; since the line-oriented block readers call `stream.readline()` with no argument, `size` defaults to `None` (`nltk/data.py:1942`) and that early break is unreachable on this path. There is no length cap on the assembled line and no wall-clock backstop, because no regular expression is involved on this path and the `nltk.redos` timeout therefore does not apply. ```text Affected function: nltk.data.SeekableUnicodeStreamReader.readline Loop with per-iteration full-buffer rescan: nltk/data.py:1968-2000 chars += new_chars; lines = chars.splitlines(True) (nltk/data.py:1977-1978) readsize *= 2 capped at 8000 (nltk/data.py:1997-1998) Default size=None on the corpus-view path: nltk/data.py:1942 Reachable via read_line_block/read_whitespace_block/read_wordpunct_block/ read_blankline_block in nltk/corpus/reader/util.py, through _open's SeekableUnicodeStreamReader construction. ``` ## Impact An attacker who controls the bytes of a corpus file processed by a line-oriented NLTK corpus reader (encoding set, which is the common case) can craft one arbitrarily long unterminated line. Reading that line pins a single CPU core for time quadratic in the line length. The effect is recoverable CPU-availability degradation in the processing worker, not a crash: memory use remains linear and the operation completes once the buffer is exhausted. Because the constant factor is bounded by the 8000-character read cap, a meaningful denial of service requires a large crafted file, which keeps the severity in the low to medium range. The worst credible deployment is an automated document-ingestion service that feeds attacker-supplied uploads into such a reader with no human step; a more typical case where a victim manually loads an attacker-supplied corpus requires user interaction and is lower. ## Proof of Concept `Requires only the nltk package under test on PYTHONPATH; no network access. Feeds one unterminated line of increasing length through SeekableUnicodeStreamReader.readline (size=None, the corpus-view path) and times each call.` ```python import io, time from nltk.data import SeekableUnicodeStreamReader def measure(n): stream = io.BytesIO(b"a" * n) # one unterminated line, no newline reader = SeekableUnicodeStreamReader(stream, "utf-8") t0 = time.perf_counter() line = reader.readline() # size=None -> corpus-view path dt = time.perf_counter() - t0 assert len(line) == n return dt prev = None for n in (1_000_000, 2_000_000,

CVSS 3.1
CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:L
Severity from
GitHub (reviewed advisory)
Weakness
CWE-400, CWE-407
Also known as
CVE-2026-129231

More NLTK advisories

All NLTK