NLTK: XMLCorpusView comment and CDATA parsing permits quadratic CPU denial of service
> affected: PyPI nltk >= 3.6 , nltk.corpus.reader.xmldocs.XMLCorpusView streaming read (and the nps_chat / framenet readers built on it)
> dynamically reproduced: 3.6 (6/6 checks), 3.9.2 (6/6), 3.10.3 (6/6) and develop 6818767b9 (6/6)
---
_Source references (nltk/corpus/reader/xmldocs.py:NNN) are to commit 6818767b99c24659107eb2be9bccd32a3d4bf045 (develop, v3.10.3-15-g6818767b9). The released v3.10.3 wheel's xmldocs.py is byte-identical to that commit, so the same line numbers apply to the released wheel._
Summary
**An attacker who can make a victim read a well-formed XML corpus file whose body contains one large comment or CDATA section can pin a CPU core for a time quadratic in the file size, because XMLCorpusView._read_xml_fragment (nltk/corpus/reader/xmldocs.py) re-runs a linear whole-buffer validation after every 1 KiB block while the comment/CDATA terminator is still ahead.** XMLCorpusView is NLTK's documented streaming API for XML corpora and is instantiated directly by the NPSChatCorpusReader and FramenetCorpusReader readers; the module header (xmldocs.py:16-20) explicitly handles untrusted corpus XML. The in_tag guard added to this function for GHSA-vp2x covers only a single oversized <...> tag and leaves the CDATA/comment path quadratic in the current release.
Details
Root cause:...
Affected versions
| Package | Affected | Fixed in |
|---|---|---|
| nltk PyPI | >= 3.6, < 3.10.4 | 3.10.4 |
Details and references
> **affected**: PyPI `nltk` `>= 3.6` , `nltk.corpus.reader.xmldocs.XMLCorpusView` streaming read (and the `nps_chat` / `framenet` readers built on it) > **dynamically reproduced**: `3.6` (6/6 checks), `3.9.2` (6/6), `3.10.3` (6/6) and `develop` `6818767b9` (6/6) --- _Source references (`nltk/corpus/reader/xmldocs.py:NNN`) are to commit `6818767b99c24659107eb2be9bccd32a3d4bf045` (`develop`, `v3.10.3-15-g6818767b9`). The released `v3.10.3` wheel's `xmldocs.py` is byte-identical to that commit, so the same line numbers apply to the released wheel._ ### Summary **An attacker who can make a victim read a well-formed XML corpus file whose body contains one large comment or CDATA section can pin a CPU core for a time quadratic in the file size, because `XMLCorpusView._read_xml_fragment` (`nltk/corpus/reader/xmldocs.py`) re-runs a linear whole-buffer validation after every 1 KiB block while the comment/CDATA terminator is still ahead.** `XMLCorpusView` is NLTK's documented streaming API for XML corpora and is instantiated directly by the `NPSChatCorpusReader` and `FramenetCorpusReader` readers; the module header (`xmldocs.py:16-20`) explicitly handles untrusted corpus XML. The `in_tag` guard added to this function for GHSA-vp2x covers only a single oversized `<...>` tag and leaves the CDATA/comment path quadratic in the current release. ### Details Root cause: `_read_xml_fragment` (`xmldocs.py:273-338`) accumulates the file into one `fragment` string in `_BLOCK_SIZE` (1024) character chunks (`xmldocs.py:123, 298-299`). After every block that is not yet a well-formed prefix, the fragment is re-validated against `_VALID_XML_RE`. Two whole-buffer operations dominate: ```python # xmldocs.py:308 if not in_tag and self._VALID_XML_RE.match(fragment): # whole-buffer match (O(len)) return fragment # xmldocs.py:312 if re.search("[<>]", fragment).group(0) == ">": # O(1) for this input class: ... # the document starts with '<' # xmldocs.py:329 if self._VALID_XML_RE.match(fragment[:last_open_bracket]): # whole-buffer prefix match (O(len)) ... ``` For this input class the `re.search` probe at `xmldocs.py:312` is O(1): the fragment always starts with `<` (e.g. `<root>…`), so the first `<`/`>` character is at offset 0 and it never contributes cost. The quadratic cost comes entirely from the two whole-buffer regex matches at `xmldocs.py:308` and `xmldocs.py:329` (the latter scans the buffer up to the most recent `<`, which for this input sits near the end of the buffer). `_VALID_XML_RE` (`xmldocs.py:238-249`) is a *linear* regex (its tempered-dot comment/CDATA bodies `(?:(?!-->).)*` / `(?:(?!\]\]>).)*` scan left-to-right in O(length) , confirmed by the embedded script's single-match check), but for a fragment whose terminator is far ahead the *match itself fails after scanning the whole buffer*. With an unclosed-across-blocks CDATA or comment, **every one of the n/1024 blocks** pays O(current buffer length) on lines 308 and 329, for a total of ~O(n²) character operations. The `in_tag` guard added for GHSA-vp2x (`xmldocs.py:283-305`) tracks only bare `<`/`>` characters: it is `True` inside an unclosed `<...>` tag and, in that state, skips only the line-308 match. It never represents "the fragment sits inside an unclosed CDATA or comment" , XML allows arbitrary `<`/`>` inside those constructs , so both the line-308 match and the line-329 backtrack match still fire on every block for a CDATA/comment fragment. The comment block at `xmldocs.py:283-289` even documents the intended linearity contract ("re-running it on every 1 KiB block otherwise makes a single oversized tag O(n^2) (CWE-407)"); the guard implements it for the tag branch only. The single-match cost is linear, not super-linear: the embedded script measures `_VALID_XML_RE.match()` on 512 KiB → 1 MiB → 2 MiB unclosed-CDATA buffers and gets ≈ 0.074 s → 0.142 s → 0.280 s (≈ 1:1.9:3.8, i.e. linear), asser