#! /usr/bin/python3 import sys, re import html_gen as h from process_funcs import bash import html_report_funcs as hr last_edit = "Last edited on 2026-06-08 19:43:47 by stolfi" def main(): global last_edit title = "[050] Repeating token sequences in the SPS and the SBJ" st = h.new_doc(title, "#eeffdd", text_width = 1600) h.section(st, 2, "Summary") h.parags(st, """This report examines the statitistics of repeated token sequences in the Starred Parags section (SPS) of the Voynich Manuscript (VMS) and in the Shennong Bencaojing (SBJ), a famous old Chinese materia medica (list of remedies and their uses). This investigation is motivated by the recent claim that the SPS is a translation or transcription of the SBJ, apparently in a phonetic notation devised by the VMS Author for that purpose. One objection that has been raised agains this claim (the "SPS is SBJ Theory, of SSP) is the observation that in the SBJ there are many sequences of five or more consecutive tokens that are repeated verbatim multiple times; whereas the SPS has no similar repetitions. Here we examine these observations in some detail, provide possible explanations for how this discrepancy could have arisen, and present some computer experiments that justify the plausibility of those explanations. """) h.section{st, 2, "The observations") h,section(st, 3, "The Shennong Bencao text used") h.parags(st, """ The Shennong Bencaojing ("The Divine Farmer's Materia Medica Classic") is believed to have been composed around 300 BCE, and is traditionally attributed to Shennong, the "Divine Farmer", a mythical emperor who is also a god in Daoism. Perhaps for the last reason, the SBJ has been revered and quoted almost like a sacred text until recently, even though it grew many layers of additions and commentaries. The SBJ originally is said to have had 365 "recipes". Each recipe is about one medicinal substance (mineral, vegetable, or animal). Some recipes were actually two or more sub-recipes for different parts of the same plant or animal. Each recipe or sub-recipe contained mainly a list of uses of the substance for treatment of specific diseases. Several of the substances were designatd not-toxic tonics, and their continuous consumption was claimed to bring several benefits like beautiful skin or longevity. Unlike later reference books, the SBJ did not have any indications about harvesting, mode of preparation, dosage, and length of treatment.""") h.section(st, 4, "The source text and omissions") h.parags(st, """By 1400 CE there were no surviving copies of the SBJ as a standalone book, in its original form. The VMS Author must have transcribed a version that was embedded in one of the materia medicas available around 1400 CE; probably the state-sponsired three-volume Zhenghe Bencao (ZHB), which was first composed in ~1080 CE. In the ZHB, and possibly other contemporary medical encyclopedias, the entire text of the SBJ was quoted in large fonts and negative (white-on-black) printing, so it could have been easily read out while skipping all the massive additions and commentaries. Other evidence indicates that the Author systematically omitted certain parts of every entry, presumably because they would be useless or meaningless to him. One such part is a field between the recipe's title and the list of indications, consisting of the keyword 味 ("flavor") and 2-3 hanzi that specify the "theoretical flavor" and the "thermal nature" of the substance; two qualities that only make sense in the context of traditional Chinese medical theory (TCM). Also omitted were the "alias" fields, with keyword 一名 ("another name"), and a vague "provenance" field, with keyword 生 ("grows in"), which appeared at the end of most SBJ recipes.""") h.section(st, 4, "The digital SBJ file") h.parags(st, """ Ideally, for this investigation we should use the ZHB-embedded SBJ text. We did obtain a digital file of the ZHB, but apparently it was OCR-scanned from from a print dated 1780 or later, when printers had given up on the negative-print convention; so that the "pure" SBJ material cannot be easily extracted from it. Thefore, while looking for a more suitable digital copi of the ZHB, we have chosen to use file that we had prepared several months ago. It was created from two files titled "Shennong Bencaojing" that we downloaded from the Chinese Texts Project and the Chinese Wikisource. Those files are supposed to reflect a relatively recent (1700s and later) scholarly reconstructon of the long-lost version of the text as it would have existed about 2000 years ago. These files differ from the proper SBJ text (as emebedded in the ZHB) in many details. For instance, the ZHB version uses always 主 ("main" or "mainly for") to introduce the list of indications of the remedy, whereas these recontructions often uses 主治 ("main uses") instead. Those two files were in hanzi (Chinese characters) encoded in Unicode. Both files contained many errors, such as a few missing recipes and multiple recipes fused into one. These errors, when detected, were fixed by hand. They were converted from "traditional" hanzi to the modern "simplified" hanzi, and merged by hand, producing a ChineseTexts/Wikisource (CTW) hybrid file. We also replaced 主治 by 主, and fixed a couple dozen isolated entries by comparison with the ZHB file -- but many differences still remain. We then converted the files from Chinese characters to the corresponding readings in modern Mandarin, in the pinyin notation, by means of Google translate and an ad-hoc script to split multi-syllable compounds into single syllables. At this point we also removed all punctuation and mapped all letters to lower case. The resulting pinyin file has 363 recipes. The file is in the Unicode UTF-8 encoding, with tones marked as diacritics on the main vowels. digital SBJ file ??? ??? evidently did not have access to a copy of the original SBJ, much less to that reconstructed version. He Thus, for this report I have chosen to use the digital SBJ file specified above. The differences should not have much impact on the occurrences of repeated phrases. Thus, for example, here is the recipe about "bee larvae" (蜂子 fēng zǐ), containing the sub-recipes for "giant hornet larva" (大黄蜂子 dà huáng fēng zǐ) and "ground-nesting wasp larva" (土蜂子 tǔ fēng zǐ). fēng zǐ zhǔ fēng tóu chú gǔ dú bǔ xū léi shāng zhōng jiǔ fú lìng rén guāng zé hǎo yán sè bù lǎo dà huáng fēng zǐ zhǔ xīn fù zhàng mǎn tòng qīng shēn yì qì tǔ fēng zǐ zhǔ yōng zhǒng Note the three zhǔ (主) keywords for the three lists of indications, and the keyword jiǔ fú (久服 "prolonged consumption") before the lists of benefits of bee larvae as dietary supplement.""") h.section(st, 3, "The Starred Parags text used") h.parags(st, """The Starred Parags section is defined here as consisting of the whole of pages f103r to f115v, plus the first 30 lines of page f116r (thus excluding the parags of the latter with no stars.) It obviously excludes the missing central bifolio of the quire, the four pages f109r to f110v. It also excludes the three "titles" f104v.45, f105r.37, and f108v.53 (but not f105r.10, which we interpret as "overflow" from line f105r.11). The SPS file used for this investigation wa extracted from a new partial transcription of the VMS made by us in 2025-2026, moslty before we found that the SPS was a transcription of the SBJ. No attempt was made to make this transcription more similar to the SBJ in any way; although its main motivation was the realization that the correct identification of parag breaks would be crucial for the future work (still ongoing) of matching the SBJ recipes to the SPS parags. My new transcription differs from Rene Zandbergen's current IVT transcription in a number of small details; mostly readings of ambiguous glyphs, and quite a few more ambiguous spaces marked by commas. These differences have relatively litte impact on this investigation, and mostly in the direction of making it harder to prove our point. The linked file has been stripped of all formatting information eacept the locus IDs like '' In particular, all the #-comment lines and inline comments '' were removed, weirdo codes were replaced by '?', the braces '{}' used to indicate ligatures braces were deleted, and all EVA letters were mapped to lowercase. (Practically all the ligature information can still be recovered by the usual rules regarding the EVA letters @c and @h.) The extracted SPS file is in the ISO-Latin-1 encoding. """) h.section(st, 2, "The observations") h.section(st, """The main observation that this report intends to detail and expand is that one finds many phrases of up to 10 tokens that occur several times in the SBJ text, whereas the SPS (or in the VMS as a whole) has no repeated phrases with more than 4 words. Let's use the term {k}-repeat for an instance of a phrase of {k} consecutive words that occurs in two places in the text. (For a precise definition, see this page.) Here are the numbers of {k}-repeats in the SBJ and in the SPS, as a function of {k}:""") h.parags(st, """??? image histogram of k-repeats in unmodified SBJ and SPS""") h.parags(st, """Note that the SBJ file has hundreds of repeats with 6 or more words (???, to be exact) while the SPS file has none. Indeed, the latter result is what one expects from a text where each token is chosen idependently from the vocabulary, according to some fixed frequency distribution. In such a hypothetical "random" text with {n} tokens, if {p} is the probability that two words chosen that way are equal, then the probability of seeing a {k}-word repeat at any two given distinct positions {i0} and {i1} is {pk(1-p)2}; and the expected number of {k}-repeats would then be about {pk(1-p)2·n2/2. If the text uses hundreds of distinct words, {p} will be a few perrcent at most. Then one would not expect to see even a single {6}- or {7}-repeat in a text of 10'000 tokens. (A different probability formula applies if the two instances overlap, that is, if {|i1-i0|} is {k} or less, or if either {i0} or {i1} is zero or more than {n-k-1}; but this caveat does not change the general conclusion.)""") h.section(st, 3, "Nature of the long SBJ repeats") h;parags(st, """The anomalously large number of long SBJ repeats has a simple explanation. The bulk of each recipe is a list of conditions that the substance can cure or benefits that its consumption may bring. These lists are highly repetitive, since several substances happen to be indicated for the same conditions or benefits. The SBJ file has six repeats of 10-word sentences. Five of them occur in six consecutive recipes of my file, which have the same 10-hanzi list of benefits: * [久食]轻身不老,延年神仙。 * [jiǔ shí] qīng shēn bù lǎo, yán nián shén xiān. * [Prolonged Consumption]: Makes the body light and prevents aging, extends life to immortality. These six recipes (b1.2.018 to b1.2.023) are about six mushroom species that the SBJ authors apparently considered to be varieties of the same mushroom, with six (theoretical, not actual?) colors: cyan (青芝 Trametes versicolor), red (赤芝 Ganoderma lingzhi), yellow (黄芝 Laetiporus sulphureus), white (白芝 Fomitopsis officinalis), black (黑芝 Amauroderma rugosum), and purple (紫芝 Ganoderma sinense). The therapeutic (as opposed to long-term) sections are all different because the color was believed to "direct" the medicine to the organ of the same (theoretical) color: respectively the liver, heart, spleen, lungs, kidneys, and joints. The other 10-word repeat, likewise, is a 6-character warning about consequences of excessive dosage, and happens to include the first 4 characters of benefits of the [Prolonged Consumption] section. It occurs in the entries of two plants: * [多食]令人狂走。[久服]轻身, * [duō shí] lìng rén kuáng zǒu. [jiǔ fú] qīng shēn, * [Excessive use] causes one to run around wildly. [Prolonged consumption] lightens the body, The two recipes (b1.2.066 and b3.2.037) are about the plants Mysore thorn (云实), Biancaea decapetala and the seed of henbane (莨荡 子), Hyoscyamus niger; both with hallucinogenic properties (hence the "run around wildly" effect). The only 9-token repeat ia the phrase * ... 傅猪疮,饲猪肥大三倍 * ... fù zhū chuāng sì zhū féi dà sān bèi" * ... apply to pig sores, fed to pigs makes them grow three times bigger. Those words occur in two recipes (b3.3.089 and b3.3.090) that are consecutive in the SBJ file. According to Google AI that part of those two recipes may not be from the original SBJ but from another lost old book, the Shangqiu Zi's Pig Raising Classic. The other long repeats in the SBJ file are usually distinct recipes that share some indications or benefits. Some common 4-hanzi sequences are * [久服]轻身 jiǔ fú qīng shēn "[Prolonged use] lightens the body" (40 times) * 咳逆上气 hāi nì shàng qì "Coughing, wheezing with reverse qi" (12 times) * 轻身耐老 qīng shēn nài lǎo "lightens the body and delays aging" (11 times) * 轻身延年 qīng shēn yán nián "lightens the body and extends life" (11 times) * 轻身不老 qīng shēn bù lǎo "lightens the body and prevents aging" (10 times) * 寒热邪气 hán rè xié qì "chills and fever due to bad qi" (10 times) """) h.section(st, 3, "The mirage of language statistics") h.parags(st, """It has been claimed that long repeats are a diagnostic feature of natural languages (and, therefore, that their absence in the VMS text "proves" that it is no such thing). However, the above examination of the SBJ long repeats shows that they are not due to the language being natural, or to it being Chinese; but rather to the special nature, format, and style of the SBJ, specifically. This observation examplifies a point that is never stressed too much: textual statistics are properties of TEXTS, not of LANGUAGES. There is no such things as "the frequency of the word 'and' in English", or "the average entropy of a Latin word", or "the short-term correlation between 's' and 'h' in German". The statements "the most common letter in English is 'e'" and "the French language follows Zipf's law" are not true, and not even false: they are nonsensical. And so is the claim that "a text in a natural language will have many long repeated phrases". In any language there can be a quite meaningful and grammatically correct text that has certain statistical properties, and another text that does not. Someone once wrote an entire novel in English without using the letter 'e', not even once; and readers will not notice that "abnormality" unless they are told about it. When people make statistical claims about a language, they are almost always referring of a particular collection of texts in that language -- usually some large set of novels and newspaper articles. One cannot expect those statistics to hold for other types of text. Not even for texts that contain mostly free prose, like chemistry textbooks or food commercials. Amost certainly not for catalogs, timetables, logbooks, and other highly structured texts -- like the SBJ. And even less so for a text that was selected because its contents looked extremely unusual...""") h.section(st, 2, "Theories") . h.parags(st, """If the tokens of a text are chosen randomly and independently from a lexicon of hundreds of words with Zipf-like frequencies, the number of {k}-repeats should be practically zero for {k} gerater than 3. Thus the high counts of {k}-repeats in the SBJ with {k} up to 6 is remarkable, while the counts for the SPS look like those of randomly gnerated text. But if the SPS is indeed a transcription of the SBJ (as we claim, based on other strong structural evidence), how do we explain this glaring discrepancy? Here are some of the possible explanations.""") h.section(st, 3, "Missing pages") h.parags(st, """The close examination of the long repeats in the SBJ suggest a possible partial explanation for their absence in the VMS. Of the six 10-word repeats, six are in consecutive recipes of my digital SBJ file:""") h.parags(st, """While the overall order of recipes in that test file seems to be diffrerent from their order in the ZHB, and (as far as we seen) from their order in the SPS, those six mushroom entries are consecutive in the ZHB too; and therefore, almost certainy, would have been consecutive in the SPS too.""") h.parags(st, """ Thus one possible explanation for why they are not found in the SPS file is that they were on the four pages that are missing at the center of that quire (4/(22.6 + 4) = 15% of the original SPS text). (And that suggests one possible explanation for why those pages are missing: after Wilfrid obtained the VMS, he (or some expert whom he asked to examine the book) noticed those six paragraphs with identical 10-word endings. Worried that prospective clients would see that and think (like many Voynichologists today do) that such repetitions could not occur in a genuine Roger Bacon manuscript. So he made sure that the entire bifolio would "accidentally" disappear...)""") h.section(st, 3, "Dittoization and omission") h.parags(st, """Another possible explanation for the absence of such repeated phrases is that the the Author, noticing the repetitions, wrote "ditto" instead of repeating the words. Or even merged several similar entries under a single general parag, detailing only their differences. Or just omitted conditions and benefits that he considered silly or uninteresting, like the many claims of "extends life to immortality".""") h.section(st, 3, "Noise") h.parags(st, """However, the main cause for the absence of such repeated phrases in the VMS is likely to the "noise" --- various kinds of errors and meaningless non-systematic variations that, by "randomly" changing some occurrences of some tokens, break and obscure the long repeat chains. Some possible sources of such noise are listed below. Some of them detectable, while others are only conjectural or may depend on hypotheses that are still not confirmed.""") h.begin_enum(st, "ul") h.enum_parags(st, """Incompetent Dictator: In the general Chinese Origin Theory (COT), a proposed scenario for how the VMS was created is that the hanzi text of the SBJ (probably from the ZHB) was read aloud by a native Dictator, while the Author wrote it down each spoken syllable in a phonetic alphabet that he had devised for the purpose, namely the Voynichese script. Ideally the Dictator should have been a doctor or scholar who knew the meaning and reading of all the hanzi in the SBJ. However, many of those hanzi are rare outside the SBJ, and some even had already fallen out of use many centuries before. Thus, assuming this scenario is, it is possible that the Dictator was not ideal, and did not know the pronunciation of many of those hanzi. Rather than admit it, he would have guessed the pronunciation of an unknon hanzi from the "phonetic" part of the character, or just made it it up -- inconsistently.""") h.enum_parags(st, """Incompetent Author: Still under the COT, the Author presumably was somewhat fluent in the local language, only unable to read Chinese characters. Even so, he may still have been not quite capable to distinguish the sounds of the spoken language, like many people today who are only partially fluent in Mandarin may still find it difficult to distinguish the sounds of (pinyin) 'sh' from 'x', 'ch' from 'q', or the four tones. If that was the case, then he may have written those sounds inconsistently, sometime mis-hearing one sound for another.""") h.enum_parags(st, """Variable spelling: The spelling system devised by the author may have allowed the same spoken syllable to be written in two or more ways. Like English "adapter" and "adaptor", or "axe" and "ax". In Culpeper's Herbal "blue" was spelled "blew" 13 times out of 40. In particular, the encoding of tones may have been variable. Pinyin rules uniquely specify the vowel that receives the tone diacritic; when a digit 1-4 is used instead, it is usually placed at the end. But those rules are arbitrary: the tone is an overall feature of each syllable, that is not attached to any specific letter. Thus, in theory, the hanzi 曹 could be written indifferently 'cáo' (the pinyn standard), 'ćao', 'caó', 'cao2', 'ca2o', 'c2ao', or '2cao' --- all implying the same sound.""") h.enum_parags(st, """Homographs: A few Chinese characters have two or more different pronunciations, depending on their intended meaning. Like the word "read" in the English phrase "this is the kind of books I read" which can be pronounced in two different ways depending on whether it is intrepreted as present or past tense. A more specific example is the Chinese character 血 ("blood"), which has a literary/technical reading 'xuè' and a common reading 'xiě'. On the SBJ it occurs, for instance, as 脓血 ("pus and blood") and in 下血 ("to discharge blood"). The first is a single compound term, thus the standard technical reading (nóng xuè) would be expected. The second is a verb-object sentence, where 血 by itself is a noun; in that case one could use either the formal reading (xià xuè) or the common reading (xià xiě). A bit like English "cranium" and "skull": the compound "intracranial" is always like that, while "affects the skull" or "affects the cranium" are both OK. But then, when reading terms with two readings, like 下血, the Dictator may have chosen either reading, inconsistently. Whereas in the digital file it would be translated into pinyin always in the same way.""") h.enum_parags(st, """Sandhi: This is the linguistics term for changes to the "normal" pronunciation of a word that are determined by the adjacent words. In English, for example, the word "the" is pronounced differently in the phrases "the cat" and "the apple". Sandhi occurs in many languages, including Mandarin. In the latter, it commonly affects tones of syllables: 你 ("you") and 好 ("well") are pronounced nǐ and hǎo in isolation, but when together in 你好 ("hello") they are read ní hǎo, with the tone of the 你 changed from "dipping" to "rising". Sandhi could affect the frequency of long repeats by shortening a potential repeat by one word at either end. For example, the sentences 我希望你好 ("I hope you [sng.] are well") and 我希望你们好 ("I hope you [pl.] are well") begin with the same four hanzi 我希望你, but one would be pronounced 'wǒ xī wàng hǎo' while the other would be 'wǒ xī wàng men hǎo' In a strictly phonetic transcription (like the SPS is persumed to be), the two sentences would have only three shared syllables, whereas in a mechanical translation of hanzi to pinyin (like the digital file used here) both 你 may be transcribed as nǐ, resulting in four shared syllables. """) h.enum_parags(st, """Recording mistakes: The Author's script seems to have been designed with speed of writing as a major goal. Still, the recording of the dictation surely would not be able to keep up with the normal reading speed. Thus the recording must have been done in a hurry. The Author must have committed many mistakes, which he or she probably did not have time to go back and correct on the fly, even if they were immediately noticed.""") h.enum_parags(st, """Shorthand: The Voynichese script was probably more a shorthand than a proper phonetic script. For the sake of writing speed, a shorthand scheme generally omits many phonetic distinctons, trusting that they can later be recovered by context. Thus, for example, an English or Latin shorthand method might use the same symbol for 'p' and 'b', for 't' and 'd', and so on. The Voynichese script, in particualr, might have omit the tones altogether, besides merging certain vowels and consonants. If applied consistently, such alphabet reduction should increase the number of repeated sentences, since it would increase the probablity of two words being equal. But if it was applied inconsistently -- sometimes recording a distinction, sometimes omitting it, at random -- it would have the opposite effect, by mapping the same spoken syllable to different written words.""") h.enum_parags(st, """Bad handwriting: For the same reason, the handwriting in that 'first draft', taken under dictation, must have been a very poor cursive form of the Voynichese script. Even if that document was later transcribed at a more leisurely pace into a 'second draft', the result may still have been hard to read. Then the Scribe(s) that the Author hired to make the vellum copy may have often misread a glyph and replaced it by one with a similar shape. There is some evidence that this indeed happened. When matching the SBJ recipes to SPS parags, one finds that occurrences of the hanzi 主 ("mainly for") generally match occurrences of the EVA string @daiin; but sometimes there is a @kaiin, @laiin, or @dair in the expected place. While other explanations are possible, it is notable that the characters @k (if written with reduced height) and @l could resemble @d in slightly sloppy handwriting; and likewise @ir resembles @iin, not only in shape but (as had been observed long ago) also in its occurrence within the various word structure paradigms. ???image k-to-d ???image n-to-ir On the other hand, there are very few cases where @taiin appears as the Voynichese for 主. And indeed the second loop of the @t makes it very hard to confuse with a @d, even with sloppy handwriting.""") h.enum_parags(st, """Careless Scribe: ???""") h.enum_parags(st, """Fading and retracing: The ink of the text and drawings faded badly in the decades or centuries after it was laid down, to the point of near or total invisibility. We believe that, for that reason, the owner of the manuscript or some hired scribe undertook a comprehnsive restoration of the book. We claim that this Restorer painstakingly retraced practically all of the text, and a variable fraction of the figure outlines, using ink with color similar as the original. The evidence for this Massive Retracing theory (MRT) is subtle but extensive, including bits of original faded ink sticking out from under the new one, abrupt variations of trace weight (not explainable by pen recharging), plumes and tails that are visibly deformed for having been traced in the wrong direaction, and more. Unlike the original Scribe, this Restorer could not have been trained to recognize and copy the glyphs of the Voynichese alphabet. As a consequence, wherever the original glyphs had become partly invisible, the attempt to restore them often resulted in "weirdos" -- missshapen glyphs that typically occur only once or twice, like an @o with the plume of an @s, or an @a with the tail of an @y, t. """) h.enum_parags(st, """Digitalization errors: ???""") h.end_enum(st, "ul") h.parags(st, """The above speculations may not be all true, but surely they are not all false either. Each of them implies that the SPS file, unlike the SBJ one, has a large number of random perturbations, which are much more likely to break long repeats than to create new ones.""") h.section(st, 2, "Experiments") h.parags(st, """In order to make the above explanation more plausible, we conducted some computer experiments that consisted in (1) applying to the SBJ file crude simulations of the perturbation processes listed above, and (2) applying to the SPS file some ad-hoc "error correcting" transformations that hopefully will counteract the effect of those perturbations. The goal is to make the two resulting files have similar numbers of {k}-repeats.""") h.section(st, 3, "Perturbing the SBJ file") h.parags(st, """???""") h.section(st, 3, "De-noising the SPS file") h.parags(st, """???""") h.section(st, 3, "Results") h.parags(st, """???""") h.section{st, 2, "Conclusions") h.parags(st, """We have identified several perturbation processes that may have occurred in the conversion of the Shennong Bencao into the Starred Parags section of the VMS, creating noise in the latter -- random modifications to the words (syllables), that spoil the repeated phrases we see in the former. The computer simulations and rough reversals of those processes, we have shown that they could indeed explain why the SPS has so few repeatd phrases.""") hr.links_section(st) h.section(st, 2, "Disclaimer about 'AI' use") h.parags(st, """In this research I used Large Language Models ("AI" systems), specifically Google AI and ChatGPT, to understand the structure and history of the SBJ and its various editions, and to translate individual Chinese SBJ recipes into English; as well as for punctual help with details of programming languages and other software tools. I also used Google AI and Google Translate to convert the Chinese text of the SBJ into phonetic Mandarin (pinyin). On the other hand, apart from these translations, I did not use any of those services to write the text of this report, or to develop any of its conclusions and conjectures. All statistical and structural analyses, of both the SBJ and the SPS, were done with standard Linux tools and programs developed by myself. Those "AI" tools were also never asked to analyze or process any part of the VMS text.""", style = "font-size:60%") h.output_doc(st, sys.stdout, 99, last_edit) return 0 # ---------------------------------------------------------------------- main()