Summary of previous notebooks
=============================
On 97-07-05 I obtained Landini's interlinear transcription of the
VMs, version 1.6 (landini-interln16.evt) from
http://sun1.bham.ac.uk/G.Landini/evmt/intrln16.zip
I manually extracted from it a homogeneous, full-text sample
bio-m-evt.evt, consisting of pages 147-166 (f75r--f84v) of the
"biological" section, in Currier's Language B, hand 2. This section
includes Currier's and Friedman's transcriptions. Currier's seems
to be the most complete of them.
The two versions have many differences (affecting 5-10% of the
words), and often disagree even in the grouping of symbols: where
one sees two words the other sees a single word, what is [A] for one
may be [CI] for the other, and so on.
So I decided to break all characters doen to individual "logical"
strokes, and use one (computer) character to encode each stroke.
I called this new encoding "jsa" (Jorge's Super-Analytic).
After mapping to jsa, I generated a "consensus" version
of the biological section, and got these digraph counts:
q o c i l g y s x j u TOT
----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- -----
. 1398 965 1877 361 60 . . . . . . 4661
q 1 . 1229 18 . 1 154 . . . 700 . 2103
o 21 486 1 63 1087 1071 . . . . . . 2729
c 4 167 176 6137 1209 232 2114 2921 1019 . . . 13979
i 4 1 1 8 1997 2 . . 560 1616 37 457 4683
l . . . . . . 16 . . . 1566 . 1582
g 52 . 74 2150 4 4 . . . . . . 2284
y 2790 26 2 47 13 43 . . . . . . 2921
s 463 1 99 1013 1 2 . . . . . . 1579
x 827 24 105 488 5 167 . . . . . . 1616
j 46 . 76 2175 6 . . . . . . . 2303
u 453 . 1 3 . . . . . . . . 457
----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- -----
TOT 4661 2103 2729 13979 4683 1582 2284 2921 1579 1616 2303 457 40897
Some conclusions we get from this and other data:
The valid \i/ sequences are \ij/ \is/ \iis/ \iiu/ \iiiu/ \ix/;
the others are likely to be scription or transcription errors.
\ci/ and \o/ are lexically similar but distinct glyphs.
The suffixes \ij/, \is/, \iiu/, and \iiiu/ are preceded
almost exclusively by \ci/ and strictly word-final. It seems
plausible that these are errors:
\oij/ (4 occurrences) should be \ciij/ ( 32 occurrences)
\oiiu/ (2 occurrences) should be \ciiiu/ (109 occurrences)
\ciiu/ (4 occurrences) should be \ciiiu/ (109 occurrences)
\oiiiu/ (9 occurrences) should be \ciiiiu/ (329 occurrences)
\ciiiiiu/ (4 occurrences) should be \ciiiiu/ (329 occurrences)
\ciiix/ (2 occurrences) should be \ciix/ (403 occurrences)
\ciiis/ (19 occurrences) may also be a misreading of \ciis/ (291 occurrences).
\cg/ is always a glyph.
\qo/ is a combination that occurs only in word-initial position.
\qc/ is likely to be a misreading/miswriting of \qo/.
\cy/ is always a glyph, almost certainly a final form of \ci/.
\qj/, \lj/, \qg/, \lg/ are glyphs.
\cs/ is a glyph closely related to (but distinct from) \c/.
\ccg/ is almost always followed by \ci/ or \cy/.
Here "glyph" means a group of strokes that can be treated as a single symbol
for analysis; it may actually be part of a larger, still unrecognized symbol.
Summarizing again:
\iiiu/, \iiu/, \iis/, \ij/
The ziggies: strictly final, preceded always by \ci/ or,
more rarely, by \o/.
\ix/
Usually initial or preceded by \ci/ or \o/;
followed by any letter except ziggies and \qo/,
\ix/, \is/
\is/
Similar to \ix/ except that it cannot be
followed by capitals or \cg/, either.
\cy/
Almost always final, but occasionaly followed by other letters.
Preceded by about the same letters as \ci/; indeed, it is
probably the final form of \ci/.
\cg/
May be followed by many letters, most often \cy/ and \ci/.
Almost always prededed by \c/, or initial; rarely by \ix/
or \o/.
\cs/
Most often followed by \c/, somewhat less often by \o/,
\ci/, or word break. Most often initial, but also
preceded by \ix/, gallows, \c/, \cy/, \cg/, \is/.
\lj/, \qj/
The H-gallows: Very similar to each other, different from the
rest, but somewhat similar to the P-gallows. They probably
combine with \c/ on both sides to make glyphs. It is very
likely that \l/ and \q/ are exactly equivalent.
\lg/, \qg/
The P-gallows: Very similar to each other, different from the
rest, but somewhat similar to the P-gallows. They probably
combine with \c/ on both sides to make glyphs. It is very
likely that \l/ and \q/ are exactly equivalent. They may be
merely ornate forms of some letter, or several letters (\cg/,
perhaps), used mainly in the first line of each paragraph (and
perhaps of each page?)
\qo/
Strictly initial, almost always followed by a capital.
Sometimes misread as \qc/?
\ci/
May be followed only by the ziggies, \ix/, or \ir/
only. Often follows a capital, but also \cg/,
\cs/, \c/, \ix/, \is/, or word break.
\o/
Similar to \a/, but is very often word-initial.
Other conclusions:
* The manuscript does not appear to use any hyphenation mark. Either
words are not broken across lines, which would be unusual, or they
are broken without any extra marks. Such word breaks may
result in statistical anomalies at the beginning and end of lines.
Could this explain Currier's claim that lines are "functional units"?
* Note that parsing sequences like \cij/, \ciis/, and \ciiis/ requires
some care: the right parsings are c+ij, c+iis, ci+iis.
* The parsing of \ciis/ is ambiguous: ci+is or c+iis. Declaring
\ciiis/ to be a misreading of \ciis/ would remove the ambiguity.
* The parsing of \ciiiu/ is ambiguous, too; but since the \iu/
series does not seem to follow a bare \c/, it seems safe to parse
it as ci+iiu.
* The gallows characters \qj/ and \lj/ appear to be closely related:
for every common word with \lj/, there appears to be a
a word with \qj/ that occurs with about 1/4 the frequency.
* There seems to be a kinship between the glyphs \cs/
(when not attached to the following \c/s)
\ir/, and the gallows \lj/ and \qj/ (also, when unattached).
* The same phenomenon can be noted with respect to prefixes
containing \cc/ and \csc/: for every word beginning with \cc/,
there is a word where the first \cc/ is replaced by \csc/,
and practically the same frequency.
* There apepars to be much confusion between the suffixes \iu/
and \iiiu/. They are almost surely distinct letters, but in
about one half of the cases, Currier sees \iiu/ where Friedman
has \iiiu/.
* There appears to be much confusions between \o/ and \ci/.
The strings of \c/, \cs/, \lj/, \qj/, \lg/, \qg/ must be treated
together, after collapsing the glyphs listed above, since there
seem to be glyphs consisting of gallows preceded and followed by
\c/ or \cc/. When this is taken into account, we can see that
a single \c/ is not a glyph, but \cs/ is. In fact, after
shrinking \ci/ to `a', \cs/ to `z', the gallows to `H' or `P', the
only possible glyphs of the form [czHp]* with length at most 3 are
freq glyph
---- -----
795 H
52 P
152 z
138 cc
70 zc
482 Hc
484 ccc
439 zcc ?
493 Hcc ?
19 cHc
4 cPc
The ones marked `?' may be composite, z+cc and H+cc, but this hypothesis
does not seem very likely (perhaps they are *sometimes* composite?)
The significant strings of length 4 that cannot be parsed into the glyphs above
are
20 cHcc
4 cPcc
Strings with 4 or more [czHP]'s tend to be quite ambiguous.
Looking at the raw texts, it seems that the main source of "?"s is
the confusion between "M" and "N" by Currier and/or Friedman. So I
decided to map both [N] and [M] (and other lookalikes) to "m".
I christened the new encoding "hop".
--- fsg2hop ------------------------
#! /n/gnu/bin/gawk -f
# Recoding an interlinear file from the FSG alphabet to
# my Lossy Ad-hoc Semi-Analytic Fault-Tolerant encoding
BEGIN {
print "# Output of fsg2hop - Stolfi's Semi-Analytic Fault-Tolerant alphabet"
}
/^ *$/ { print; next }
/^ *#/ { print; next }
/^<[^>.;]*>/ { print; next }
/^<[^>]*\.[^>]*;[A-Z]> / {
curtxt = substr($0,20)
# We discard "%" and "!" since the conversion
# will destroy synchronism anyway.
gsub(/[%!]/, "", curtxt);
# First, the conversion from FSG to JSA (Stolfi's super-analytic)
gsub(/IIIK/, "iiiij", curtxt);
gsub(/IIIL/, "iiiiu", curtxt);
gsub(/IIIR/, "iiiis", curtxt);
gsub(/IIIE/, "iiiix", curtxt);
gsub(/IIE/, "iiix", curtxt);
gsub(/IIR/, "iiis", curtxt);
gsub(/IIK/, "iiij", curtxt);
gsub(/HZ/, "cqjc", curtxt);
gsub(/PZ/, "cqgc", curtxt);
gsub(/DZ/, "cljc", curtxt);
gsub(/FZ/, "clgc", curtxt);
gsub(/IE/, "iix", curtxt);
gsub(/IR/, "iis", curtxt);
gsub(/IK/, "iij", curtxt);
gsub(/2/, "cs", curtxt);
gsub(/4/, "q", curtxt);
gsub(/6/, "cj", curtxt);
gsub(/7/, "ig", curtxt);
gsub(/8/, "cg", curtxt);
gsub(/A/, "ci", curtxt);
gsub(/C/, "c", curtxt);
gsub(/D/, "lj", curtxt);
gsub(/E/, "ix", curtxt);
gsub(/F/, "lg", curtxt);
gsub(/G/, "cy", curtxt);
gsub(/H/, "qj", curtxt);
gsub(/I/, "i", curtxt);
gsub(/K/, "ij", curtxt);
gsub(/L/, "iu", curtxt);
gsub(/M/, "iiiu", curtxt);
gsub(/N/, "iiu", curtxt);
gsub(/O/, "o", curtxt);
gsub(/P/, "qg", curtxt);
gsub(/R/, "is", curtxt);
gsub(/S/, "csc", curtxt);
gsub(/T/, "cc", curtxt);
gsub(/V/, "?", curtxt);
gsub(/Y/, "?", curtxt);
# Now, the conversion from JSA to HOP:
gsub(/[ql]j/, "H", curtxt);
gsub(/[ql]g/, "P", curtxt);
gsub(/cs/, "z", curtxt);
gsub(/ij/, "k", curtxt);
gsub(/ix/, "e", curtxt);
gsub(/is/, "r", curtxt);
gsub(/iiu/, "n", curtxt);
gsub(/y/, "i", curtxt);
gsub(/ci/, "a", curtxt);
gsub(/cg/, "8", curtxt);
gsub(/ir/, "w", curtxt);
gsub(/i*n/, "m", curtxt);
print (substr($0,1,19) curtxt);
next
}
------------------------------------
After mapping Currier and Friedman to the "hop" encoding, I created
a consensus bio-j-hop.evt.
I also created by hand a file bio-j-hop.evj, which is like
bio-j-hop.evt except that it has " " instead of "." as word-space,
and " //" instead of "-" for end-of-line, and " =" instead of "="
for end-of-paragraph. It allows me to find the page and line numbers
of a word, given its "hop" encoding.
Extracted the text files:
extract-words-from-interlin \
-chars "aocz8HPerqkmw" \
bio-j-hop.evt \
bio-j-hop
lines words bytes file
------ ------- --------- ------------
7670 7670 41815 bio-j-hop.wds
1510 1510 9982 bio-j-hop.dic
5894 5894 33804 bio-j-hop-gut.wds
949 949 6236 bio-j-hop-gut.dic
843 843 2464 bio-j-hop-fun.wds
5 5 24 bio-j-hop-fun.dic
933 933 5547 bio-j-hop-bad.wds
556 556 3722 bio-j-hop-bad.dic
Digraph counts:
a o c z 8 H P e r q k m w TT
----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- -----
. 251 1235 757 912 472 276 86 313 103 1489 . . . 5894
a 3196 2 4 19 26 14 78 2 491 345 5 39 802 23 5046
o 28 5 1 39 6 21 1776 68 1173 240 6 5 19 1 3388
c 10 1059 226 4047 44 1865 408 33 15 4 . . 5 . 7716
z 58 109 90 957 10 3 4 1 1 . . . . . 1233
8 64 2245 50 45 32 1 5 . 5 1 . . . . 2448
H 12 1125 98 1479 47 5 . . 9 . . . 1 . 2776
P 2 20 43 116 17 3 . . . . . . . . 201
e 1121 130 117 216 122 61 227 10 4 2 1 . . . 2011
r 514 90 48 24 15 3 1 . . . . . . . 695
q 1 5 1474 17 2 . 1 1 . . . . . . 1501
k 43 . 1 . . . . . . . . . . . 44
m 822 4 1 . . . . . . . . . . . 827
w 23 1 . . . . . . . . . . . . 24
----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- -----
TOT 5894 5046 3388 7716 1233 2448 2776 201 2011 695 1501 44 827 24 33804
Next-symbol probability (× 99):
a o c z 8 H P e r q k m w TT
-- -- -- -- -- -- -- -- -- -- -- -- -- -- --
. 4 21 13 15 8 5 1 5 2 25 . . . 99
a 63 . . . 1 . 2 . 10 7 . 1 16 . 99
o 1 . . 1 . 1 52 2 34 7 . . 1 . 99
c . 14 3 52 1 24 5 . . . . . . . 99
z 5 9 7 77 1 . . . . . . . . . 99
8 3 91 2 2 1 . . . . . . . . . 99
H . 40 3 53 2 . . . . . . . . . 99
P 1 10 21 57 8 1 . . . . . . . . 99
e 55 6 6 11 6 3 11 . . . . . . . 99
r 73 13 7 3 2 . . . . . . . . . 99
q . . 97 1 . . . . . . . . . . 99
k 97 . 2 . . . . . . . . . . . 99
m 98 . . . . . . . . . . . . . 99
w 95 4 . . . . . . . . . . . . 99
-- -- -- -- -- -- -- -- -- -- -- -- -- -- --
TOT 17 15 10 23 4 7 8 1 6 2 4 0 2 0 99
Previous-symbol probability (× 99):
a o c z 8 H P e r q k m w TT
-- -- -- -- -- -- -- -- -- -- -- -- -- -- --
. 5 36 10 73 19 10 42 15 15 98 . . . 17
a 54 . . . 2 1 3 1 24 49 . 88 96 95 15
o . . . 1 . 1 63 33 58 34 . 11 2 4 10
c . 21 7 52 4 75 15 16 1 1 . . 1 . 23
z 1 2 3 12 1 . . . . . . . . . 4
8 1 44 1 1 3 . . . . . . . . . 7
H . 22 3 19 4 . . . . . . . . . 8
P . . 1 1 1 . . . . . . . . . 1
e 19 3 3 3 10 2 8 5 . . . . . . 6
r 9 2 1 . 1 . . . . . . . . . 2
q . . 43 . . . . . . . . . . . 4
k 1 . . . . . . . . . . . . . 0
m 14 . . . . . . . . . . . . . 2
w . . . . . . . . . . . . . . 0
-- -- -- -- -- -- -- -- -- -- -- -- -- -- --
TOT 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99
Rebuilt .fix.wds, .fix.dic:
cat bio-j-hop.wds \
| sed -e '/?/s/^.*$/???/g' \
> .fix.wds
cat .fix.wds \
| sort | uniq \
> .fix.dic
cat .fix.wds \
| wfreq \
> .fix.frq
lines words bytes file
------ ------- --------- ------------
955 955 6264 .fix.dic
957 2871 17757 .fix.frq
7670 7670 40000 .fix.wds