305 lines
9.7 KiB
XML
305 lines
9.7 KiB
XML
|
<chapter id="clusters">
|
||
|
|
<sect1 id="clusters">
|
||
|
|
<title>Clusters</title>
|
||
|
|
<para>
|
||
|
|
In shaping text, a <emphasis>cluster</emphasis> is a sequence of
|
||
|
|
code points that needs to be treated as a single, indivisible unit.
|
||
|
|
</para>
|
||
|
|
<para>
|
||
|
|
When you add text to a HB buffer, each character is associated with
|
||
|
|
a <emphasis>cluster value</emphasis>. This is an arbitrary number as
|
||
|
|
far as HB is concerned.
|
||
|
|
</para>
|
||
|
|
<para>
|
||
|
|
Most clients will use UTF-8, UTF-16, or UTF-32 indices, but the
|
||
|
|
actual number does not matter. Moreover, it is not required for the
|
||
|
|
cluster values to be monotonically increasing, but pretty much all
|
||
|
|
of HB's tests are performed on monotonically increasing cluster
|
||
|
|
numbers. Nevertheless, there is no such assumption in the code
|
||
|
|
itself. With that in mind, let's examine what happens with cluster
|
||
|
|
values during shaping under each cluster-level.
|
||
|
|
</para>
|
||
|
|
<para>
|
||
|
|
HarfBuzz provides three <emphasis>levels</emphasis> of clustering
|
||
|
|
support. Level 0 is the default behavior and reproduces the behavior
|
||
|
|
of the old HarfBuzz library. Level 1 tweaks this behavior slightly
|
||
|
|
to produce better results, so level 1 clustering is recommended for
|
||
|
|
code that is not required to implement backward compatibility with
|
||
|
|
the old HarfBuzz.
|
||
|
|
</para>
|
||
|
|
<para>
|
||
|
|
Level 2 differs significantly in how it treats cluster values.
|
||
|
|
Levels 0 and 1 both process ligatures and glyph decomposition by
|
||
|
|
merging clusters; level 2 does not.
|
||
|
|
</para>
|
||
|
|
<para>
|
||
|
|
The conceptual model for what the cluster values mean, in levels 0
|
||
|
|
and 1, is this:
|
||
|
|
</para>
|
||
|
|
<itemizedlist spacing="compact">
|
||
|
|
<listitem>
|
||
|
|
<para>
|
||
|
|
the sequence of cluster values will always remain monotone
|
||
|
|
</para>
|
||
|
|
</listitem>
|
||
|
|
<listitem>
|
||
|
|
<para>
|
||
|
|
each value represents a single cluster
|
||
|
|
</para>
|
||
|
|
</listitem>
|
||
|
|
<listitem>
|
||
|
|
<para>
|
||
|
|
each cluster contains one or more glyphs and one or more
|
||
|
|
characters
|
||
|
|
</para>
|
||
|
|
</listitem>
|
||
|
|
</itemizedlist>
|
||
|
|
<para>
|
||
|
|
Assuming that initial cluster numbers were monotonically increasing
|
||
|
|
and distinct, then all adjacent glyphs having the same cluster
|
||
|
|
number belong to the same cluster, and all characters belong to the
|
||
|
|
cluster that has the highest number not larger than their initial
|
||
|
|
cluster number. This will become clearer with an example.
|
||
|
|
</para>
|
||
|
|
</sect1>
|
||
|
|
<sect1 id="a-clustering-example-for-levels-0-and-1">
|
||
|
|
<title>A clustering example for levels 0 and 1</title>
|
||
|
|
<para>
|
||
|
|
Let's say we start with the following character sequence and cluster
|
||
|
|
values:
|
||
|
|
</para>
|
||
|
|
<programlisting>
|
||
|
|
A,B,C,D,E
|
||
|
|
0,1,2,3,4
|
||
|
|
</programlisting>
|
||
|
|
<para>
|
||
|
|
We then map the characters to glyphs. For simplicity, let's assume
|
||
|
|
that each character maps to the corresponding, identical-looking
|
||
|
|
glyph:
|
||
|
|
</para>
|
||
|
|
<programlisting>
|
||
|
|
A,B,C,D,E
|
||
|
|
0,1,2,3,4
|
||
|
|
</programlisting>
|
||
|
|
<para>
|
||
|
|
Now if, for example, <literal>B</literal> and <literal>C</literal>
|
||
|
|
ligate, then the clusters to which they belong "merge".
|
||
|
|
This merged cluster takes for its cluster number the minimum of all
|
||
|
|
the cluster numbers of the clusters that went in. In this case, we
|
||
|
|
get:
|
||
|
|
</para>
|
||
|
|
<programlisting>
|
||
|
|
A,BC,D,E
|
||
|
|
0,1 ,3,4
|
||
|
|
</programlisting>
|
||
|
|
<para>
|
||
|
|
Now let's assume that the <literal>BC</literal> glyph decomposes
|
||
|
|
into three components, and <literal>D</literal> also decomposes into
|
||
|
|
two. The components each inherit the cluster value of their parent:
|
||
|
|
</para>
|
||
|
|
<programlisting>
|
||
|
|
A,BC0,BC1,BC2,D0,D1,E
|
||
|
|
0,1 ,1 ,1 ,3 ,3 ,4
|
||
|
|
</programlisting>
|
||
|
|
<para>
|
||
|
|
Now if <literal>BC2</literal> and <literal>D0</literal> ligate, then
|
||
|
|
their clusters (numbers 1 and 3) merge into
|
||
|
|
<literal>min(1,3) = 1</literal>:
|
||
|
|
</para>
|
||
|
|
<programlisting>
|
||
|
|
A,BC0,BC1,BC2D0,D1,E
|
||
|
|
0,1 ,1 ,1 ,1 ,4
|
||
|
|
</programlisting>
|
||
|
|
<para>
|
||
|
|
At this point, cluster 1 means: the character sequence
|
||
|
|
<literal>BCD</literal> is represented by glyphs
|
||
|
|
<literal>BC0,BC1,BC2D0,D1</literal> and cannot be broken down any
|
||
|
|
further.
|
||
|
|
</para>
|
||
|
|
</sect1>
|
||
|
|
<sect1 id="reordering-in-levels-0-and-1">
|
||
|
|
<title>Reordering in levels 0 and 1</title>
|
||
|
|
<para>
|
||
|
|
Another common operation in the more complex shapers is when things
|
||
|
|
reorder. In those cases, to maintain monotone clusters, HB merges
|
||
|
|
the clusters of everything in the reordering sequence. For example,
|
||
|
|
let's again start with the character sequence:
|
||
|
|
</para>
|
||
|
|
<programlisting>
|
||
|
|
A,B,C,D,E
|
||
|
|
0,1,2,3,4
|
||
|
|
</programlisting>
|
||
|
|
<para>
|
||
|
|
If <literal>D</literal> is reordered before <literal>B</literal>,
|
||
|
|
then the <literal>B</literal>, <literal>C</literal>, and
|
||
|
|
<literal>D</literal> clusters merge, and we get:
|
||
|
|
</para>
|
||
|
|
<programlisting>
|
||
|
|
A,D,B,C,E
|
||
|
|
0,1,1,1,4
|
||
|
|
</programlisting>
|
||
|
|
<para>
|
||
|
|
This is clearly not ideal, but it is the only sensible way to
|
||
|
|
maintain monotone indices and retain the true relationship between
|
||
|
|
glyphs and characters.
|
||
|
|
</para>
|
||
|
|
</sect1>
|
||
|
|
<sect1 id="the-distinction-between-levels-0-and-1">
|
||
|
|
<title>The distinction between levels 0 and 1</title>
|
||
|
|
<para>
|
||
|
|
So, the above is pretty much what cluster levels 0 and 1 do. The
|
||
|
|
only difference between the two is this: in level 0, at the very
|
||
|
|
beginning of the shaping process, we also merge clusters between
|
||
|
|
base characters and all Unicode marks (combining or not) following
|
||
|
|
them. E.g.:
|
||
|
|
</para>
|
||
|
|
<programlisting>
|
||
|
|
A,acute,B
|
||
|
|
0,1 ,2
|
||
|
|
</programlisting>
|
||
|
|
<para>
|
||
|
|
will become:
|
||
|
|
</para>
|
||
|
|
<programlisting>
|
||
|
|
A,acute,B
|
||
|
|
0,0 ,2
|
||
|
|
</programlisting>
|
||
|
|
<para>
|
||
|
|
This is the default behavior. We do it because Windows did it and
|
||
|
|
old HarfBuzz did it, so this remained the default. But this behavior
|
||
|
|
makes it impossible to color diacritic marks differently from their
|
||
|
|
base characters. That's why in level 1 we do not perform this
|
||
|
|
initial merging step.
|
||
|
|
</para>
|
||
|
|
<para>
|
||
|
|
For clients, level 0 is more convenient if they rely on HarfBuzz
|
||
|
|
clusters for cursor positioning. But that's wrong anyway: cursor
|
||
|
|
positions should be determined based on Unicode grapheme boundaries,
|
||
|
|
NOT shaping clusters. As such, level 1 clusters are preferred.
|
||
|
|
</para>
|
||
|
|
<para>
|
||
|
|
One last note about levels 0 and 1. We currently don't allow a
|
||
|
|
<literal>MultipleSubst</literal> lookup to replace a glyph with zero
|
||
|
|
glyphs (i.e., to delete a glyph). But in some other situations,
|
||
|
|
glyphs can be deleted. In those cases, if the glyph being deleted is
|
||
|
|
the last glyph of its cluster, we make sure to merge the cluster
|
||
|
|
with a neighboring cluster.
|
||
|
|
</para>
|
||
|
|
<para>
|
||
|
|
This is, primarily, to make sure that the starting cluster of the
|
||
|
|
text always has the cluster index pointing to the start of the text
|
||
|
|
for the run; more than one client currently relies on this
|
||
|
|
guarantee.
|
||
|
|
</para>
|
||
|
|
<para>
|
||
|
|
Incidentally, Apple's CoreText does something else to maintain the
|
||
|
|
same promise: it inserts a glyph with id 65535 at the beginning of
|
||
|
|
the glyph string if the glyph corresponding to the first character
|
||
|
|
in the run was deleted. HarfBuzz might do something similar in the
|
||
|
|
future.
|
||
|
|
</para>
|
||
|
|
</sect1>
|
||
|
|
<sect1 id="level-2">
|
||
|
|
<title>Level 2</title>
|
||
|
|
<para>
|
||
|
|
Level 2 is a different beast from levels 0 and 1. It is simple to
|
||
|
|
describe, but hard to make sense of. It simply doesn't do any
|
||
|
|
cluster merging whatsoever. When things ligate or otherwise multiple
|
||
|
|
glyphs turn into one, the cluster value of the first glyph is
|
||
|
|
retained.
|
||
|
|
</para>
|
||
|
|
<para>
|
||
|
|
Here are a few examples of why processing cluster values produced at
|
||
|
|
this level might be tricky:
|
||
|
|
</para>
|
||
|
|
<sect2 id="ligatures-with-combining-marks">
|
||
|
|
<title>Ligatures with combining marks</title>
|
||
|
|
<para>
|
||
|
|
Imagine capital letters are bases and lower case letters are
|
||
|
|
combining marks. With an input sequence like this:
|
||
|
|
</para>
|
||
|
|
<programlisting>
|
||
|
|
A,a,B,b,C,c
|
||
|
|
0,1,2,3,4,5
|
||
|
|
</programlisting>
|
||
|
|
<para>
|
||
|
|
if <literal>A,B,C</literal> ligate, then here are the cluster
|
||
|
|
values one would get under the various levels:
|
||
|
|
</para>
|
||
|
|
<para>
|
||
|
|
level 0:
|
||
|
|
</para>
|
||
|
|
<programlisting>
|
||
|
|
ABC,a,b,c
|
||
|
|
0 ,0,0,0
|
||
|
|
</programlisting>
|
||
|
|
<para>
|
||
|
|
level 1:
|
||
|
|
</para>
|
||
|
|
<programlisting>
|
||
|
|
ABC,a,b,c
|
||
|
|
0 ,0,0,5
|
||
|
|
</programlisting>
|
||
|
|
<para>
|
||
|
|
level 2:
|
||
|
|
</para>
|
||
|
|
<programlisting>
|
||
|
|
ABC,a,b,c
|
||
|
|
0 ,1,3,5
|
||
|
|
</programlisting>
|
||
|
|
<para>
|
||
|
|
Making sense of the last example is the hardest for a client,
|
||
|
|
because there is nothing in the cluster values to suggest that
|
||
|
|
<literal>B</literal> and <literal>C</literal> ligated with
|
||
|
|
<literal>A</literal>.
|
||
|
|
</para>
|
||
|
|
</sect2>
|
||
|
|
<sect2 id="reordering">
|
||
|
|
<title>Reordering</title>
|
||
|
|
<para>
|
||
|
|
Another tricky case is when things reorder. Under level 2:
|
||
|
|
</para>
|
||
|
|
<programlisting>
|
||
|
|
A,B,C,D,E
|
||
|
|
0,1,2,3,4
|
||
|
|
</programlisting>
|
||
|
|
<para>
|
||
|
|
Now imagine <literal>D</literal> moves before
|
||
|
|
<literal>B</literal>:
|
||
|
|
</para>
|
||
|
|
<programlisting>
|
||
|
|
A,D,B,C,E
|
||
|
|
0,3,1,2,4
|
||
|
|
</programlisting>
|
||
|
|
<para>
|
||
|
|
Now, if <literal>D</literal> ligates with <literal>B</literal>, we
|
||
|
|
get:
|
||
|
|
</para>
|
||
|
|
<programlisting>
|
||
|
|
A,DB,C,E
|
||
|
|
0,3 ,2,4
|
||
|
|
</programlisting>
|
||
|
|
<para>
|
||
|
|
In a different scenario, <literal>A</literal> and
|
||
|
|
<literal>B</literal> could have ligated
|
||
|
|
<emphasis>before</emphasis> <literal>D</literal> reordered; that
|
||
|
|
would have resulted in:
|
||
|
|
</para>
|
||
|
|
<programlisting>
|
||
|
|
AB,D,C,E
|
||
|
|
0 ,3,2,4
|
||
|
|
</programlisting>
|
||
|
|
<para>
|
||
|
|
There's no way to differentiate between these two scenarios based
|
||
|
|
on the cluster numbers alone.
|
||
|
|
</para>
|
||
|
|
<para>
|
||
|
|
Another problem happens with ligatures under level 2 if the
|
||
|
|
direction of the text is forced to opposite of its natural
|
||
|
|
direction (e.g. left-to-right Arabic). But that's too much of a
|
||
|
|
corner case to worry about.
|
||
|
|
</para>
|
||
|
|
</sect2>
|
||
|
|
</sect1>
|
||
|
|
</chapter>
|