Text Handling Done Right: ASCII, Unicode, and Designing for Localisation

TextText looks like the easiest data type in the language — a string is just a sequence of characters — right up until a program has to work with more than one language, script, or region. At that point almost every convenient assumption a Latin-alphabet, English-only programmer makes breaks quietly: that one character is one byte, that a string's length is a stable number, that a sentence can be built by gluing fragments together, that sorting a list of names is a solved problem. This page covers two problems usually taught separately that are really the same problem at two different layers: what a byte actually represents (encoding), and what a program is allowed to assume about the text it shows a user (localisation) — followed by a worked comparison of how seven languages, including this site's own PatLang, handle the encoding layer differently, and where each design pushes the resulting bugs.

ASCII, and the assumption everything else inherited

ASCIIASCII assigns 128 code points (0–127) to unaccented Latin letters, digits, punctuation, and a handful of control characters. It fits in seven bits, with the eighth bit of a byte historically left free for parity checking or vendor extensions. It is also the root of the single most persistent bad assumption in text handling: that one character is one byte. A huge amount of early file-format design, network protocol design, and fixed-width UI layout assumed this permanently, not provisionally — and every extension since (the 8-bit code pages, then Unicode) has had to work around that assumption rather than replace it cleanly, because too much existing text was already valid, meaningful ASCII.

Unicode is a numbering scheme, not an encoding

TheThe single most common conceptual confusion is naming Unicode and an encoding as though they were the same thing, and Joel Spolsky's widely-read primer opens by drawing this line¹. Unicode itself is a table: it assigns every character in essentially every script in current and historical use a number, called a code point, written like U+0041 for the capital letter A. It says nothing about how those numbers become bytes in memory or on disk. That is a second, separate decision, and there are several different, mutually incompatible answers to it — UTF-8, UTF-16, and UTF-32 are three ways of writing down the same set of Unicode code points as bytes, and the older single-byte code pages are older, narrower schemes Unicode was built to eventually replace. "Unicode" and "UTF-8" are not synonyms, even though everyday usage collapses them constantly: Unicode is the numbering; UTF-8 is one of several ways to encode it.

Why UTF-8 became the practical default

UTF-8,UTF-8, standardised as RFC 3629², encodes each code point as one to four bytes. Three properties explain why it displaced almost everything else as the default interchange format. First, backward compatibility: every code point in the original ASCII range encodes to the single byte ASCII already used for it, so any file that was already valid ASCII is automatically valid, unchanged UTF-8. Second, it is self-synchronising: the leading bits of any byte tell you immediately whether it starts a new character or continues one, so a program can find a safe character boundary from any byte offset without scanning from the start of the string. Third, and often overlooked, UTF-8 never produces the byte value 0x00 for anything except the code point U+0000 — which means C's decades-old null-terminated string convention keeps working, unmodified, on UTF-8 text. That property is a large part of why UTF-8 could spread through existing C-based operating systems and tooling without forcing a flag-day rewrite of everything that touched a string.

Three different "lengths" for one string, and where the bugs live

AA single piece of text has at least three different, all-legitimate notions of "length," and most encoding bugs come from code that silently assumes two of them are the same number:

• Byte length — how much storage the encoded text actually occupies.
• Code point count — how many Unicode numbers the text decodes to.
• Grapheme cluster count — how many "characters" a user would actually perceive on screen. Unicode Standard Annex #29 defines where these boundaries fall³, and the answer is often more than one code point per cluster: a letter with a combining accent, a flag, or a family emoji built from a zero-width-joiner sequence are each one character to a user and several code points underneath.

Conflating these produces a recognisable family of bugs: truncating user text at a fixed byte count and cutting a multi-byte sequence in half, which mangles the tail into replacement characters or, in older non-validating code, something worse; a "maximum 20 characters" validation rule checked against byte length rather than code points or grapheme clusters, which silently makes the real limit far stricter for users typing in Chinese, Japanese, Korean, or many other scripts, as a side effect of the encoding rather than any deliberate constraint; naive slicing in a UTF-16-based language landing in the middle of a two-unit surrogate pair; and sorting or case-folding that assumes one universal order when collation is inherently locale-dependent — German phone-book order differs from German dictionary order for ß, and Turkish's dotted/dotless i/ı pair breaks the naive uppercase/lowercase round-trip that most mainstream languages ship as a locale-blind default.

Designing for localisation from the first line, not the last

LocalisationLocalisation is not "translate the strings at the end." The mistakes that make translation expensive, slow, or simply wrong are architectural, made long before anyone opens a translation tool. A short set of rules catches most of them:

• Never build a sentence by concatenating fragments. The W3C's own internationalisation guidance names this directly as a primary authoring hazard⁴: "Returned " + count + " results" breaks the moment a language needs a different word order, a different plural form, or puts the number somewhere else in the sentence entirely. Treat the whole sentence as one translatable template with placeholders, and hand plural and grammatical agreement to a message-formatting layer (ICU MessageFormat, gettext's ngettext, or equivalent) instead of scattering hand-rolled if (count == 1) checks through the codebase.
• Grammatical number is not binary. Many languages have more than the singular/plural split English uses: several Slavic languages have three or four plural forms depending on the last digit of the count, and Arabic has six. A locale-aware pluraliser needs the count and the target language, not a boolean.
• Dates, numbers, currency, sort order, and text direction are locale properties, not constants. Field order (day/month/year versus month/day/year), the decimal separator, right-to-left scripts needing the whole layout mirrored rather than just the characters reversed — none of these are safe to bake into a fixed format string.
• A name does not have a fixed shape. Patrick McKenzie's widely-cited list of assumptions programmers make about names⁵ is the canonical reference for how many "obviously true" properties — a first and last name, one canonical spelling, ASCII characters, a bounded length, uniqueness — turn out to be false somewhere in the world. Validation copied from one culture's naming convention reliably rejects real users from every other one.
• Never derive logic from the current wording of a translatable string. A comparison like if (status == "Complete") against user-facing display text breaks the instant that string is translated, or even just reworded. Keep a stable internal code separate from whatever string is shown to a user.
• Test with a real non-Latin script early, not at the end. A build only ever exercised in English will not tell you that a German label is three times longer than the box drawn for it, or that a right-to-left layout needs more than swapped text. Testing against an artificially expanded pseudo-locale, or simply against Arabic, Japanese, or German, catches layout assumptions while they are still cheap to fix.

How seven languages actually handle the encoding layer

TheThe theory above is the same everywhere. What a specific language actually gives you for "the length of this string" or "the character at this position" varies enough to matter in practice:

• Python 3. A str is a sequence of Unicode code points; the interpreter's internal storage width is an implementation detail invisible to the programmer. len(s) counts code points, not bytes and not grapheme clusters. Text and bytes are deliberately separate types with no implicit coercion between them — s.encode('utf-8') and b.decode('utf-8') are the only crossing points — a hard boundary Python 3 introduced specifically to catch the string/bytes confusion that Python 2's implicit coercion let slide until it broke in production. Grapheme-cluster-aware operations still need a third-party library; the standard library only understands code points.
• JavaScript. Strings are sequences of UTF-16 code units, not code points. .length counts UTF-16 units, and any code point outside the Basic Multilingual Plane — many emoji, some CJK extension characters — is stored as a two-unit surrogate pair, so "😀".length is 2, not 1. Naive slicing can split a surrogate pair and produce an unpaired half that renders as a broken glyph. for...of, the spread operator, and Array.from() are code-point aware; Intl.Segmenter goes further and is grapheme-cluster aware for user-facing operations like cursor movement or truncation.
• Java. char is a 16-bit UTF-16 code unit, inherited from the same early-1990s assumption that Unicode would fit in 16 bits that JavaScript inherited. String.length() counts UTF-16 units and charAt(i) can return half a surrogate pair. codePointAt and codePoints() (Java 8+) are the code-point-aware alternatives.
• Go. A string is an immutable slice of bytes with no encoding guaranteed by the type itself, though Go source and its standard library treat strings as UTF-8 by firm convention. len(s) is byte length. Ranging over a string with for i, r := range s decodes UTF-8 automatically and yields rune values (Go's name for code points) — but i is the byte offset of each rune, not its ordinal position, which trips up anyone expecting a plain index. utf8.RuneCountInString(s) gives the code point count directly when that is what's actually needed.
• Rust. String/str are guaranteed valid UTF-8 at the type level; there is no safe way to construct one that isn't. .len() is byte length, and there is deliberately no way to index a str by integer position at all — s[i] does not compile, because "the i-th character" is not a well-defined operation without first choosing bytes, code points, or grapheme clusters. Byte-range slicing (&s[0..4]) panics at runtime if the boundary lands mid-code-point rather than silently returning corrupted text. .chars() iterates code points; grapheme clusters again need a crate (unicode-segmentation, which implements UAX #29 directly). Rust's design here is a direct, deliberate response to the class of bug the JavaScript and Java entries above describe — it pushes the byte/code-point/grapheme distinction into the type system rather than hoping the programmer remembers it every time.
• C. There is no built-in string type, only a null-terminated array of bytes (char*). Standard library functions like strlen and strcmp operate purely on bytes and know nothing about multi-byte sequences — which is safe for UTF-8 specifically, thanks to the null-byte property described above, but the classic <ctype.h> functions (isalpha, toupper) are byte-oriented and only correct for the "C" locale or other single-byte encodings. The locale-aware, wide-character alternatives (<wctype.h>, wchar_t) exist, but wchar_t's own size is platform-defined — 2 bytes on Windows (effectively UTF-16-like) versus 4 bytes on Linux/glibc (effectively UTF-32-like) — which is its own portability trap. Modern practice treats C strings as opaque UTF-8 byte buffers and does text processing (case-folding, collation, segmentation) through a dedicated Unicode library such as ICU rather than through libc.
• PatLang. This site's own self-hosted language indexes strings by Unicode code point, not by byte: char_code, substr, and plain indexing (s[i]) all walk code points via the runtime's chars() iterator for any string containing non-ASCII bytes, so — unlike Go or naive C code — indexing into a non-ASCII PatLang string does not risk landing mid-character. The trap PatLang actually has is a performance one, not a correctness one: as a fast path, the runtime checks whether a string is pure ASCII (in which case a byte offset and a code-point offset are the same number, so it can skip decoding entirely) — but that ASCII check itself has to scan the whole string, and if it is repeated inside a per-character loop rather than cached once, an intended O(1) per-character access silently becomes O(n) per access, turning a linear scan into a quadratic one. This is a documented bug this site's own PatLang material found and fixed: an html_unescape pass over a 1.2MB article went from never finishing to 65× faster once the ASCII check was cached once at intern time (str_intern) instead of recomputed on every character read (see PatLang Best Practices for the full account). Like most of the languages above, PatLang has no documented grapheme-cluster segmentation of its own — code-point counting is as close to "character counting" as its standard library gets.

A practical checklist

• Decode at the input boundary, encode at the output boundary; keep everything in between as text, not bytes with an assumed encoding.
• Pick UTF-8 as the interchange encoding unless something external forces otherwise.
• Never truncate, slice, or validate the length of text by raw byte count when what you actually mean is characters — and remember that even "characters" has three different legitimate meanings.
• Never build a user-facing sentence by concatenating translated fragments; use a template with placeholders and a pluralisation/formatting layer.
• Treat date, number, currency, sort order, and text-direction formatting as locale lookups, never as constants.
• Keep internal control-flow codes separate from user-facing display strings.
• Test against a different script, or a pseudo-locale, before layout and length assumptions get expensive to unwind.

Where this connects

• PatLang Best Practices — the full account of the ASCII-fast-path performance trap this page's PatLang entry summarises, including the fix and the migration pattern for character-scanning loops.
• Testing Fundamentals — property-based testing with generated strings is one of the more reliable ways to catch encoding-boundary bugs before a real user does.
• Risk Management: Lessons from Testing Real Systems — the same pattern this page's PatLang example follows: a real bug, found by testing at real scale, not invented for the write-up.

References