5 March 2026 · 3 min
The Internet relies on text formats. Thus, we spend a lot of time producing and consuming data encoded in text.
Your web pages are HTML. The code running in them is JavaScript, sent as text (JavaScript source), not as already-parsed code. Your emails, including their attachments, are sent as text (your binary files are sent as text).
It does not stop there. The Python code that runs your server is stored as text. It queries data by sending text queries. It often gets back the answer as text that must then be decoded.
JSON is the universal data interchange format online today. We share maps as JSON (GeoJSON).
Not everything is text, of course. There is no common video or image format that is shared as text. Transmissions over the Internet are routinely compressed to binary formats. There are popular binary formats that compete with JSON.
But why is text dominant?
It is not because, back in the 1970s, programmers did not know about binary formats.
In fact, we did not start with text formats. Initially, we worked with raw binary data. Those of us old enough will remember programming in assembly using raw byte values.
Why text won?
1.Text is efficient.
In the XML era, when everything had to be XML, there were countless proposals for binary formats. People were sometimes surprised to find that the binary approach was not much faster in practice. Remember that many text formats date back to an era when computers were much slower. Had text been a performance bottleneck, it would not have spread. Of course, there are cases where text makes things slower. You then have a choice: optimize your code further or transition to another format. Often, both are viable.
It is easy to make wrong assumptions about binary formats, such as that you can consume them without any parsing or validation. If you pick up data from the Internet, you must assume that it could have been sent by an adversary or someone who does not follow your conventions.
2.Text is easy to work with.
If you receive text from a remote source, you can often transform it, index it, search it, quote it, version it… with little effort and without in-depth knowledge of the format. Text is often self-documenting.
In an open world, when you will never speak with the person producing the data, text often makes everything easier and smoother.
If there is an issue to report and the data is in text, you can usually copy-paste the relevant section into a message. Things are much harder with a binary format.
Daniel Lemire, "Text formats are everywhere. Why?," in Daniel Lemire's blog, March 5, 2026, https://lemire.me/blog/2026/03/05/text-formats-are-everywhere-why/.
[BibTeX]
Do you think we’ll see new text formats that are more aligned with the way LLMs work and parse content? I know JSON is notorious in this regard (unless you have a separate schema available).
It looks like the TOON format is addressing this. More compact, less repeated key material.
https://www.toon-kit.com/docs
https://openapi.com/blog/what-the-toon-format-is-token-oriented-object-notation
Good question.
The TOON format addresses this issue. It is more compact, especially because it avoids repeated key values for lists of homogenous record types. TOON stands for “Token-Oriented Object Notation.”
https://www.toon-kit.com/docs
https://openapi.com/blog/what-the-toon-format-is-token-oriented-object-notation
There is a common image format as text. NetPBM (from 1988) converts various binary/proprietary image formats to a common (text) format, offers a series of tools to manipulate that format, and then converts back to binary formats.
https://en.wikipedia.org/wiki/Netpbm
Was quite useful at one time.
Jef Poskanzer (the guy originally behind NetPBM) has done a number of interesting things over the years.
> The Internet relies on text formats.
None of IP, TCP, UDP, SCTP, DNS, SSH, or, say, BitTorrent are “textual”. The WWW relies on text, for one definition of the word.
> The Python code that runs your server is stored as text.
Yes, but code can be stored in better ways, as Lisp systems have done. Storing code in a form which must be parsed increases the space of errors, yet the direction should be to narrow this space.
> It queries data by sending text queries. It often gets back the answer as text that must then be decoded.
Much of this manner of thing falls downwards from the inexpressiveness of our computers. I’m aware both VMS and Microsoft Windows both had language-agnostic communications systems, but UNIX never added one.
> Not everything is text, of course. There is no common video or image format that is shared as text.
That’s because it’s very hard to pretend such a thing would be sensible, too hard.
> It is not because, back in the 1970s, programmers did not know about binary formats.
Yes, but they didn’t know what they were doing, the pair at AT&T I’ve got in mind.
> 1.Text is efficient.
JSON is regularly a bottleneck and, no matter how many SIMD parsers are thrown at it, elimination remains the single best optimization of such inefficiencies.
> Had text been a performance bottleneck, it would not have spread.
What if, say, AT&T weren’t allowed to sell it and people prefer gratis over all other considerations?
> It is easy to make wrong assumptions about binary formats, such as that you can consume them without any parsing or validation.
In comparison to “textual formats” that’s absolutely the case. There may be parsing, literally splitting into parts, but it’s orders of magnitude simpler, as is verification.
> 2.Text is easy to work with.
This is true, for ASCII. As soon as Unicode gets involved, many of the problems return, but some people labour under the delusional idea that ASCII should be privileged forevermore by pushing UTF-8 as the one true Unicode encoding. Even then, Unicode is constantly changing, which makes it unsuitable for a great deal of things. The only reasonable way to cope with Unicode is a whitelist of characters.
> If you receive text from a remote source, you can often transform it, index it, search it, quote it, version it… with little effort and without in-depth knowledge of the format.
I’ve used the example of a database as compared to CSV before. A database is not merely its tables, but constraints and assumptions too, whose need to be properly documented and encoded never leaves. It’s certainly easier to misinterpret data in a more common format.
I once sent an e-mail to Doug McIllroy asking if the “textual” approach were chosen simply because of the ostensibly generic tooling, but he never gave a direct and satisfactory response to this simple question. Yes, one may vomit forth the contents of a file to an emulated terminal device, but that’s neither generic nor particularly useful.
> If there is an issue to report and the data is in text, you can usually copy-paste the relevant section into a message. Things are much harder with a binary format.
It goes without saying that “non-textual” formats, I call them “numerical formats”, should have a system which makes them comprehensible to the operator. The trick is believing such things to be unnecessary with what passes for “text” nowadays. That’s how one gets poorly documented configuration formats and other nonsense.