5 December 2023 · 2 min
When you recover textual content from the disk or from the network, you may expect it to be a Unicode string in UTF-8. It is the most common format. Unfortunately, not all sequences of bytes are valid UTF-8 and accepting invalid UTF-8 without validating it is a security risk.
How might you validate a UTF-8 string in a JavaScript runtime?
You might use the valid-8 module:
import valid8 from "valid-8"; if(!valid8(file_content)) { console.log("not UTF-8"); }
new TextDecoder("utf8", { fatal: true }).decode(file_content)
import { isUtf8 } from "node:buffer"; if(!isUtf8(file_content)) { console.log("not UTF-8"); }
| Arabic | Chinese | Latin | |
| valid-8 | 0.14 GB/s | 0.17 GB/s | 0.50 GB/s |
| TextDecoder | 0.18 GB/s | 0.19 GB/s | 7 GB/s |
| node:buffer | 17 GB/s | 17 GB/s | 44 GB/s |
The current isUtf8 function in Node.js was implemented by Yagiz Nizipli. It uses the simdutf library underneath. John Keiser should be credited for the UTF-8 validation algorithm.
Daniel Lemire, "How fast can you validate UTF-8 strings in JavaScript?," in Daniel Lemire's blog, December 5, 2023, https://lemire.me/blog/2023/12/05/how-fast-can-you-validate-utf-8-strings-in-javascript/.
[BibTeX]
How about limiting it to “in the browser”? Could one compile
simdutfto WebAssembly and use its SIMD instruction set to out-doTextDecoderandvalid-8? Seems like maybe not yet, there’s an open issue for adding WebAssembly SIMD support.The proper solution would be from browsers to adopt simdutf.