{"id":21809,"date":"2025-01-02T16:07:31","date_gmt":"2025-01-02T16:07:31","guid":{"rendered":"https:\/\/lemire.me\/blog\/?p=21809"},"modified":"2025-02-08T01:38:26","modified_gmt":"2025-02-08T01:38:26","slug":"how-does-your-url-parser-handle-unicode","status":"publish","type":"post","link":"https:\/\/lemire.me\/blog\/2025\/01\/02\/how-does-your-url-parser-handle-unicode\/","title":{"rendered":"How does your URL parser handle Unicode?"},"content":{"rendered":"

Most strings today in software are Unicode strings. It means that you can include mathematical symbols, emojis and so forth. There are many different versions of the letter ‘M’, for example: the Roman letter M (U+004D) is semantically different from the Roman numeral \u216f (U+216F) while they both often have the same visual representation. John Cook has an interesting post on Unicode Stegonography<\/em><\/a>: you can possibly use this ambiguity to hide messages in plain view. E.g., if you need to warn someone that you are in danger, you could send a text with the Roman numeral M. Normal people reading the text would not notice the difference.<\/p>\n

What about URLs like Microsoft.com? What if you replace the Roman letter by a Roman numeral, is it still the same domain?<\/p>\n

It is. URL parsers are required to normalize the URLs which involves, among other things, replacing look-alike letters with Roman letters if they are to be compliant with the WHATWG URL specification.<\/p>\n

But do they? Do the URL parsers actually do this hard work? Let us check.<\/p>\n

Java<\/strong>. I could not get the standard Java library to return to me the host. It simply returns a null String.<\/p>\n

 String<\/span> url =<\/span> \"<\/span>https:\/\/microsoft.co\u216f<\/span>\"<\/span>;<\/span><\/span>\r\n URI uri =<\/span> new<\/span> URI(<\/span>url)<\/span>;<\/span><\/span>\r\n String<\/span> host =<\/span> uri.<\/span>getHost(<\/span>)<\/span>;<\/span><\/span><\/pre>\n
C#<\/strong>. The .NET library seems to just returns the domain as-is with the Roman numeral when using the Host attribute, but it works correct with the IdnHost property.<\/p>\n
string<\/span> url =<\/span> \"<\/span>https:\/\/microsoft.co\u216f<\/span>\"<\/span>;<\/span><\/span>\r\nUri uri =<\/span> new<\/span> Uri(<\/span>url)<\/span>;<\/span><\/span>\r\nstring<\/span> host =<\/span> uri.<\/span>IdnHost;<\/span><\/span><\/pre>\n
PHP<\/strong>. The standard PHP interpreter just returns the domain as-is, with the Roman numeral<\/p>\n
$url =<\/span> \"<\/span>https:\/\/microsoft.co\u216f<\/span>\"<\/span>;<\/span><\/span>\r\n$parsed_url =<\/span> parse_url(<\/span>$url)<\/span>;\r\nif<\/span> (<\/span>$parsed_url =<\/span>=<\/span>=<\/span> false<\/span>)<\/span> {\r\n echo \"<\/span>URL could not be parsed.<\/span>\"<\/span>;\r\n} else<\/span> {\r\n $host =<\/span> $parsed_url[<\/span>'host'<\/span>]<\/span>;\r\n}\r\n<\/span><\/span>\r\n\r\n<\/pre>\n
Go<\/strong>. Go also does not do normalization.<\/p>\n
urlString :<\/span>=<\/span> \"<\/span>https:\/\/microsoft.co\u216f<\/span>\"<\/span><\/span>\r\nparsedURL,<\/span> err :<\/span>=<\/span> url.<\/span>Parse(<\/span>urlString)<\/span><\/span>\r\nif<\/span> err !<\/span>=<\/span> nil {<\/span><\/span>\r\n        fmt.<\/span>Println(<\/span>\"<\/span>URL could not be parsed:<\/span>\"<\/span>,<\/span> err)<\/span><\/span>\r\n        return<\/span><\/span>\r\n}<\/span><\/span>\r\nhost :<\/span>=<\/span> parsedURL.<\/span>Host<\/span><\/pre>\n
Python<\/strong>. You guessed it: no normalization. It happily returns the Roman numeral.<\/p>\n
url =<\/span> \"<\/span>https:\/\/microsoft.co\u216f<\/span>\"<\/span><\/span>\r\nparsed_url =<\/span> urllib.<\/span>parse.<\/span>urlparse(<\/span>url)<\/span><\/span>\r\nhost =<\/span> parsed_url.<\/span>netloc<\/span><\/pre>\n
JavaScript<\/strong>. JavaScript does it correctly. It will convert https:\/\/microsoft.co\u216f to https:\/\/microsoft.com.<\/p>\n
const<\/span> url =<\/span> \"<\/span>https:\/\/microsoft.co\u216f<\/span>\"<\/span>;<\/span><\/span>\r\nconst<\/span> urlObj =<\/span> new<\/span> URL(<\/span>url)<\/span>;<\/span><\/span>\r\nconst<\/span> host =<\/span> urlObj.<\/span>hostname;<\/span><\/span><\/pre>\n
auto<\/span> url =<\/span> ada::<\/span>parse(<\/span>\"<\/span>https:\/\/microsoft.co\u216f<\/span>\"<\/span>)<\/span>;<\/span><\/span>\r\nif<\/span> (<\/span>!<\/span>url)<\/span> {<\/span> \/* failure *\/<\/span> }<\/span><\/span>\r\nstd<\/span>::<\/span>string_view host =<\/span> url-<\/span>><\/span>get_host(<\/span>)<\/span>;<\/span><\/span><\/pre>\n
Further reading<\/strong>: Host\/Split: Exploitable Antipatterns in Unicode Normalization<\/a> by Jonathan Birch (Microsoft), Unicode URL Hack<\/a> by Bruce Schneier, Equivocal URLs: Understanding the Fragmented Space of URL Parser Implementations<\/a>. (European Symposium on Research in Computer Security 2022), NorthSec 2020 \u2013 Philippe Arteau \u2013 Unicode vulnerabilities that could by\u0365te you.<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
Most strings today in software are Unicode strings. It means that you can include mathematical symbols, emojis and so forth. There are many different versions of the letter ‘M’, for example: the Roman letter M (U+004D) is semantically different from the Roman numeral \u216f (U+216F) while they both often have the same visual representation. John … Continue reading How does your URL parser handle Unicode?<\/span><\/a><\/p>\n","protected":false},"author":56,"featured_media":21879,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[84],"tags":[],"class_list":["post-21809","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-84"],"_links":{"self":[{"href":"https:\/\/lemire.me\/blog\/wp-json\/wp\/v2\/posts\/21809","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/lemire.me\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/lemire.me\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/lemire.me\/blog\/wp-json\/wp\/v2\/users\/56"}],"replies":[{"embeddable":true,"href":"https:\/\/lemire.me\/blog\/wp-json\/wp\/v2\/comments?post=21809"}],"version-history":[{"count":5,"href":"https:\/\/lemire.me\/blog\/wp-json\/wp\/v2\/posts\/21809\/revisions"}],"predecessor-version":[{"id":21823,"href":"https:\/\/lemire.me\/blog\/wp-json\/wp\/v2\/posts\/21809\/revisions\/21823"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/lemire.me\/blog\/wp-json\/wp\/v2\/media\/21879"}],"wp:attachment":[{"href":"https:\/\/lemire.me\/blog\/wp-json\/wp\/v2\/media?parent=21809"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/lemire.me\/blog\/wp-json\/wp\/v2\/categories?post=21809"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/lemire.me\/blog\/wp-json\/wp\/v2\/tags?post=21809"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
C++<\/strong>. C++ does not have a standard URL parser, but if you use the ada URL parser,<\/a> you will get correct results. If you are using the Node.js runtime environment, the underlying parser is the C++ ada URL parsing library.<\/p>\n