Most strings today in software are Unicode strings. It means that you can include mathematical symbols, emojis and so forth. There are many different versions of the letter ‘M’, for example: the Roman letter M (U+004D) is semantically different from the Roman numeral Ⅿ (U+216F) while they both often have the same visual representation. John Cook has an interesting post on Unicode Stegonography: you can possibly use this ambiguity to hide messages in plain view. E.g., if you need to warn someone that you are in danger, you could send a text with the Roman numeral M. Normal people reading the text would not notice the difference.
What about URLs like Microsoft.com? What if you replace the Roman letter by a Roman numeral, is it still the same domain?
It is. URL parsers are required to normalize the URLs which involves, among other things, replacing look-alike letters with Roman letters if they are to be compliant with the WHATWG URL specification.
But do they? Do the URL parsers actually do this hard work? Let us check.
Java. I could not get the standard Java library to return to me the host. It simply returns a null String.
String url = "https://microsoft.coⅯ"; URI uri = new URI(url); String host = uri.getHost();
C#. The .NET library seems to just returns the domain as-is with the Roman numeral when using the Host attribute, but it works correct with the IdnHost property.
string url = "https://microsoft.coⅯ"; Uri uri = new Uri(url); string host = uri.IdnHost;
PHP. The standard PHP interpreter just returns the domain as-is, with the Roman numeral
$url = "https://microsoft.coⅯ"; $parsed_url = parse_url($url); if ($parsed_url === false) { echo "URL could not be parsed."; } else { $host = $parsed_url['host']; }
Go. Go also does not do normalization.
urlString := "https://microsoft.coⅯ" parsedURL, err := url.Parse(urlString) if err != nil { fmt.Println("URL could not be parsed:", err) return } host := parsedURL.Host
Python. You guessed it: no normalization. It happily returns the Roman numeral.
url = "https://microsoft.coⅯ" parsed_url = urllib.parse.urlparse(url) host = parsed_url.netloc
JavaScript. JavaScript does it correctly. It will convert https://microsoft.coⅯ to https://microsoft.com.
const url = "https://microsoft.coⅯ"; const urlObj = new URL(url); const host = urlObj.hostname;
C++. C++ does not have a standard URL parser, but if you use the ada URL parser, you will get correct results. If you are using the Node.js runtime environment, the underlying parser is the C++ ada URL parsing library.
auto url = ada::parse("https://microsoft.coⅯ"); if (!url) { /* failure */ } std::string_view host = url->get_host();
Further reading: Host/Split: Exploitable Antipatterns in Unicode Normalization by Jonathan Birch (Microsoft), Unicode URL Hack by Bruce Schneier, Equivocal URLs: Understanding the Fragmented Space of URL Parser Implementations. (European Symposium on Research in Computer Security 2022), NorthSec 2020 – Philippe Arteau – Unicode vulnerabilities that could byͥte you.
Daniel Lemire, "How does your URL parser handle Unicode?," in Daniel Lemire's blog, January 2, 2025, https://lemire.me/blog/2025/01/02/how-does-your-url-parser-handle-unicode/.
[BibTeX]
IDNA Mentioned? 🙂
In the .NET case, you’d use the Uri.IdnHost property instead of Host for that specific purpose: https://learn.microsoft.com/en-us/dotnet/api/system.uri.idnhost
Your comparison of URL normalization across programming languages is fascinating! Have you come across any real-world scenarios where inconsistent handling of Unicode in URLs led to security vulnerabilities or usability issues?
I have updated the blog post, please see the further reading section for details. There are definitively security concerns.
//Java
//import java.text.Normalizer;
//import java.text.Normalizer.Form;
//ONLY NFKC or NFKD Compatibility decomposition Form works
var uri = new URI(Normalizer.normalize(url, Form.NFKC));
Handling Unicode in URL parsing can be surprisingly tricky, especially when dealing with normalization and percent-encoding inconsistencies across different libraries. Have you come across cases where domain name internationalization (IDN) conversions introduce unexpected edge cases? Would love to hear your thoughts on best practices for ensuring consistency across different programming languages!
The .NET Uri.IdnHost vs Host distinction is crucial! I recently encountered an issue where our authentication system broke because some users registered with IDN emails containing Unicode characters. This would have saved us days of debugging. Are there any performance considerations when using IdnHost in high-volume applications?
This discussion about NFKC normalization in Java is spot on. We found that using the wrong form could actually create security vulnerabilities through homograph attacks. Has anyone implemented a comprehensive test suite for these edge cases they’d recommend sharing?
As a security engineer, I can’t stress enough how important proper URL normalization is. We recently discovered an API bypass vulnerability stemming from inconsistent Unicode handling between our load balancer and application servers. What monitoring strategies do others use to catch these normalization discrepancies in production?
The internationalization challenges with URLs go even deeper when you consider right-to-left languages. We had a nasty bug where Arabic domains would sometimes get parsed backwards. Does the .NET IdnHost property handle these bidirectional text cases correctly?
Interesting to see the language-specific approaches! For web applications accepting user-generated URLs, would you recommend normalizing on input, storage, or both? We’ve seen cases where early normalization actually loses important contextual information.
Forget normalizing—it’s like every language is auditioning for a role in a comedy. At least JavaScript knows the script! 😂