28 September 2024 · 3 min
with open(file_path, 'r') as file: data = json.load(file) wealth_str = data['user']['wealth'] wealth = float(wealth_str)
He replaced the Python code by the following C++ routine embedded in a small console application:
simdjson::ondemand::parser parser; simdjson::padded_string json = simdjson::padded_string::load(myfile); simdjson::ondemand::document doc = parser.iterate(json); std::string_view url_string = doc["user"]["wealth"].get_string(); double result; fast_float::from_chars(url_string.data(), url_string.data() +url_string.size(), result);
Naturally, I assumed that a C++ solution would outperform Python in this scenario. However, I questioned whether the difference justified the development effort. The engineer was adamant that it did. Intrigued, I chose to conduct a small benchmark to test this claim. If the application was sophisticated, I would have used a command-line argument parser such as cxxopts, but I just went bare metal.
I measure how many times a second each solution can query the content of a small JSON file. Importantly, I record the entire time required, from the launch of the process. I use hyperfine (an open source tool) to get accurate measurements. I use my macBook (M2) with Apple LLVM 15 and Python 3.12.
| solution | times/second |
|---|---|
| C++ | 270 |
| Python | 22 |
include(cmake/CPM.cmake) CPMAddPackage("gh:fmtlib/fmt#11.0.2") CPMAddPackage("gh:simdjson/[email protected]") CPMAddPackage("gh:fastfloat/[email protected]") add_executable(main main.cpp) target_link_libraries(main fmt::fmt simdjson::simdjson FastFloat::fast_float)
Daniel Lemire, "It is never too late to write your own C/C++ command-line utilities," in Daniel Lemire's blog, September 28, 2024, https://lemire.me/blog/2024/09/28/it-is-never-too-later-to-write-your-own-c-c-command-line-utilities/.
[BibTeX]
Before going to C++ I’d probably have sprung for something like DuckDB which is a sort of Swiss Army knife in the data analysis field. It’s written in C++ and can read among other things json.
I wonder if this would have closed the gap.
Imagine keeping the parsed JSON in memory and prepare only if the file changes … That would increase the speed many orders faster.
Or, just do not use JSON file at all. 😁
For the steps in README.md, what is script.py?
I had botched the README file. Corrected. The script was checked in.
I did my own test. The Python runs at about 35ms a run.
Doing a duckdb -c “select user.wealth::numeric from read_json(‘./wealth.json’)” in a shell script took about 15ms a run.
The C++ is far faster as expected.
If the script is really just parsing this json then he could’ve tried swaping json.loads for orjson.loads which is often a magnitude faster. Maybe also read the file with ‘rb’ instead of ‘r’. That can make a small difference on some platforms.
However, in my opinion the real problem seems to be having to query and parse the json so often in the first place and then accessing a single value from it. That can’t be solved by using a different implementation.
I’m curious, if the problem at hand really is that simple as you’ve described it, how does jq measure up to it? Couldn’t find an answer in the hackernews thread, but I did see others wondering the same.
Indeed. I don’t think that jq alone was possible for the real problem, but I could have added it to my tests. I did not because I wanted to keep things simple.
I’ve tried recreating what you did. Two quick observations:
1) it looks like almost the entire time is just python start up. It’s slower than c++ but not that much slower than the C++ code for the actual parsing time.
If I use a short python script that is literally just one instruction, “pass”, hyperfine measures it at 48.3ms, your script.py executes in 51.1ms, so ~2ms actual time doing what the script was there to do.
In fact, even throwing a 1000 loop _within_ the python script only sees execution time reach 59.8ms, suggesting that json parsing gets really fast.
2) system python is about twice as slow to start on my laptop (Ubuntu 24.04), as a venv python created from the system python.
system python of “script.py wealth.json”:
51.1 ms ± 5.2 ms
venv python of “script.py wealth.json”:
16.6 ms ± 1.9 ms
I haven’t attempted to do things like strace the execution to see what the difference is there, but I’m guessing that the venv is cleaner and python is having to touch fewer files during start-up/import. That’s a pure guess though.
I created a venv because I was curious enough to try both pyinstaller and nuitka, both produced significantly slower binaries (80ms for nuitka, 108ms for pyinstaller)
it looks like almost the entire time is just python start up.
Yes, that is what I expect. Even in Python, parsing a small JSON file is going to be much shorter than milliseconds… or it should be.
Well, if something’s so important, then calling the same program over and over again is insanity. I’d like to see more comparisons with good dynamic languages like Common Lisp over just Python, because otherwise these kinds of results can mislead people into thinking the dynamic nature be the problem.
> If your favorite video game was powered by this C++ app, it could seamlessly update every frame without breaking a sweat.
Notice video games don’t use the extremely inefficient UNIX model of interprocess communication and the like.
Of course, this is kind of like reading about the hyper optimized string length routines seen in the Linux kernel: Anyone can take worthless work like this and make it faster, whereas the real solution is to eliminate the work entirely. If this actually saved his employer tens of thousands of dollars, then imagine how much could be saved by dumping the JSON filth entirely for something no less convenient and yet much more efficient.
I was able to speed up the C++ implementation by about 25% on my Linux machine simply by statically linking it. I think people frequently forget about the costs of dynamic linking. Maybe there are a few thousands more dollars to be saved annually!
That’s a very good point.
Sadly, I don’t think we can portably use static linkage. Works fine under Linux though!
Out of curiosity, I tried the obvious piece of jq, and got within a factor of 2 of the tightly specialized C++ code, so maybe the first trick to try is “pick a suitable command-line utility.”
$ hyperfine -N –runs 1000 “jq .user.wealth wealth.json” –warmup 100
Benchmark 1: jq .user.wealth wealth.json
Time (mean ± σ): 5.3 ms ± 1.1 ms [User: 3.3 ms, System: 1.6 ms]
Range (min … max): 2.4 ms … 7.9 ms 1000 runs
$ hyperfine -N –runs 1000 “./build/main wealth.json” –warmup 100
Benchmark 1: ./build/main wealth.json
Time (mean ± σ): 2.5 ms ± 0.6 ms [User: 1.1 ms, System: 1.1 ms]
Range (min … max): 1.0 ms … 4.3 ms 1000 runs