My latest project requires a web crawler of sorts (technically, it just needs to grab the HTML and index it, not following any links), so fetching web pages very fast will be important. I’ve whipped up a quick script using stdlib’s Benchmark to pit several options against each other: open-uri, rio, net/http, and Zed Shaw’s RFuzz. (If I’m leaving any out, please let me know.)
The Test
Briefly, the test fetches the Google search results for testing 5 times each.
The Results
user system total real
RFuzz 0.010000 0.000000 0.010000 ( 2.486962)
net/http 0.040000 0.020000 0.060000 ( 2.880106)
open-uri 0.070000 0.040000 0.110000 ( 3.203510)
rio 0.140000 0.060000 0.200000 ( 3.674174)
Conclusion
It appears as though RFuzz cranks through HTTP the fastest, with net/http coming in a close second. It appears as through both open-uri’s and rio’s prettiness in syntax also slows them down quite a bit earning them 3rd and 4th place, respectively.
