Can you imagine if your job was to trawl competitor websites and jot prices down by hand, again and again and again? You’d burn your whole office down by lunchtime.
So, little wonder web crawlers are huge these days. They can keep track of customer sentiment and trending topics, monitor job openings, real estate transactions, UFC results, all sorts of stuff.
For those of a certain bent, this is fascinating stuff. Which is how I found myself playing around with Scrapy, an open source web crawling framework written in Python.
Being wary of the potential to do something catastrophic to my computer while poking with things I didn’t understand, I decided to install it on my main machine but a Raspberry Pi.
And wouldn’t you know it? It actually didn’t run too shabby on the little tacker. Maybe this is a good use case for an ARM server?
Google had no solid answer. The nearest thing I found was this Drupal hosting drag race, which showed an ARM server outperforming a much more expensive x86 based account.
That was definitely interesting. I mean, isn’t a web server kind of like a crawler in reverse? But with one operating on a LAMP stack and the other on a Python interpreter, it’s hardly the exact same thing.
So what could I do? Only one thing. Get some VPS accounts and make them race each other.Read more “SPEED TEST: x86 vs. ARM for Web Crawling in Python”