I noticed this morning in the stats someone arriving here looking for “scroogle got shut down 2012″ (btw, thanks for not masking your DDG query, see it can be useful ). Indeed Scroogle already had some scraping issues in the past (hence the 2012 in the query), regularly being blocked for short periods of time. But this time, it seems more serious: it has really frequently been displaying a message about not being able to search Google during the last weeks, and today it even seems totally offline.
Edit: if you’re just here wondering what happened to Scroogle, feel free to join the fun.
Update (2012-02-21): just found this post, “Scroogle May Have Been a Victim of Hackers, Not Google”, about what might have happened to Scroogle (see also a post by Daniel Brandt about that). Apparently, they were DDoSed in December 2011, and as a result one of their 2 hosts terminated 4 of their servers, leaving them only 2 servers at their other host, which is why they were often blocked by Google in January/early February (too many queries from same IP = auto-block by Google anti-bot protection). I guess it’s a possibility that those remaining 2 servers were DDoSed too recently, although at the moment that’s only pure speculation.
So without further ado, “the kind is dead, long live the king”. Here are 4 other options:
- Duckduckgo. It’s not an equivalent to Scroogle, but I put it at the top of the list precisely for this reason: with this one you don’t use Google at all. I posted a bit more details about its strengths and weaknesses in a previous post.
- Privatelee. This one seems even like a superior equivalent to Scroogle. It scrapes results from both Google and Bing, plus has a big missing feature in Scroogle: image search, yay! The only downside is that I didn’t find a way to configure it to search only on Google or only on Bing. There are checkboxes that seem to allow configuring that, but they are disabled… Still, that seems like a neat alternative.
- GoogleSharing. This one seems a bit more complicated/intrusive, and seems to be Firefox-only (but anyway, you don’t use Scroogle with Chrome, do you?). I didn’t try it yet (I’m rather on DDG, actually), but I really like their motto : “Who knows more about the citizens in their own country, Kim Jong-Il or Google?”. Basically, they provide a Firefox extension that will then redirect your Google traffic through their proxies (NB: they also provide the proxy’s source code, so that anyone can run a proxy). The Google services proxied by the extension are configurable, however the following services can’t be proxied (for privacy reasons I guess): Mail, Checkout, Health, Sites, Docs, and Reader.
- Startpage. Weird name for a search engine (or rather a search proxy). But it seems to do the same as Privatelee. Except for the image results, which completely differed from the Google Image results on the query I tested. Note that Startpage and Ixquick seem to be 2 aliases of the very same service.
That’s it. My main source is this thread, Google is Blocking Scroogle – says they are a Bot, which was started a month ago, about the time when Scroogle indeed became regularly blocked by Google. I mainly compiled the suggestions posted there.
Edit (2012-03-27): I also just found out about another alternative: Stealth, which claim to scrape from Google, Bing and Yahoo + also do some crawling of their own.