433
With all the talk about privacy, people seem to be forgetting that censorship is also a major problem for today's internet.
(lemmy.world)
Many "alternative" search engines are better for privacy, but they are still vulnerable to censorship, because they rely on g**gle and m*crosoft's indices for their search results. This isn't a deep-hidden secret either, many of them disclose what search index they use on the "about" page, for example:
- https://duckduckgo.com/duckduckgo-help-pages/results/sources/
- https://support.startpage.com/hc/en-us/articles/5138782571796-Why-isn-t-a-particular-site-appearing-in-the-results
- https://www.ecosia.org/privacy
There are still search engines that (claim to) maintain their own index. Most surprisingly, br*ve:
if you look at the repo they give thanks to:
"The commoncrawl organization for crawling the web and making the dataset readily available. Even though we have our own crawler now, commoncrawl has been a huge help in the early stages of development."
There is nothing I can find which says how much of the index is CC and how much is their own; if there's a decent amount of CC, this is originally for researchers etc. it's not the best resource in the world for a search index: https://commoncrawl.org/
That being said, as an independent search engine, it's always good to see people take on the massive task of actually building an index, not becoming a proxy.