The problem with search is that not only is Google getting worse, but I've also mostly outgrown it, in that it isn't sophisticated to answer pretty much any scientific question I would want to ask.
- No way to search for a scientific question and get a summary of the current scientific consensus or viewpoints on specific issues
- It's really hard to access academic journal articles online.
- Even when you can access journal articles, it's hard to know which ones to look in to answer your question. Sometimes it's hard to even know which field(s) your question falls under.
- Even if you vaguely know which field your question falls under, you don't necessarily know any of the vocabulary used by that field.
- No way to search by dependent and independent variables, confounding variables, etc.
- No way to sort articles by the quality of their methodology, the quality of the journal they were published in, the quality of the researchers, etc.
I know this isn't a product that more than 1% of the population would use, but if someone built it then maybe there are other things it could be used for.
You're talking about highly-focused, or micro, search. Yeah, Google doesn't seem to do that very well. They have a few segments, like book search and image search, but it's not specific enough.
One thing I search for sometimes are code examples in a particular language. Search for something in C on Google and you end up with lots of stuff for C++, C#, etc... Github, with its large repository of public code, lets you filter by programming language, and is much better than Google in these cases.
Bing copies Google. DuckDuckGo returns different things than Google, but otherwise is a copy. There's no micro search engine for specific topics and sub-topics, outside of site-specific search. Market opportunity...
We're trying to do something like this, i.e, let the user build more complex queries (than a free text search) for their specialization without having them write actual SQL ;) We've started off with Biotech - http://www.distilbio.com
> It's really hard to access academic journal articles online.
We obviously need something better than the status quo here, but the status quo isn't as bad as it seems if you know how it works.
Quick hints: email the article author. They'll probably more than happy to send you an article and a quick summary worded for a lay audience (not to mention talking your ear off about their more recent work...) They're not worried about you not paying the publishers, they want to spread their work around.
Another trick is knowing people in academia. Maybe you have a friend who's doing graduate work, or lecturing. You could ask your old lecturers if you went to university and if the paper you're after is in their field. There are also communities like /r/scholar on Reddit, though I imagine some people are against that sort of thing.
Actually the use case you described seems very ripe for disruption if you ask me. Because it's hacking your way to the solution, whereas we could need a better solution.
> The problem with search is that not only is Google getting worse, but I've also mostly outgrown it, in that it isn't sophisticated to answer pretty much any scientific question I would want to ask.
This simply means that Google doesn't work very well for you, and I mean no offense, but what you are searching is a very, very small minority of search queries. Google still serves a crushing majority of people very well.
You are making the same mistake that Paul is making throughout his essay: he wants startups to build products for him and not for regular users. Seriously, email is actually a todo list? Come on, now.
Google holds the elite back, holds science back, holds are collective knowledge back. Even if it is a minority of queries, these queries are more important than your typical query.
We're surely working all the time to make search "more sophisticated"; many special case queries are already smart (from "2+2" to geo, stocks etc., you don't need to go to special sites like calculator, maps and finance). And we surely have plans to go way beyond, but generally, this is Hard Stuff(TM). For things like journal articles, the information is often behind paywalls like ACM, and even when it's not, specialized engins like citeseer are hard to beat because the info has very special organization needs like collecting and measuring citations. On your most advanced requirements, I think only an Asimov's positronic robot would be that smart ;) unless there's a specific effort to curate this data... which requires tons of human labor, so ads served to the very small amount of people who needs this service will not make it viable. It's the same problem we have with patents (see http://arstechnica.com/tech-policy/news/2012/03/opinion-the-...).
No; these reasons show why the are is ripe for anyone to do it. The only fallacy in Paul Graham's comment (or in possible interpretations of it) is that Google has a weak spot there so it's an opportunity for somebody else to beat Google. Trust me, we have a ton of resources dedicated to improvements in search and we have lots of cool things coming down the pipe, although maybe not in the velocity that one could dream (e.g. something like intelligent research for scientific papers is firmly in the sci-fi realm today, at least for fully-automated computing).
BTW, Paul's article has a big #fail when he mentions code search as a possible idea; dude, we do have that and it's amazing (but unfortunately we recently shut it down; not sure if this will eventually resurface as part of some other product).
All that said, of course some company can always make an effort dedicated to a specialized niche that we are overlooking and beat Google Search in that niche; an excellent example of that is Wolfram Alpha. Still, not a big deal; to really "beat Google" you need a new general-purpose, full-Web search engine that beats Google's. Not impossible either, but the barrier to entry is simply colossal and it amazes me that people don't realize that and dream that it just takes some cool new idea or clever new algorithm to do that--we are not anymore in 1998, when Google, still working off a garage, started to beat the current top engines like Yahoo! and AltaVista.
> The problem with search is that not only is Google getting worse,
Google, today, after all is a stock-holders company that is aimed to generate PROFITS, and maximize those. It should be kind of obvious that at some point they (as a company) will try to maximize cash coming in, and minimize going out (spent). Therefore, their product [search] is narrowed towards the ones who push the most obvious questions/search queries: what do they play in theatres, what car to buy, best lcd tv, pharmacy near me, etc. Thats probably 90% of search queries they getting. I say, as long as they work in this zone and make sure simple queries return the best results, they are winning - winning biggest chunk of market and smile on shareholders' faces.
> - It's really hard to access academic journal articles online.
That's because of the business model around funded research. Academic research funding is driven by a limited number of funding agencies being bombarded by huge numbers of proposals. One of the key metrics they use is how many peer reviewed journals has the author been accepted by. Those journals make money by charging access fees and by being semi-trusted gate keepers. Journals WANT it to be hard to access them online since they view the Internet publishing paradigm as a threat.
People have been trying to disrupt that business model since the mid 90s.
I use it more as a way to shortcut sites. For example, if I want to Wikipedia "pi" instead of typing www.wikipedia.org, in the address bar, then typing in "pi" in the site's search bar, I enter it in Google and find the link. Firefox's awesome bar is gradually taking over as I can favorite things and "search" for them using that just by typing in a couple letters, but I still use Google for anything I haven't favorited.
I've been doing that for years on Opera. Just gotta right click a search bar, give it a few letters to ID it and off I go with 'w pi' to search wikipedia. I do not miss having to go through google first if I just use the search bar, or even going to wikipedia.org or wherever first.
I like Firefox's Keyword Search bookmarks. The Awesome Bar becomes a "web command line". Some example search bookmarks I've configured:
* "w pi" to search Wikipedia articles
* "d pi" to search Dictionary.com definitions
* "am pi" to search Amazon products
* "map pie" to search Google Maps locations
* "g pi" to search Google
I'm honestly surprised not every hacker does this. The vast majority of popular browsers supports keyword searching either out of the box or via a plugin.
This is something I've been thinking about seriously, building an "academic-level" search engine. I have the IR/NLP background. If anyone is interested discussing/collaborating, ping me!
- No way to search for a scientific question and get a summary of the current scientific consensus or viewpoints on specific issues
- It's really hard to access academic journal articles online.
- Even when you can access journal articles, it's hard to know which ones to look in to answer your question. Sometimes it's hard to even know which field(s) your question falls under.
- Even if you vaguely know which field your question falls under, you don't necessarily know any of the vocabulary used by that field.
- No way to search by dependent and independent variables, confounding variables, etc.
- No way to sort articles by the quality of their methodology, the quality of the journal they were published in, the quality of the researchers, etc.
I know this isn't a product that more than 1% of the population would use, but if someone built it then maybe there are other things it could be used for.