Comment Re:Simple solution for Google & Facebook (Score 4, Informative) 168
Right. But isn't this was robots.txt is for? Perhaps we need to update the RFC to indicate that the page(s) are okay for search results, but not okay for aggregators? Seems like a simple fix that doesn't involve lawyers.
Actually there is no need for that, google allows multiple ways to block or limit the search results these news organizations can display already. Google has their different crawlers listed on their support pages and also provides examples of how to block specific crawlers from indexing their sites or limit what is displayed. In addition they also support blocking via meta tags and even http headers. So if the news organizations wanted to show up in search results but not in google news they could easily set the following on the web server:
X-Robots-Tag: Googlebot-News: noindex
And just like that the news stories would not be indexed at all, or if they did not want snippets but just a normal link they could replace noindex with nosnippet and they would have blocked news snippets being shown site wide. With the solution to their woes being so simple I am not sure why they have not done it yet.