These measures still offer immediate protection. In any case, AI corporations can’t use what they’ll’t obtain, no matter how courts rule on copyright and fair use. However the effect is that giant web publishers, forums, and sites are sometimes raising the drawbridge to crawlers—even those who pose no threat. That is even the case once they ink lucrative deals with AI corporations that need to preserve exclusivity over that data. Ultimately, the online is being subdivided into territories where fewer crawlers are welcome.
How we stand to lose out
As this cat-and-mouse game accelerates, big players are inclined to outlast little ones. Large web sites and publishers will defend their content in court or negotiate contracts. And big tech corporations can afford to license large data sets or create powerful crawlers to bypass restrictions. But small creators, reminiscent of visual artists, YouTube educators, or bloggers, may feel they’ve only two options: hide their content behind logins and paywalls, or take it offline entirely. For real users, that is making it harder to access news articles, see content from their favorite creators, and navigate the online without hitting logins, subscription demands, and captchas each step of the way in which.
Perhaps more concerning is the way in which large, exclusive contracts with AI corporations are subdividing the online. Each deal raises the web site’s incentive to stay exclusive and block anyone else from accessing the info—competitor or not. It will likely result in further concentration of power within the hands of fewer AI developers and data publishers. A future where only large corporations can license or crawl critical web data would suppress competition and fail to serve real users or lots of the copyright holders.
Put simply, following this path will shrink the biodiversity of the online. Crawlers from academic researchers, journalists, and non-AI applications may increasingly be denied open access. Unless we are able to nurture an ecosystem with different rules for various data uses, we may find yourself with strict borders across the online, exacting a price on openness and transparency.Â
While this path is just not easily avoided, defenders of the open web can insist on laws, policies, and technical infrastructure that explicitly protect noncompeting uses of web data from exclusive contracts while still protecting data creators and publishers. These rights will not be at odds. Now we have a lot to lose or gain from the fight to get data access right across the web. As web sites look for methods to adapt, we mustn’t sacrifice the open web on the altar of economic AI.
.
