Web scraping for AI
In short
Web scraping for AI is the automated collection of online pages and files so they can be used in an AI dataset. A crawler can find and download pages. Later curation may filter, clean or compile what was collected.
- Also called
- AI web scraping, Web crawling for AI, Data scraping for AI
- Topic
- Safety, ethics and risk
- Level
- Beginner
The US Copyright Office describes one common way of building training data as downloading material that is publicly available on the internet. That includes automated tools which systematically scrape online sources, and text taken by web crawlers. Raw data then usually goes through curation. The Office gives filtering, cleaning and compiling as examples, and names deduplication as one reason to filter. It also says that 'publicly available' is not the same as authorised.1
Collecting pages is not the same step as training a model. The Office treats acquisition, training and later deployment as different stages, and says copies made while collecting a dataset can implicate a copyright owner's exclusive rights.1 The same pages may instead feed a search product. OpenAI's GPTBot crawls content that may be used to train its generative models, while OAI-SearchBot is used to surface sites in ChatGPT search.2 If the material includes personal data and is used to train, test or deploy an AI system, the UK Information Commissioner's Office says data protection law can apply.3
Site owners can publish instructions in a file named robots.txt. RFC 9309, the Robots Exclusion Protocol, says crawlers are requested to honour those rules when they access addresses. It also says the rules are not a form of access authorisation. The protocol is not a substitute for a security control. The RFC gives HTTP authentication as the example.4
Companies then add their own tokens. Google-Extended is not a separate crawler. Google says the token lets a publisher manage whether content already crawled may be used to train future Gemini models and for grounding, and that it does not affect inclusion or ranking in Google Search.5 OpenAI says disallowing GPTBot indicates that the content should not be used to train its generative models, and that this setting is independent of the search bot.2
Example
A publisher disallows one named crawler in robots.txt. RFC 9309 says a crawler is requested to skip the blocked paths, but the file does not itself decide whether copying the pages for training is lawful. That copyright question is the one the Copyright Office analyses for scraped datasets.41
Why it matters
Scraping is a main route from the public web into a training set. The Copyright Office says those downloads can copy protected works, and a robots.txt rule is only a request to the crawler.14
Questions people ask
Is web scraping for AI legal?
There is no single answer. The Copyright Office says copies made to build a training set can implicate a copyright owner's rights, and that material being publicly available is not the same as being authorised. If personal data is used to train, test or deploy a system, the UK Information Commissioner's Office says data protection law can apply.13
Does robots.txt stop AI scraping?
It asks compliant crawlers not to fetch the listed paths. RFC 9309 says those rules are not access authorisation, and that they are not a substitute for a security measure such as authentication. A disallow line works only if the crawler honours it.4
What is the difference between scraping and crawling?
RFC 9309 describes crawlers as automated clients, such as the clients search engines use to follow links. The Copyright Office describes scraping as automated downloading from online sources, including text collected by web crawlers, before filtering and cleaning. An AI dataset pipeline often does both.41