Translation: Crawling is the process in which automated programs (called crawlers or bots) visit web pages to collect information about their content and links. It is mainly used to discover and map web pages so they can be indexed by search engines. Crawling is also used to gather large datasets for training AI systems such as large language models (LLMs).