Web scraping — also called screen scraping, web data extraction, or web harvesting — is the automated extraction of data from websites, fetching pages and parsing their content into structured records.
Mechanically, a scraper requests pages the way a browser would, parses the returned markup, and writes selected fields to a file or database. Simple targets need only an HTTP client and an HTML parser. Sites that assemble their content in the browser require a headless browser that executes JavaScript before anything can be read. Where a site publishes an API, using it is faster, more stable, and less adversarial than scraping the rendered page — scraping is what you do when no such door exists.
The applications are broad: price and inventory monitoring, research corpora, lead generation, search indexing, and training data for machine learning. So are the countermeasures — rate limiting, bot detection, CAPTCHAs, and markup that changes shape specifically to break parsers.
Legality is jurisdictional and genuinely unsettled. In the United States, hiQ v. LinkedIn established that scraping publicly accessible data is not unauthorized access under the Computer Fraud and Abuse Act, but terms of service, copyright, and data-protection regimes like GDPR all still apply independently. The workable ethic is narrower than the law: respect 📝robots.txt, rate-limit so the target does not absorb your cost, and do not collect personal data you could not justify holding if asked.