More and more websites are offering APIs nowadays. Previously, we've talked about XML-RPC and REST. Even though web services are growing exponentially there are a lot of websites out there that offer information in unstructured format. Especially, the government websites. If you want to consume information from those websites, web scraping is your only choice.
What is web scraping?
Web scraping is a technique used in programs that mimic a human browsing the website. In order to scrape a website in your programs you need tools to
- Make HTTP requests to websites
- Parse the HTTP response and extract content