Fake Oakley 'Hidage' 'Amount due'
Cheap Knock Off Oakleys Sunglasses Oakley Sunglasses Sale
I'm sharing the slides in this entry.
It was an introductory talk on XML for PHP developers. There are hundreds of technologies built on top of XML. We have all heard about RSS, Atom, XML-RPC, SOAP, etc. The goal of the talk was to get PHP developers to start using XML. In the talk, I presented three recipes:
A while ago, we discussed how to scrape information from websites that don't offer information in a structured format like XML or JSON. We noted that urllib and lxml are indispensable tools in web scraping. While urllib enables us to connect to websites and retrieve information, lxml helps convert HTML, broken or not, to valid XML and parse it. In this post, I will demonstrate how to retrieve information from web pages that require a login session.
I have created a sample website for this task - http://toscrape.techchorus.net
The website has a page that requires a login session - http://toscrape.techchorus.net/only_authenticated.php
To login to the sample website, use the credentials:
If you visit http://toscrape.techchorus.net/ you will notice that the server sends the response with headers:
Date: Tue, 19 Oct 2010 17:33:43 GMT
Server: Apache mod_fcgid/2.3.5 mod_auth_passthrough/2.1 mod_bwlimited/1.4 FrontPage/22.214.171.12435
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Set-Cookie: PHPSESSID=c84456d65e5b9da95b09abd4092f860b; path=/
Keep-Alive: timeout=5, max=100
In order to maintain the session, you have to send the cookie PHPSESSID with the value c84456d65e5b9da95b09abd4092f860b in all subsequent requests. Of course, the value varies for each user. If the server resets the value of the cookie in a subsequent response, you have to send the updated cookie value in further requests.
Python standard library offers the module cookielib to manage cookies in the client side. We can use this as a cookie jar. In essence, cookielib offers a container to hold cookies. We use this container in urllib2.
Let's start writing the code.
More and more websites are offering APIs nowadays. Previously, we've talked about XML-RPC and REST. Even though web services are growing exponentially there are a lot of websites out there that offer information in unstructured format. Especially, the government websites. If you want to consume information from those websites, web scraping is your only choice.
Web scraping is a technique used in programs that mimic a human browsing the website. In order to scrape a website in your programs you need tools to