[requests] Interacting with Web Pages

1. What is HTTP and requests module?

HTTP stands for Hyper Text Transfer Protocol. WWW is about communication between web clients and servers. Communication between client computer and web servers is done by sending HTTP requests and receiving HTTP Response.

 

The requests module allows us to send HTTP requests using Python. The HTTP requests returns a Response object with all the response data(content, encoding, status, etc). Requests is one of the most popular Python libraries that is not included with Python.

 

2. How to use request module?

Step 1 : Import Library and get response object

import requests 

r = requests.get('https://xkcd.com/353/') 
print(r)

# <Response [200]>

 

Step 2 : Extract contents from response object

# Print html contents of page 
print(r.text) 

# Get image on page based on bytes object
r_img = requests.get('https://imgs.xkcd.com/comics/python.png') 
print(r_img.content)

 

Step 3 : Write contents of page into file 

import requests 

r_img = requests.get('https://imgs.xkcd.com/comics/python.png')
with open('comic.png', 'wb') as f: 
    f.write(r_img.content)

 

3. Useful attributes of requests module

• respones.status_code : We can check the response statust code from response.status_code attribute. If we made a bad requests (a 4XX client error and 5XX sever error response), we can raise it with response.raise_for_status().
• response.raise_for_status() : Raise error if not 2XX ok code.
• response.ok : Return True if status of response object is under 4XX client error.
• response.headers : We can view the server's response headers using a Python dictionary.
• response.json() : If the type of response is json, convert to dictionary type.

 

4. Response object with parameters

4.1 Get request

We can request complicated http url with params keyword.

 

payload = {'page' : 2, 'count' : 25} 
r = requests.get('https://httpbin.org/get', params=payload) 
print(r.url)

# https://httpbin.org/get?page=2&count=25

 

4.2 Post request

payload = {'username' : 'corey', 'password' : 'testing'}
r = requests.post('https://httpbin.org/post', data=payload)
print(r.url) 

# https://httpbin.org/post

 

4.3 Difference between GET and POST

The GET method is a method used to request information from any resouce from the client to the server. More simply, it's a method used to read or retrieve data.

 

The POST method is used to send data to the server to create/update resources. Unlike Get, the data to be transmitted is contained in the body of the HTTP message and transmitted.

 

 

 

Source from :

• https://www.w3schools.com/whatis/whatis_http.asp   
• https://www.youtube.com/watch?v=tb8gHvYlCFs&list=PL-osiE80TeTskrapNbzXhwoFUiLCjGgY7&index=23 
• http://httpbin.org/#/