Python HTTP Client Request – Essential Guide
Python HTTP module defines the classes which provide the client-side of the HTTP and HTTPS protocols. In most of the programs, the HTTP module is not directly used and is clubbed with the urllib module to handle URL connections and interaction with HTTP requests. Today we will learn how to use a Python HTTP client to fire HTTP request and then parse response status and get response body data.
Python HTTP Client
In this post on python HTTP module, we will try attempting making connections and making HTTP requests like GET, POST and PUT. Let’s get started.
Making HTTP Connections
We will start with the simplest thing HTTP module can do. We can easily make HTTP connections using this module. Here is a sample program:
import http.client
connection = http.client.HTTPConnection('www.python.org', 80, timeout=10)
print(connection)
Python HTTP GET
Now, we will use HTTP client to get a response and a status from a URL. Let’s look at a code snippet:
import http.client
connection = http.client.HTTPSConnection("www.journaldev.com")
connection.request("GET", "/")
response = connection.getresponse()
print("Status: {} and reason: {}".format(response.status, response.reason))
connection.close()
In above script, we used a URL and checked the status with the connection object.
Remember to close a connection once you’re done with the connection object. Also, notice that we used a HTTPSConnection to establish the connection as the website is served over HTTPS protocol.
Getting SSL: CERTIFICATE_VERIFY_FAILED Error?
When I first executed above program, I got following error related to SSL certificates.
$ python3.6 http_client.py
Traceback (most recent call last):
File "http_client.py", line 4, in
connection.request("GET", "/")
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1239, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1285, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1234, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1026, in _send_output
self.send(msg)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 964, in send
self.connect()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1400, in connect
server_hostname=server_hostname)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 401, in wrap_socket
context=self, session=session)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 808, in init
self.do_handshake()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 1061, in do_handshake
self._sslobj.do_handshake()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 683, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:748)
$
From the output, it was clear that it has to do something with the SSL certificates. But website certificate is fine, so it has to be something with my setup. After some googling, I found that on MacOS, we need to run Install Certificates.command file present in the Python installation directory to fix this issue.
Getting the Header list from Response
From the response we receive, the headers usually also contain important information about the type of data sent back from the server and the response status as well. We can get a list of headers from the response object itself. Let’s look at a code snippet which is a little-modified version of the last program:
import http.client
import pprint
connection = http.client.HTTPSConnection("www.journaldev.com")
connection.request("GET", "/")
response = connection.getresponse()
headers = response.getheaders()
pp = pprint.PrettyPrinter(indent=4)
pp.pprint("Headers: {}".format(headers))
Python HTTP POST
We can POST data to a URL as well with the HTTP module and get a response back. Here is a sample program:
import http.client
import json
conn = http.client.HTTPSConnection('www.httpbin.org')
headers = {'Content-type': 'application/json'}
foo = {'text': 'Hello HTTP #1 **cool**, and #1!'}
json_data = json.dumps(foo)
conn.request('POST', '/post', json_data, headers)
response = conn.getresponse()
print(response.read().decode())
Python HTTP PUT Request
Of course, we can also perform a PUT request using the HTTP module itself. We will use the last program itself. Let’s look at a code snippet:
import http.client
import json
conn = http.client.HTTPSConnection('www.httpbin.org')
headers = {'Content-type': 'application/json'}
foo = {'text': 'Hello HTTP #1 **cool**, and #1!'}
json_data = json.dumps(foo)
conn.request("PUT", "/put", json_data)
response = conn.getresponse()
print(response.status, response.reason)
Conclusion
In this lesson, we studied simple HTTP operations which can be done using http.client. We can also create python http server using SimpleHTTPServer module. Python HTTP Client Request – Essential Guide
