Python requests Tutorial for API Calls: From Basic to Advanced – ITFROMZERO

Introduction to APIs and Why Python requests is Important?

In IT, applications ‘talking’ to each other is a daily occurrence. Perhaps you need to retrieve data from an external service, send updates to another platform, or automate tasks via APIs. Python requests is the tool that helps you do just that, efficiently.

Initially, many programmers might consider urllib – Python’s built-in library. However, they quickly realize that urllib is quite complex and cumbersome for common HTTP tasks.

The problem is that urllib requires you to manually handle many low-level HTTP details, such as encoding parameters, managing headers, or decoding responses. This results in verbose and hard-to-read code, especially for beginners. That’s when the requests library becomes the optimal solution.

requests was created to simplify sending HTTP requests in Python. It cleverly hides the complexities of web communication, allowing you to focus on application logic rather than HTTP protocol details. In fact, requests has become the top choice for most Python programmers when interacting with APIs, thanks to its convenience and power.

Quick Start: Make API Calls in 5 Minutes

To make it easy to grasp, let’s start immediately with requests. Open your terminal and follow along.

Step 1: Install the requests library

Make sure you have Python installed. Then, use pip to install requests:


pip install requests

Step 2: Send a simple GET request

GET requests are used to retrieve data from a server. Let’s say I want to get a list of posts from a public API (e.g., JSONPlaceholder).


import requests

# Send a GET request to a public API
response = requests.get('https://jsonplaceholder.typicode.com/posts/1')

# Check the HTTP status code
if response.status_code == 200:
    # Print the response content as JSON
    print("GET successful!")
    print(response.json())
else:
    print(f"An error occurred: {response.status_code}")

Quite simple, isn’t it? With just a few lines of code, you’ve successfully retrieved data from an API.

Step 3: Send a POST request to create data

POST requests are typically used to send data to the server, for example, to create a new post.


import requests

url = 'https://jsonplaceholder.typicode.com/posts'
new_post = {
    "title": "My New Post",
    "body": "Interesting content about requests API.",
    "userId": 1
}

# Send POST request with JSON data
response = requests.post(url, json=new_post)

# Check and print the result
if response.status_code == 201: # 201 Created
    print("POST successful!")
    print(response.json())
else:
    print(f"An error occurred: {response.status_code}")

In this example, the argument json=new_post helps requests automatically encode the new_post dictionary into JSON and set the Content-Type: application/json header for you. Extremely convenient!

Detailed Explanation: Optimizing API Communication

Basic HTTP Methods

HTTP has several methods (verbs) to specify the type of action you want to perform on a resource:

GET: Retrieve data.
POST: Send new data to create a resource.
PUT: Update an entire existing resource.
PATCH: Partially update an existing resource.
DELETE: Delete a resource.

requests provides corresponding functions: requests.get(), requests.post(), requests.put(), requests.patch(), requests.delete().

Parameters and Headers

When sending a GET request, you often need to pass parameters in the URL. requests handles this elegantly by using the params argument.


import requests

# Example: get posts for userId = 1
params = {'userId': 1}
response = requests.get('https://jsonplaceholder.typicode.com/posts', params=params)
print("Posts for User ID 1:")
for post in response.json():
    print(f"- {post['title']}")

Headers contain important metadata about the request or response, such as content type or authentication tokens. You can easily add custom headers:


import requests

headers = {
    'User-Agent': 'My Python App v1.0',
    'Accept': 'application/json'
}
response = requests.get('https://jsonplaceholder.typicode.com/posts', headers=headers)
print("Response Headers:")
print(response.headers)

Handling Responses

The response object from requests is very powerful. Some important attributes:

response.status_code: HTTP status code (200 OK, 404 Not Found, 500 Internal Server Error, etc.).
response.text: The response content as a string.
response.json(): Converts JSON content into a Python object (dictionary/list). Will raise an error if the content is not valid JSON.
response.content: The response content as bytes, useful when working with binary files (images, videos).
response.headers: Dictionary containing the response headers.

Error and Exception Handling

In practice, APIs don’t always return the desired results. We need to check the status_code and handle error cases. requests also provides a very useful method called response.raise_for_status(). This method will automatically raise an HTTPError if the status_code indicates an error (from 4xx or 5xx), helping you quickly detect issues.


import requests

url_invalid = 'https://jsonplaceholder.typicode.com/non-existent-path'
url_valid = 'https://jsonplaceholder.typicode.com/posts/1'

try:
    response = requests.get(url_invalid)
    response.raise_for_status() # Will raise an error for 404 status code
    print("GET successful!")
except requests.exceptions.HTTPError as err:
    print(f"HTTP Error: {err}")
except requests.exceptions.ConnectionError as err:
    print(f"Connection Error: {err}")
except requests.exceptions.Timeout as err:
    print(f"Timeout Error: {err}")
except requests.exceptions.RequestException as err:
    print(f"General Error: {err}")

print("\n--- Trying with a valid URL ---")
try:
    response = requests.get(url_valid)
    response.raise_for_status()
    print("Valid GET successful!")
    print(response.json())
except requests.exceptions.RequestException as err:
    print(f"An error occurred: {err}")

Authentication

Many APIs require authentication for access. requests supports various methods:

Basic Authentication: Uses username and password.
Bearer Token (OAuth2, JWT-based Token): Sends the token in the Authorization header.


import requests

# Basic Auth
# response = requests.get('https://api.example.com/user', auth=('username', 'password'))

# Bearer Token (most common today)
token = 'your_super_secret_bearer_token'
headers_with_auth = {
    'Authorization': f'Bearer {token}',
    'Content-Type': 'application/json'
}

# Assuming an API requires a Bearer Token
# response = requests.get('https://api.example.com/data', headers=headers_with_auth)
# print(response.json())

print("The above are illustrative examples of Basic Auth and Bearer Token authentication.")

Advanced: Deeper Dive into requests

Sessions: When you need to maintain state

When interacting with an API multiple times, re-establishing TCP/IP connections and handling cookies and headers for each individual request can be time-consuming. requests.Session was created to effectively solve this problem.

A Session object automatically persists settings such as headers, cookies, and authentication information across requests. This is especially useful when you only need to log in once and then perform many subsequent tasks without re-authenticating.


import requests

# Create a Session
with requests.Session() as session:
    # General session configuration (e.g., add default headers)
    session.headers.update({
        'User-Agent': 'My Custom Python Client',
        'Accept-Language': 'en-US,en;q=0.5'
    })

    # Send the first request (e.g., login, get cookies)
    # login_data = {'username': 'test', 'password': 'password'}
    # login_response = session.post('https://api.example.com/login', json=login_data)
    # print(f"Login status: {login_response.status_code}")

    # Send subsequent requests. Cookies and headers are maintained.
    # data_response = session.get('https://api.example.com/protected_data')
    # print(f"Protected data status: {data_response.status_code}")
    # print("Current session cookies:", session.cookies.get_dict())

    # Example with a public API to illustrate the session mechanism
    print("\n--- Illustrating Session with a public API ---")
    resp1 = session.get('https://httpbin.org/cookies/set/sessioncookie/12345')
    print("Cookies after request 1:", session.cookies.get_dict())

    resp2 = session.get('https://httpbin.org/cookies')
    print("Cookies after request 2 (maintained from request 1):")
    print(resp2.json())

Using requests.Session() not only boosts performance but also makes your code cleaner. This is especially evident when sending multiple consecutive requests to the same domain.

Timeout: Avoiding Endless Waits

When calling an API, the server might respond slowly or not at all. Without a timeout, your application could hang indefinitely. Always set a timeout to control the waiting period:


import requests

try:
    # Wait for a maximum of 5 seconds for the server to respond
    response = requests.get('https://api.github.com/events', timeout=5)
    print("GET successful with timeout.")
except requests.exceptions.Timeout:
    print("Request timed out.")
except requests.exceptions.RequestException as e:
    print(f"Another error occurred: {e}")

Proxy: When you need anonymity or to bypass firewalls

If you need to send requests through a proxy, for example, to bypass a firewall or hide your IP address, requests offers excellent support:


import requests

proxies = {
    'http': 'http://10.10.1.10:3128',
    'https': 'http://10.10.1.10:1080',
}

# response = requests.get('http://example.org', proxies=proxies)
print("The code above configures a proxy for requests.")
print("Note: Change the proxy address to your actual address if needed.")

File Uploads

To upload files to an API, use the files argument. requests will automatically handle the Content-Type: multipart/form-data header.


import requests

# Create a dummy file for testing
with open('test_file.txt', 'w') as f:
    f.write('This is the content of the test file.')

url = 'https://httpbin.org/post' # An API to test POST requests

with open('test_file.txt', 'rb') as f:
    files = {'file': f} # 'file' is the field name the backend API expects
    response = requests.post(url, files=files)

    if response.status_code == 200:
        print("File upload successful!")
        print(response.json()['files'])
    else:
        print(f"Error uploading file: {response.status_code}")

Practical Tips: Lessons from Real Projects

Use Sessions Correctly

When writing a script to process 100K records, I learned that connection pooling is extremely important. The common approach is to call requests.get() or requests.post() for each record individually.

However, this creates and closes many HTTP connections, wasting resources and significantly slowing down processing. Instead, using requests.Session() throughout the 100K record processing reuses the TCP connection. This dramatically reduces latency and reduces the load on both the client and server.


import requests

def process_records_efficiently(record_ids):
    with requests.Session() as session:
        # Session can be configured once here (headers, auth, etc.)
        session.headers.update({
            'Authorization': 'Bearer my_token',
            'Content-Type': 'application/json'
        })

        for record_id in record_ids:
            try:
                # Use the session to send requests
                response = session.get(f'https://api.example.com/data/{record_id}', timeout=10)
                response.raise_for_status()
                print(f"Successfully processed record {record_id}.")
                # Process data from response.json()
            except requests.exceptions.RequestException as e:
                print(f"Error processing record {record_id}: {e}")

# Example usage (assuming 100K record_ids)
# large_record_list = range(1, 100001)
# process_records_efficiently(large_record_list)
print("This illustrates how to use Session to efficiently process a large number of records.")

Check SSL Certificate (SSL Certificate Verification)

By default, requests verifies SSL certificates for HTTPS requests. This is crucial for security. If you are working with self-hosted APIs or have self-signed certificates in a development environment, you can disable verification with verify=False. Absolutely do not do this in a production environment!


import requests

# response = requests.get('https://bad-ssl.example.com/', verify=True) # Default is True
# If you encounter an SSL error in a development environment and are sure about the source:
# response = requests.get('https://bad-ssl.example.com/', verify=False)
print("Note on SSL verification: Always keep True in a production environment.")

Handle Rate Limiting

Many APIs impose limits on the number of requests you can send within a certain period (rate limiting) to prevent abuse. If you send too many requests, the API will return a 429 Too Many Requests error. Respect this limit by adding a delay between requests or using a library like tenacity for smart retries.


import requests
import time

def fetch_data_with_rate_limit(urls):
    for url in urls:
        try:
            response = requests.get(url)
            if response.status_code == 429:
                print("Rate limit hit, waiting 60 seconds...")
                time.sleep(60) # Wait for a period
                # Then you can retry this request
                response = requests.get(url)
                response.raise_for_status()
            else:
                response.raise_for_status()
            print(f"Data from {url}: {response.json()['args']}") # Assuming httpbin.org/get API
        except requests.exceptions.RequestException as e:
            print(f"Error fetching data from {url}: {e}")
        time.sleep(1) # Wait 1 second between requests to avoid rate limit

# Example
# api_urls = ['https://httpbin.org/get?item=1', 'https://httpbin.org/get?item=2']
# fetch_data_with_rate_limit(api_urls)
print("Illustrates how to handle Rate Limiting using time.sleep.")

Logging Requests and Responses

During debugging or development, logging HTTP requests and responses is extremely necessary. You can configure Python logging to display detailed information from the requests library.


import requests
import logging

# Enable logging for the requests library
logging.basicConfig(level=logging.DEBUG)
#logging.getLogger("requests").setLevel(logging.DEBUG)
#logging.getLogger("urllib3").setLevel(logging.DEBUG)

# Send a request
requests.get('https://httpbin.org/get')
print("\nCheck console output to see requests logs.")

Conclusion

The Python requests library is truly an indispensable tool when working with APIs. It not only makes your code cleaner and more readable but also provides powerful features, effectively handling most web communication scenarios. Whether it’s basic API calls or advanced tasks like session management, error handling, authentication, and performance optimization, requests excels.

By applying the knowledge and experience I’ve shared, you can confidently build efficient and reliable Python applications that interact with web services. Start practicing now to master this excellent tool!