Post

23. Working with APIs and HTTP Requests

🌐 Master Python API interactions with the requests library! Learn GET/POST requests, error handling, authentication, and build real-world API clients. ✨

23. Working with APIs and HTTP Requests

What we will learn in this post?

  • πŸ‘‰ Introduction to APIs and HTTP
  • πŸ‘‰ The requests Library
  • πŸ‘‰ Making GET Requests
  • πŸ‘‰ Making POST Requests
  • πŸ‘‰ Handling Errors and Exceptions
  • πŸ‘‰ Authentication and Headers
  • πŸ‘‰ Building a Simple API Client

Welcome to the World of APIs! πŸ‘‹

Ever wondered how apps talk to each other? That’s where APIs come in!

What are APIs? 🀝

Imagine an API (Application Programming Interface) like a friendly waiter in a restaurant. You tell them what you want (a β€œrequest”), and they go to the kitchen (another program/server) to get it for you, then bring back your order (a β€œresponse”). It’s a set of rules enabling different software to communicate!

RESTful APIs & HTTP Methods 🌐

RESTful APIs are very popular APIs that use standard web communication (HTTP). Think of HTTP methods as actions you can take:

  • GET: Fetch data (like asking for a menu).
  • POST: Create new data (like ordering food).
  • PUT: Update existing data (like changing your order).
  • DELETE: Remove data (like canceling an order).

Status Codes & JSON 🚦

After your request, you get a status code (the waiter’s reply):

  • 200 OK: Success! πŸŽ‰ Your request worked.
  • 404 Not Found: Oops, what you asked for isn’t there.
  • 500 Internal Server Error: Something went wrong on the server’s side. Data is often sent back in JSON (JavaScript Object Notation) – a super-easy-to-read format, like a structured shopping list for computers.

Let’s See an Example! πŸ§‘β€πŸ’»

Here’s how you’d GET data using Python:

1
2
3
4
5
6
7
8
import requests # A popular library for making web requests

# This sends a GET request to a public API to fetch a 'todo' item
response = requests.get('https://jsonplaceholder.typicode.com/todos/1') 

print(f"Status Code: {response.status_code}") # Checks if the request was successful
print(f"Data Received: {response.json()}")    # Prints the data in JSON format
# Output: Status Code: 200, Data Received: {'userId': 1, 'id': 1, 'title': 'delectus aut autem', 'completed': False}
πŸš€ Try this Live β†’ Click to open interactive PYTHON playground

How API Communication Works πŸ’¬

graph LR
    A["πŸ§‘β€πŸ’» Your Application"]:::style1 -- "1. HTTP Request" --> B["βš™οΈ API Server"]:::style2
    B -- "2. HTTP Response" --> A

    classDef style1 fill:#ff4f81,stroke:#c43e3e,color:#fff,font-size:16px,stroke-width:3px,rx:14,shadow:6px;
    classDef style2 fill:#6b5bff,stroke:#4a3f6b,color:#fff,font-size:16px,stroke-width:3px,rx:14,shadow:6px;

    linkStyle default stroke:#e67e22,stroke-width:3px;

The requests Library: Your API Swiss Army Knife πŸ”§

Python’s requests library is the go-to tool for making HTTP requests. It’s elegant, powerful, and widely used in production systems worldwide.

Installing and Importing πŸ“¦

First, install requests using pip:

1
2
3
4
5
# Install from command line
pip install requests

# Or using pip3
pip3 install requests

Then import it in your Python code:

1
2
3
4
import requests

# Check the version
print(requests.__version__)  # e.g., '2.31.0'

Why Use requests Over urllib? πŸ€”

The built-in urllib works but is verbose. The requests library offers:

  • Simpler syntax - Less code for the same task
  • Automatic JSON parsing - .json() method does the work
  • Session management - Persistent connections and cookies
  • Better error handling - Clear exception types
  • File uploads - Easy multipart/form-data handling

Quick Comparison πŸ†š

1
2
3
4
5
6
7
8
9
10
11
12
13
14
# Using urllib (built-in, more complex)
import urllib.request
import json

req = urllib.request.Request('https://api.github.com/users/python')
with urllib.request.urlopen(req) as response:
    data = json.loads(response.read().decode())
    print(data['name'])

# Using requests (cleaner, more intuitive)
import requests

response = requests.get('https://api.github.com/users/python')
print(response.json()['name'])

The requests library makes API interactions feel natural and Pythonic!


What are GET Requests? 🌐

Imagine asking a website for information – that’s often a GET request! It’s how your browser fetches a webpage or an app retrieves data without changing anything on the server. Think of it as β€œread-only” – you’re simply getting data.

The Request Part: Asking for Info πŸ“€

When you send a GET request, you might include:

  • Parameters

    These are details you add to the URL after a ? to specify what you want. For example, in api.example.com/items?_limit=10&page=1, _limit=10 and page=1 are parameters (key-value pairs).

  • Headers

    Like an envelope for your letter, headers provide extra info about your request for the server. This could be what kind of data you prefer (Accept: application/json) or authentication tokens. They aren’t visible in the URL.

graph TD
    A["πŸ“± Client"]:::style1 -->|"πŸ” GET Request"| B["πŸ–₯️ Server"]:::style2
    B -->|"πŸ“¦ Response"| A

    classDef style1 fill:#43e97b,stroke:#38f9d7,color:#fff,font-size:16px,stroke-width:3px,rx:14,shadow:6px;
    classDef style2 fill:#6b5bff,stroke:#4a3f6b,color:#fff,font-size:16px,stroke-width:3px,rx:14,shadow:6px;

    linkStyle default stroke:#e67e22,stroke-width:3px;

The Response Part: Getting Data Back πŸŽ‰

When the server replies, you get a response object containing everything sent back:

  • response.text: This gives you the raw content as a string, often HTML or plain JSON.
  • response.json(): If the response is JSON, this magically converts it into a Python dictionary or list, making it super easy to use!
  • response.status_code: A crucial number telling you the request’s outcome. 200 means β€œOK!” βœ…, 404 means β€œNot Found” 🚫.

Let’s See It in Action! πŸš€

Using Python’s friendly requests library with the Star Wars API:

1
2
3
4
5
6
7
8
9
10
11
12
13
import requests

url = "https://swapi.dev/api/people/1/" # Asking for Luke Skywalker!
params = {"format": "json"} # Optional: ensuring we get JSON
headers = {"Accept": "application/json"} # Requesting JSON data

response = requests.get(url, params=params, headers=headers)

print(f"Status Code: {response.status_code}") # E.g., 200
if response.status_code == 200:
    data = response.json()
    print(f"Character Name: {data['name']}") # Accessing the name!
    # print(f"Raw Content Sample: {response.text[:50]}...")
πŸš€ Try this Live β†’ Click to open interactive PYTHON playground

Here, we made a GET request, sent parameters and headers, and then easily accessed the structured data!

Sending Data with POST Requests! πŸš€

Ever need to send information to a website or API, like submitting a form or creating a new user? That’s where POST requests shine! Unlike GET (which fetches data), POST is designed to securely transmit data to the server for processing. Think of it as mailing a package – you’re sending something new!

Packing Your Data πŸ“¦

When sending data with POST, you typically use one of two main ways to β€œpack” it:

  • data parameter: Great for traditional HTML form submissions (e.g., username=alice&password=123). This sends data as application/x-www-form-urlencoded or multipart/form-data.
  • json parameter: Perfect for sending structured data, especially common with APIs. Your data is sent as application/json (e.g., {"name": "Alice", "age": 30}). Libraries like Python’s requests automatically convert your dictionary to JSON!

Adding Important Details: Headers & Auth πŸ”‘

  • Headers: These are like labels on your package, telling the server more about what you’re sending.
    • Content-Type: Crucial! It tells the server if you’re sending application/json or application/x-www-form-urlencoded.
    • Authorization: Often used for authentication. You might include a token (e.g., Bearer YOUR_TOKEN) here to prove you have permission to send data.
graph TD
    A["πŸ“± Client"]:::style1 -->|"πŸ“€ POST Request"| B{"πŸ–₯️ Server"}:::style2
    B -->|"πŸ” Validate"| C{"βœ… Valid?"}:::style3
    C -- "Yes" --> D["πŸ’Ύ Save Data"]:::style4
    C -- "No" --> E["❌ Return Error"]:::style5
    D -->|"200 OK"| A
    E -->|"400 Error"| A

    classDef style1 fill:#43e97b,stroke:#38f9d7,color:#fff,font-size:16px,stroke-width:3px,rx:14,shadow:6px;
    classDef style2 fill:#6b5bff,stroke:#4a3f6b,color:#fff,font-size:16px,stroke-width:3px,rx:14,shadow:6px;
    classDef style3 fill:#ffd700,stroke:#d99120,color:#222,font-size:16px,stroke-width:3px,rx:14,shadow:6px;
    classDef style4 fill:#00bfae,stroke:#005f99,color:#fff,font-size:16px,stroke-width:3px,rx:14,shadow:6px;
    classDef style5 fill:#ff9800,stroke:#f57c00,color:#fff,font-size:16px,stroke-width:3px,rx:14,shadow:6px;

    linkStyle default stroke:#e67e22,stroke-width:3px;

Real-World Example: Creating a User πŸ‘€

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
import requests

# API endpoint for creating users
url = "https://jsonplaceholder.typicode.com/users"

# User data to send
user_data = {
    "name": "Alice Johnson",
    "email": "alice@example.com",
    "username": "alice_dev"
}

# Send POST request with JSON data
response = requests.post(url, json=user_data)

if response.status_code == 201:  # 201 Created
    created_user = response.json()
    print(f"βœ… User created with ID: {created_user.get('id')}")
else:
    print(f"❌ Failed with status: {response.status_code}")
πŸš€ Try this Live β†’ Click to open interactive PYTHON playground

Handling requests Errors Like a Pro! 🚧

Making web requests is super useful, but sometimes things don’t go as planned! Learning to gracefully handle errors makes your code robust and user-friendly. Let’s see how requests helps us!

Checking Status Codes & raise_for_status() ✨

When you make a request, the server sends back a status code. 200 OK means everything’s perfect, while 4xx (like 404 Not Found) or 5xx (server errors) mean trouble. The easiest way to catch these bad codes is response.raise_for_status(). It automatically raises an HTTPError if the status isn’t 200-299!

1
2
3
4
5
6
7
8
import requests

try:
    response = requests.get('https://httpbin.org/status/404') # This URL will return a 404
    response.raise_for_status() # This line will raise an HTTPError!
    print("Success!") # This won't print if there's an error
except requests.exceptions.HTTPError as e:
    print(f"Oops! HTTP Error occurred: {e}") # Catches errors like 404, 500
graph TD
    A["πŸš€ Make Request"]:::style1 --> B{"πŸ” Check Status"}:::style2
    B -- "βœ… 2xx Success" --> C["πŸ“Š Process Data"]:::style3
    B -- "❌ Error Code" --> D["⚠️ Raise HTTPError"]:::style4

    classDef style1 fill:#43e97b,stroke:#38f9d7,color:#fff,font-size:16px,stroke-width:3px,rx:14,shadow:6px;
    classDef style2 fill:#ffd700,stroke:#d99120,color:#222,font-size:16px,stroke-width:3px,rx:14,shadow:6px;
    classDef style3 fill:#6b5bff,stroke:#4a3f6b,color:#fff,font-size:16px,stroke-width:3px,rx:14,shadow:6px;
    classDef style4 fill:#ff9800,stroke:#f57c00,color:#fff,font-size:16px,stroke-width:3px,rx:14,shadow:6px;

    linkStyle default stroke:#e67e22,stroke-width:3px;

Dealing with Network Hiccups! πŸ”Œ

Timeouts ⏳

Sometimes, a server just takes too long. You can set a timeout limit. If the server doesn’t respond in time, a Timeout exception is raised.

1
2
3
4
try:
    requests.get('https://httpbin.org/delay/5', timeout=2) # This will wait 5s, but we only allow 2s
except requests.exceptions.Timeout:
    print("Request timed out after 2 seconds!") # Catches when the server takes too long

Connection & Other Errors 🚫

What if there’s no internet connection, or the domain doesn’t even exist? These are often ConnectionError or other RequestException types. A broad try-except block is your best friend here!

1
2
3
4
5
6
try:
    requests.get('http://nonexistent-domain-12345.com') # This URL won't resolve
except requests.exceptions.ConnectionError:
    print("Could not connect to the server! Check your internet or URL.")
except requests.exceptions.RequestException as e:
    print(f"An unexpected error occurred: {e}") # Catches any other requests-related issue
πŸš€ Try this Live β†’ Click to open interactive PYTHON playground

Authentication and Headers: The Keys to the Kingdom πŸ”

Most real-world APIs require authentication to verify who’s making the request. Headers carry this authentication data securely!

Common Authentication Methods πŸ”‘

1. API Keys

The simplest form – you get a unique key and include it in your request:

1
2
3
4
5
6
7
8
9
10
11
12
import requests

api_key = "your_secret_api_key_here"

# Method 1: As a query parameter
response = requests.get('https://api.example.com/data', params={'api_key': api_key})

# Method 2: As a header (more secure)
headers = {'X-API-Key': api_key}
response = requests.get('https://api.example.com/data', headers=headers)

print(response.json())

2. Bearer Tokens (OAuth) 🎫

Used by modern APIs like GitHub, Twitter, and Google. You get a token after authentication:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
import requests

access_token = "ghp_your_github_personal_access_token"

headers = {
    'Authorization': f'Bearer {access_token}',
    'Accept': 'application/vnd.github.v3+json'
}

# Get your GitHub user info
response = requests.get('https://api.github.com/user', headers=headers)

if response.status_code == 200:
    user_data = response.json()
    print(f"Hello, {user_data['login']}! πŸ‘‹")
πŸš€ Try this Live β†’ Click to open interactive PYTHON playground

3. Basic Authentication πŸ”’

Username and password encoded in Base64:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
import requests
from requests.auth import HTTPBasicAuth

# Method 1: Using auth parameter
response = requests.get(
    'https://api.example.com/protected',
    auth=HTTPBasicAuth('username', 'password')
)

# Method 2: Shorthand tuple
response = requests.get(
    'https://api.example.com/protected',
    auth=('username', 'password')
)

Custom Headers for API Requests πŸ“‹

Headers provide metadata about your request:

1
2
3
4
5
6
7
8
9
10
11
import requests

headers = {
    'User-Agent': 'MyApp/1.0',              # Identify your application
    'Accept': 'application/json',            # Preferred response format
    'Content-Type': 'application/json',      # Data format you're sending
    'Accept-Language': 'en-US',              # Language preference
    'Cache-Control': 'no-cache'              # Cache behavior
}

response = requests.get('https://api.example.com/data', headers=headers)

Real-World Example: Weather API Client β›…

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
import requests
import os

class WeatherAPI:
    def __init__(self, api_key):
        self.base_url = "https://api.openweathermap.org/data/2.5"
        self.headers = {
            'User-Agent': 'PythonWeatherApp/1.0'
        }
        self.api_key = api_key
    
    def get_current_weather(self, city):
        """Fetch current weather for a city"""
        endpoint = f"{self.base_url}/weather"
        params = {
            'q': city,
            'appid': self.api_key,
            'units': 'metric'  # Celsius
        }
        
        try:
            response = requests.get(endpoint, params=params, headers=self.headers)
            response.raise_for_status()
            return response.json()
        except requests.exceptions.HTTPError as e:
            print(f"HTTP Error: {e}")
            return None
        except requests.exceptions.RequestException as e:
            print(f"Error: {e}")
            return None

# Usage
api = WeatherAPI(api_key="your_openweathermap_api_key")
weather = api.get_current_weather("London")

if weather:
    print(f"🌑️ Temperature: {weather['main']['temp']}°C")
    print(f"☁️ Conditions: {weather['weather'][0]['description']}")

Building a Simple API Client πŸ—οΈ

Let’s combine everything we’ve learned to build a production-ready API client!

Design Principles for API Clients 🎯

  1. Encapsulation: Wrap API logic in a class
  2. Error Handling: Gracefully handle all failure modes
  3. Session Management: Reuse connections for efficiency
  4. Rate Limiting: Respect API limits
  5. Logging: Track requests for debugging

Complete GitHub API Client Example πŸ™

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
import requests
from typing import Optional, Dict, List
import logging

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

class GitHubAPIClient:
    """Production-ready GitHub API client"""
    
    def __init__(self, token: Optional[str] = None):
        self.base_url = "https://api.github.com"
        self.session = requests.Session()  # Reuse connections
        
        # Set default headers
        self.session.headers.update({
            'Accept': 'application/vnd.github.v3+json',
            'User-Agent': 'Python-GitHub-Client/1.0'
        })
        
        # Add authentication if token provided
        if token:
            self.session.headers['Authorization'] = f'Bearer {token}'
    
    def get_user(self, username: str) -> Optional[Dict]:
        """Fetch user information"""
        url = f"{self.base_url}/users/{username}"
        
        try:
            response = self.session.get(url, timeout=10)
            response.raise_for_status()
            logger.info(f"βœ… Fetched user: {username}")
            return response.json()
        
        except requests.exceptions.HTTPError as e:
            if response.status_code == 404:
                logger.error(f"❌ User not found: {username}")
            else:
                logger.error(f"❌ HTTP Error: {e}")
            return None
        
        except requests.exceptions.Timeout:
            logger.error("⏱️ Request timed out")
            return None
        
        except requests.exceptions.RequestException as e:
            logger.error(f"❌ Request failed: {e}")
            return None
    
    def get_repos(self, username: str, max_results: int = 10) -> List[Dict]:
        """Fetch user repositories"""
        url = f"{self.base_url}/users/{username}/repos"
        params = {
            'per_page': max_results,
            'sort': 'updated'
        }
        
        try:
            response = self.session.get(url, params=params, timeout=10)
            response.raise_for_status()
            repos = response.json()
            logger.info(f"βœ… Fetched {len(repos)} repositories")
            return repos
        
        except requests.exceptions.RequestException as e:
            logger.error(f"❌ Failed to fetch repos: {e}")
            return []
    
    def search_repositories(self, query: str, language: Optional[str] = None) -> List[Dict]:
        """Search GitHub repositories"""
        url = f"{self.base_url}/search/repositories"
        
        # Build search query
        search_query = query
        if language:
            search_query += f" language:{language}"
        
        params = {
            'q': search_query,
            'sort': 'stars',
            'order': 'desc',
            'per_page': 10
        }
        
        try:
            response = self.session.get(url, params=params, timeout=10)
            response.raise_for_status()
            
            data = response.json()
            logger.info(f"βœ… Found {data['total_count']} repositories")
            return data['items']
        
        except requests.exceptions.RequestException as e:
            logger.error(f"❌ Search failed: {e}")
            return []
    
    def close(self):
        """Close the session"""
        self.session.close()
        logger.info("πŸ”’ Session closed")


# Example usage
if __name__ == "__main__":
    # Create client (no token for public API access)
    client = GitHubAPIClient()
    
    # Get user information
    user = client.get_user("torvalds")
    if user:
        print(f"\nπŸ‘€ User: {user['name']}")
        print(f"πŸ“Š Public Repos: {user['public_repos']}")
        print(f"πŸ‘₯ Followers: {user['followers']}")
    
    # Get repositories
    repos = client.get_repos("python", max_results=5)
    print(f"\nπŸ“š Top {len(repos)} Python repositories:")
    for repo in repos:
        print(f"  ⭐ {repo['name']} - {repo['stargazers_count']} stars")
    
    # Search repositories
    search_results = client.search_repositories("machine learning", language="python")
    print(f"\nπŸ” Top ML repositories in Python:")
    for repo in search_results[:3]:
        print(f"  ⭐ {repo['full_name']} - {repo['stargazers_count']:,} stars")
    
    # Clean up
    client.close()

Benefits of This Pattern 🎁

  • Session Reuse: requests.Session() maintains connection pooling for faster requests
  • Type Hints: Makes code more maintainable with Optional[Dict] and List[Dict]
  • Comprehensive Error Handling: Catches all exception types gracefully
  • Logging: Production-ready logging for debugging
  • Timeout Protection: Prevents hanging requests
  • Clean Interface: Simple methods that hide complexity
graph TD
    A["πŸ§‘β€πŸ’» User Code"]:::style1 --> B["πŸ—οΈ GitHubAPIClient"]:::style2
    B --> C["πŸ”„ Session Management"]:::style3
    B --> D["πŸ” Authentication"]:::style4
    B --> E["⚠️ Error Handling"]:::style5
    C --> F["🌐 HTTP Requests"]:::style6
    D --> F
    E --> F
    F --> G["πŸ“‘ GitHub API"]:::style7

    classDef style1 fill:#43e97b,stroke:#38f9d7,color:#fff,font-size:16px,stroke-width:3px,rx:14,shadow:6px;
    classDef style2 fill:#ff4f81,stroke:#c43e3e,color:#fff,font-size:16px,stroke-width:3px,rx:14,shadow:6px;
    classDef style3 fill:#6b5bff,stroke:#4a3f6b,color:#fff,font-size:16px,stroke-width:3px,rx:14,shadow:6px;
    classDef style4 fill:#ffd700,stroke:#d99120,color:#222,font-size:16px,stroke-width:3px,rx:14,shadow:6px;
    classDef style5 fill:#ff9800,stroke:#f57c00,color:#fff,font-size:16px,stroke-width:3px,rx:14,shadow:6px;
    classDef style6 fill:#00bfae,stroke:#005f99,color:#fff,font-size:16px,stroke-width:3px,rx:14,shadow:6px;
    classDef style7 fill:#9e9e9e,stroke:#616161,color:#fff,font-size:16px,stroke-width:3px,rx:14,shadow:6px;

    linkStyle default stroke:#e67e22,stroke-width:3px;

🎯 Hands-On Assignment: Build a Multi-API News Aggregator πŸ“°

πŸ“ Your Mission

Create a Python application that aggregates news from multiple public APIs, handles errors gracefully, implements rate limiting, and presents data in a user-friendly format. Build a production-ready news aggregator that demonstrates mastery of HTTP requests, authentication, and error handling!

🎯 Requirements

  1. Create a NewsAggregator class that:
    • Uses requests.Session() for connection pooling
    • Implements methods: get_top_headlines(), search_news(query), get_sources()
    • Handles authentication with API keys in headers
    • Implements retry logic for failed requests (3 attempts with exponential backoff)
  2. Implement comprehensive error handling:
    • Catch HTTPError, Timeout, ConnectionError
    • Use response.raise_for_status() to validate responses
    • Log all errors with timestamps and context
  3. Add rate limiting protection:
    • Track request counts per minute
    • Sleep when approaching API limits
    • Display warning messages to users
  4. Parse and format JSON responses:
    • Extract: title, description, author, published date, URL
    • Handle missing fields gracefully with default values
    • Format dates using datetime module
  5. Create a caching mechanism:
    • Cache responses for 15 minutes to reduce API calls
    • Use dictionary with timestamps: {url: (data, timestamp)}
    • Implement cache invalidation logic

πŸ’‘ Implementation Hints

  1. Use NewsAPI (https://newsapi.org) or similar free APIs - get your API key first
  2. Store API keys in environment variables: os.getenv('NEWS_API_KEY')
  3. Implement exponential backoff: time.sleep(2 ** attempt) for retries
  4. Use requests.Session() and set session.headers once for all requests
  5. For caching, check: time.time() - cached_timestamp < 900 (15 minutes)
  6. Handle pagination with params={'page': 1, 'pageSize': 20}

πŸš€ Example Input/Output

# Example usage
from news_aggregator import NewsAggregator
import os

# Initialize with API key
api_key = os.getenv('NEWS_API_KEY', 'your_api_key_here')
aggregator = NewsAggregator(api_key)

# Get top headlines
print("πŸ“° Top Headlines:")
headlines = aggregator.get_top_headlines(category='technology', country='us')
for article in headlines[:5]:
    print(f"  πŸ“Œ {article['title']}")
    print(f"     πŸ”— {article['url']}")
    print()

# Search for specific topics
print("\nπŸ” Searching for 'Python programming':")
results = aggregator.search_news('Python programming')
print(f"Found {len(results)} articles")

# Check cache statistics
print(f"\nπŸ’Ύ Cache hits: {aggregator.cache_hits}")
print(f"πŸ“‘ API calls: {aggregator.api_calls}")

# Output:
# πŸ“° Top Headlines:
#   πŸ“Œ Python 3.12 Released with Major Performance Improvements
#      πŸ”— https://example.com/python-312
#   
#   πŸ“Œ AI and Machine Learning: Top Python Libraries in 2025
#      πŸ”— https://example.com/ml-libraries
# 
# πŸ” Searching for 'Python programming':
# Found 42 articles
# 
# πŸ’Ύ Cache hits: 3
# πŸ“‘ API calls: 7

πŸ† Bonus Challenges

  • Level 2: Add support for multiple news APIs (NewsAPI, Guardian, Reddit) with unified interface
  • Level 3: Implement async requests using aiohttp for parallel API calls
  • Level 4: Add sentiment analysis using TextBlob to classify articles (positive/negative/neutral)
  • Level 5: Create a Flask/FastAPI web interface with real-time updates
  • Level 6: Store articles in SQLite database with full-text search capability
  • Level 7: Implement OAuth2 flow for Twitter API integration

πŸ“š Learning Goals

  • Master HTTP GET/POST requests with requests library 🌐
  • Implement production-ready error handling and retry logic ⚠️
  • Work with API authentication (API keys, Bearer tokens) πŸ”
  • Parse and validate JSON responses from real APIs πŸ“Š
  • Build efficient caching systems to reduce API calls πŸ’Ύ
  • Apply rate limiting to respect API quotas ⏱️
  • Use requests.Session() for connection pooling πŸ”„
  • Handle timeouts and network errors gracefully 🚧

πŸ’‘ Pro Tip: This pattern is used by production systems like Zapier, IFTTT, and Hootsuite for aggregating data from multiple APIs! Session management and caching are critical for scalable API clients. Always store API keys in environment variables, never hardcode them!

Share Your Solution! πŸ’¬

Completed the project? Post your code in the comments below! Show us your Python API mastery! πŸš€βœ¨


Conclusion πŸŽ‰

You’ve mastered the essentials of working with APIs in Python using the requests library – from basic GET/POST requests to building production-ready API clients with authentication, error handling, and rate limiting! With these skills, you can integrate any web service into your Python applications and build powerful, scalable systems that interact with the modern web ecosystem. Happy coding! πŸš€βœ¨

This post is licensed under CC BY 4.0 by the author.