API Pagination: Making Billions of Products Scrolling Possible

Key Concepts:

1. Introduction to API Pagination

When APIs return large datasets, pagination is essential to break the data into smaller, manageable chunks.
Instead of sending thousands of records at once, data is sent in batches (e.g., 10 or 100 records per page).
Benefits: Manages server load, reduces network traffic, and keeps the application responsive.

2. Offset-Based Pagination

Definition: Uses an offset to specify the starting point for retrieving data.
Types:
- Page-based: Sets the number of items per page and calculates the starting point.
- Direct Offset: Uses offset and limit parameters directly in the query.
Example SQL Query: A basic offset query in SQL is shown (specific query not provided in transcript).
Problems:
- Performance: As the offset increases, query performance degrades because the database must process every row up to the offset.
- Data Consistency: When the dataset changes rapidly, offset-based pagination can miss records or show them twice due to shifting data.

3. Cursor-Based Pagination

Definition: Uses a cursor (typically an indexed column value) to track the last seen item and retrieve the next batch of data.
Implementation Steps:
1. Choose Index Column: Select an index column (e.g., ID) as the cursor.
2. Hash Cursor Value: Hash the cursor value for security.
3. Client Sends Last Seen Cursor: The client sends the hashed cursor value of the last item they received.
4. Filter and Fetch Next Batch: Use the cursor to filter the data and fetch the next batch of records.
5. Return Result and New Cursor: Send back the result set along with a new cursor value for the last item in the current batch.
6. Client Uses New Cursor: The client uses the new cursor for their next request.
Advantages:
- Consistency: Remains consistent even when records are added or deleted between requests.
- Performance: Works well with large, fast-changing datasets.
Use Cases: Real-time feeds or any data that changes frequently.

4. Types of Cursor-Based Pagination

Key Set Pagination:
- Uses an index key (e.g., primary key) to efficiently retrieve the next set of results.
- Directly accesses rows without scanning all preceding ones.
Time-Based Pagination:
- Uses timestamps as cursors to segment and retrieve records within specific time ranges.
- Practical for applications dealing with time-series data.

5. Key Takeaway and Conclusion

For large and fast-changing datasets, cursor-based pagination is recommended due to its consistency and performance advantages.
While more complex to implement, the effort is worthwhile for applications requiring reliable pagination.

6. Recommendation

The video recommends a system design newsletter for topics and trends in large-scale system design.