Anurag’s Read

From 2,500 requests to 24: Rebuilding star history without the stargazers.

A clear API redesign story that shows why a smaller response is not always a cheaper query, with three designs compared and real database numbers.
Score
8.6
Verdict
Must read
Blog and date
GitHub,
Original length and level
7 min, intermediate
Topics
api design, database performance, pagination, privacy
Read the original

Context

Maintainers use a star history curve to see whether a launch or release changed the growth of a project. Old tools paged through the stargazer list, kept the timestamps, and dropped the identities. That exposed user identities to spam harvesting and made the database do heavy work on popular repositories.

GitHub limited stargazer lists to administrators and collaborators, which cut off tools that only needed counts.

Architecture

The old endpoint was GET /repos/{owner}/{repo}/stargazers, with up to 100 people per page. The new endpoint takes page and per_page and turns them into a calendar-week range on the created_at column. The database filters the star rows with an indexed date range first.

It then groups them into daily counts and returns up to 30 weeks per request.

Mechanisms

M1

Cost of the old endpoint

The star table holds the current state, so removing a star deletes its row. Each page request counted all current stars to compute the page count, although it returned at most 100 rows. A full curve for react/react, with about 251,000 stars, needed about 2,511 requests. In the week before the restriction, about 37,000 count queries ran slowly, averaging nearly 3 seconds.

Database safeguards killed more than 1,000 of them.

M2

Time-range pagination

The page and per_page values select weeks and not rows. The database uses the index on created_at to read only the rows in that range. It groups those rows by day, so each star is examined about once across a full history. For react/react this gives about 24 requests.

M3

Cached counts

The repository stargazers page on github.com now uses the cached star count. The same fix went to other stargazer API endpoints. Slow queries from that page, in the hundreds each week, fell to zero.

Design decisions

No.ChoseInstead ofWhy
D1Filter by time range before groupingAggregate every star first, then paginate with LIMIT and OFFSETPagination at the end still scans, groups, and sorts every star. For react/react, 24 requests would examine about 6 million rows.
D2A new aggregate endpointReturning star times without identities from the old endpointClients would still page through thousands of records, and the people-list design stays.

Key numbers

  • react/react has about 251,000 stars.
  • The old method needed about 2,511 requests, and the new one needs about 24.
  • About 37,000 slow count queries averaged nearly 3 seconds in the week before the restriction.
  • After the restriction, stargazer requests fell about 60% and count query time about 80%.

Failures and limits

  • Time-range bounding limits the period and not the number of stars, so a very busy week can still be slow.
  • Removed stars delete their rows, so the history cannot fully show them.
  • Results follow calendar weeks, which affects the edges of the window.

Lessons you can use

  • A smaller response does not mean a cheaper query, because the order of operations sets the cost.
  • Start from the user goal and not from the existing data shape.
  • Return only the data the use case needs.

Glossary

StargazerA user who stars a repository.
OFFSET paginationA paging method that skips a number of rows, so the database still reads them.
Indexed rangeA read of a value range through a database index instead of a full scan.