Nightly Full Ingest

The most common use of the Data API is a scheduled full ingest: walk every learning opportunity once per night and upsert it into a downstream store.

Strategy

1. Page through GET /v1/learning-opportunities using limit and offset.
2. Use the totalCount in each response to know when you have read every record.
3. Upsert each record by its stable id.

Because list responses are sorted by updatedAt descending and include totalCount, a fixed-page walk is straightforward, but it is not snapshot-safe: records updated mid-run can shift between pages, causing duplicates or misses. De-duplicate and upsert by stable id to absorb this, and treat the walk as best effort rather than a consistent snapshot.

Example

const PAGE_SIZE = 100

async function fetchAllLearningOpportunities(host: string, apiKey: string) {
  const headers = { Authorization: `Bearer ${apiKey}` }
  const all: unknown[] = []
  let offset = 0

  for (;;) {
    const url = new URL("/v1/learning-opportunities", host)
    url.searchParams.set("limit", String(PAGE_SIZE))
    url.searchParams.set("offset", String(offset))

    const response = await fetch(url, { headers })
    if (!response.ok) {
      throw new Error(`Request failed: ${response.status}`)
    }

    const page = await response.json()
    all.push(...page.results)

    offset += PAGE_SIZE
    if (offset >= page.pagination.totalCount) {
      break
    }
  }

  return all
}

tip

Persist the highest updatedAt you have seen so a future incremental sync can pick up where the full ingest left off once incremental endpoints are added.