Data Feed: Changefiles

Each changefile has a timestamp in its filename that tells you the most recent update it contains.

• Weekly: changed_dois_with_versions_YYYY-MM-DDThhmmss_to_YYYY-MM-DDThhmmss.jsonl.gz
• Daily: changed_dois_with_versions_YYYY-MM-DDThhmmss.jsonl.gz

You can use the snapshot and changefiles together to keep your copy up to date by following these steps:

1. Download and import the current daily snapshot.
2. Determine the last update timestamp in the snapshot.
   • If you've already imported the snapshot, this is the maximum value of the updated field among all rows.
   • The timestamp in the snapshot file name is an upper bound on the max-updated value: unpaywall_snapshot_2021-08-04T083001.jsonl.gz
   • The file name is provided in the Content-Disposition header if you download it with a utility like curl: Content-Disposition: attachment; filename="unpaywall_snapshot_2021-08-04T083001.jsonl.gz"
3. Download all changefiles, starting with the most recent file wiith a last-updated timestamp before that of the snapshot. If the available changefiles are:
   1. changed_dois_with_versions_2021-05-01T080001.jsonl.gz
   2. changed_dois_with_versions_2021-05-02T080001.jsonl.gz
   3. changed_dois_with_versions_2021-05-03T080001.jsonl.gz
   and the snapshot timestamp is 2021-05-02T120000, get changefiles 2 and 3 in that order.
4. Import each changefile by reading it line by line, overwriting or updating the previous record for that row's DOI.
5. Continue to import changefiles as above, as they are released.

JSON Endpoint

{
  "list": [
    {
      "date": string (YYYY-MM-DD),
      "filename": string,
      "filetype": string ("jsonl", or "csv"),
      "last_modified": string (YYYY-MM-DDThh:mm:ss)
      "lines": integer,
      "size": integer,
      "url": string
    },
    …
  ]
}