I’m looking to sync my Algolia search index to my CMS and was looking to get some tips on the best approach to retrieve all the collection items in a single collection. I have a Make.com scenario setup that calls the Get List Collection Items API (GET /collections/:collection_id/items) and imports that Algolia data fine but I’m only able to get the first 100 due to the limit.
Is the only option here to make multiple calls with an increasing offset parameter until the offset exceeds the total?
The offset and total fields should do the trick. You’ll most likely need to keep track of how many total records you’ve retrieved. Play around with that.
Another option is to manually export your data, then import it into Algolia one time:
Initially make the request to clear the Algolia index
Make the Webflow API request to grab the total count of CMS collection items needed to iterate through
Set the total value in a variable
Repeater module repeats each step based off the number of rounds needed to process all records.
Make another Webflow Get Request for to retrieve this round’s batch of items.
Iterate through the data array retrieved
Set data into the JSON object
Make the Algolia Post request with the JSON object to update index.
Surely the ideal option is to setup the webhooks and update the Algolia index on every update on the Webflow collection. Glad I could MacGyver a no-code solution though
Byteline’s Webflow - Get Items task automatically returns all the Webflow collection records by internally performing pagination. Here’s the corresponding Byteline flow.
After getting the records, you can process them for whatever automation you’re building. Byteline’s API Connector task can be used to make outbound HTTP calls.