I’m looking to sync my Algolia search index to my CMS and was looking to get some tips on the best approach to retrieve all the collection items in a single collection. I have a Make.com scenario setup that calls the Get List Collection Items API (GET /collections/:collection_id/items) and imports that Algolia data fine but I’m only able to get the first 100 due to the limit.
Is the only option here to make multiple calls with an increasing offset parameter until the offset exceeds the total?
@gemin that shouldn’t be an issue.
total fields should do the trick. You’ll most likely need to keep track of how many total records you’ve retrieved. Play around with that.
Another option is to manually export your data, then import it into Algolia one time:
Then set up Webhooks to automatically keep your Algolia index in sync:
Let us know what ends up working for you!
Appreciate the detailed reply, @ChrisDrit !
Here’s the solution I ended up with. Maybe this could help someone in the future!
I ended up with the following Make scenario -
- Initially make the request to clear the Algolia index
- Make the Webflow API request to grab the total count of CMS collection items needed to iterate through
- Set the total value in a variable
- Repeater module repeats each step based off the number of rounds needed to process all records.
- Make another Webflow Get Request for to retrieve this round’s batch of items.
- Iterate through the data array retrieved
- Set data into the JSON object
- Make the Algolia Post request with the JSON object to update index.
Surely the ideal option is to setup the webhooks and update the Algolia index on every update on the Webflow collection. Glad I could MacGyver a no-code solution though
Byteline’s Webflow - Get Items task automatically returns all the Webflow collection records by internally performing pagination. Here’s the corresponding Byteline flow.
After getting the records, you can process them for whatever automation you’re building. Byteline’s API Connector task can be used to make outbound HTTP calls.
Thanks @dpstechy. This would’ve definitely saved me some time having to figure out pagination myself. Will checkout Byteline.