Need help converting parsed JSON data into CMS Collection

Apologies if this is the wrong subforum. I know there’s a lot of these threads, so let me explain my situation.

For a project, I’m trying to integrate some data from a third party API, Eventive in this case, into a Webflow collection. We don’t forsee the data we’re retrieving, films in this case, being updated more than maybe one to three times a week, but we’d like to set up something that would listen maybe once a day at least.

Unfortunately, this isn’t as simple as just using Zapier or Integromat. The official Zapier for Eventive isn’t compatible with Webflow, and Integromat doesn’t have Eventive listed.

Hoping I could drum up some solution, I wrote up some code that’s supposed to update if the number of films has changed. For testing purposes, I have the code to load on page load. Not too shockingly, after working out all the kinks, I got something that would convert the third party API to a parsed JSON file, but the actual updating parts were blocked by CORS policy.

Seeing as to how we really don’t want to make more than a few API calls a week, we don’t really want to go through the trouble of setting up a backend server. Is there an easier way to schedule a daily task to run the script, AND not have nintety-something CORS blocks in my console?

Don’t have much of a Webflow site at the moment, as this is still in early development, but just so it’s here:
Read-Only Link: Webflow - Fantastic Fest 2023 Dev Test

I also have the code I’d like to run (give or take a few modifications)

var event_bucket = "EVENTIVE_EVENT_BUCKET"; 
var public_api_key = "EVENTIVE_PUBLIC_API_KEY"; 

// Open a new connection, using the GET request on the URL endpoint
fetch('https://api.eventive.org/event_buckets/' + event_bucket + '/films?api_key=' + public_api_key, {
  method: 'GET',
  headers: {
    'Content-Type': 'application/json'
  }
})
.then(response => response.json())
.then(data => {
  // Begin accessing JSON data here
  Object.keys(data).forEach(key => {
    const film = data[key];
    console.log("TEST: Film Data");
    console.log(film);

    for (let i = 0; i < film.length; i++) { // Cycle through each Film
      const collectionId = 'WEBFLOW_COLLECTION_ID';
      const token = 'WEBFLOW_TOKEN';
      const apiUrl = 'https://api.webflow.com/collections/' + collectionId + '/items';

      // Retrieve the total count of items in the Webflow CMS collection
      fetch(apiUrl, {
        method: 'GET',
        headers: {
          Authorization: 'Bearer ' + token,
          'Content-Type': 'application/json'
        }
      })
        .then(response => response.json())
        .then(result => {
          // Compare the count of items to the number of films in Eventive
          if (result.total !== film.length) {
            const newItem = {
              fields: {
                name: film[i].name,
                long_rich: film[i].description,
                long_txt: film[i].description,
                poster: film[i].poster_image
              }
            };

            // Make a request to the Webflow API to create the new CMS item
            fetch(apiUrl, {
              method: 'POST',
              headers: {
                Authorization: 'Bearer ' + token,
                'Content-Type': 'application/json'
              },
              body: JSON.stringify(newItem)
            })
              .then(response => response.json())
              .then(result => {
                console.log(result);
                // The ID of the newly created CMS item will be in the "result._id" property
                const itemId = result._id;

                // Now you can update the CMS item using the ID
                const modifiedItem = {
                  fields: {
                    name: film[i].name,
                    long_rich: film[i].description,
                    long_txt: film[i].description,
                    poster: film[i].poster_image
                    // Add additional fields as needed
                  }
                };

			// Make a request to the Webflow API to update the existing CMS item
                fetch(apiUrl + '/' + itemId, {
                  method: 'PUT',
                  headers: {
                    Authorization: 'Bearer ' + token,
                    'Content-Type': 'application/json'
                  },
                  body: JSON.stringify(modifiedItem)
                })
                  .then(response => response.json())
                  .then(result => {
                    console.log(result);
                  })
                  .catch(error => {
                    console.error(error);
                  });
              })
              .catch(error => {
                console.error(error);
              });
          }
        })
        .catch(error => {
          console.error(error);
        });
    }
  });
})
.catch(error => {
  console.error(error);
});

At first glance, you may have gone down the wrong path on this, it sounds like you’re trying to solve the “keep the Webflow CMS current” problem with client side script running inside of your Webflow pages.

Even if you could make that work and overcome the CORS issues and security problems, that code would get executed hundreds of times a day depending on your website traffic.

You definitely want a server-side approach here.

I wasn’t clear on the issue you encountered with Zapier- Eventive doesn’t need to be “compatible” with Webflow. That’s the abstraction that automation services like Zapier provide- the services each do their tasks and return their data, and neither knows that the other exists.

Make should also be an option here, as long as you have good API documentation for Eventive, you don’t need the automation service to have a pre-built connector. It’s nice, but it’s a convenience.

My recommendation would be to go back to your Zapier approach since you’re closest there- or if you want to abandon Zapier, start with the API docs and Make.com to do your api call.

If you’re totally stuck, I do automations work, feel free to PM me for rates.

Hi @MCThompson

You’re hitting the CORS because Webflow doesn’t allow you to call their API from front-end Javascript due to the need for you to include your secret API token that anyone could view and abuse.

“Authentication to the API is performed via HTTP Basic Auth. Provide your API key as the basic auth username value. You do not need to provide a password.”

It looks like that service allows you to directly connect to them. If you don’t want to setup a backend server yourself I’d suggest directly connecting via Make (Integromat) or NoCodeAPI.

With Make, use their HTTP module to make that basic auth request.

Make also has parsing modules that may get you everything you need for the conversion you’re after. I’ve had good success with that in the past.

Hope that helps!

Hey all, I’m actually actually having some luck with Make. There were so many options when I first checked that I didn’t realize you could do basic HTTP requests. Still working out kinks, but I’m better off than I was trying to code a solution myself. Thanks!

Hey, so I’ve ran into issues using Make. I’m able to connect to the third party API with no issues. I was even able to push some of the items into the Webflow CMS but we hit our 50 item limit on the free Webflow CMS plan. We’ve just upgraded the Webflow tier, so I’m able to get back to getting this to work. The problem I have now is that I can’t figure out Make’s Webflow module to update the CMS with the items that aren’t already in the CMS. I’ve tried filters, but it just populates the CMS with repeat data.

The biggest problem I’m having is testing the Make scenario, since it has such a small number of operations on their free tier. In practice, it should only run once a week, and we’re not expecting a huge number of new entries each week (maybe like five at most.) I just want it to update with the entries not already in the CMS, but every filter I’ve tried either just keeps listing the first entry from the thrid party API, or it starts at the first entry and creates duplicates.

How can I fix this without running out of Make operations? I can’t find much documentation on how to achieve this. Doesn’t help that Integromat changed it’s name to something as SEO unfriendly as “Make”.

Yea, keep searching for “integromat” instead.

1,000 operations/mo can get you really far. It’s ample. If that’s not enough, yes, go spend $9/mo for 10k operations/mo. If that’s out of budget, your out of luck using Make.

You can’t search the Webflow CMS from their API. Filters won’t help fix that. If you want to add entries that don’t exist it’s on you to track that.

The most common approach is to store your data in Airtable or the Make Datastore and search it.

Lookup tables solve that problem for you.

Without knowing more, that’s an estimated 2 to 3 operations per request. That get’s you 333 to 500 requests per/mo for free.

With what you’ve mentioned, you’re between 40 to 60 per month (five at most each week), meaning you can grow volume by 88% and still be on the free tier.

I’d guess your real problem is that you’re blowing through operations while trying to figure out how to make it work. The common solution for this is to either test this outside of Make or purchase a paid tier for a month.

That works better, thanks. Wasn’t sure if that much changed from Integromat to Make.

Right now, I’m using the data from last year’s festival to test with. That’s 98 films on the Eventive side (38 of which are currently in the Webflow CMS.) So 2-3 operations is probably a fair estimation right there. I don’t see anything really changing with these items once they’re in Webflow, and if we do change something about an item, I’d probably just change it on the Webflow side. So I don’t really need the ability to update existing items, just add new ones.

But yeah, the biggest problem is that I’m burning through operations trying to update the. Not that we couldn’t afford $9/mo if we needed to.

Was not aware of this. That certainly explains a lot. What I was trying to do earlier was add the other 50 films from last year, before I hit my limit.

I’m reading up on both Lookup tables and Make’s Datastore to try and figure out a solution. I’m not there yet, but this puts me on track. I’m still kind of confused as to which data I need to store; Eventive, Webflow or both?

I’ll update once I have a working solution or have additional questions, whichever comes first.