Automated URL detection with Percy.io (automated visual testing)

Community resources Show & Tell
1

Hi all!

I wanted to share a handy script I created that automates the selection of URLs to be visually tested by https://percy.io/ for websites hosted on Webflow’s staging environment.

For those who don’t know it, visual testing is a great way to ensure that your website looks and behaves as expected across different URLs and devices, and that no random page got invertedly changed by the work you were doing on other pages.

The script below, written in TypeScript, allows you to automatically detect URLs from Webflow’s sitemap and generate a YAML file that Percy.io can use to capture snapshots of your web pages.

You need to appropriately setup Percy.io on your environment first, including credentials and etc. But once that’s done, you can use the following script to automate the selection of URLs for testing:

src/scripts/generateSnapshotsYaml.ts 

/* eslint-disable import/no-extraneous-dependencies */

import fetch from 'node-fetch';
import { parseString } from 'xml2js';
import * as fs from 'fs';
import yaml from 'js-yaml';

const LIVE_DOMAIN = 'https://www.acme-corp.com/';
const STAGING_DOMAIN = 'https://acme-corp.webflow.io/';

async function fetchSitemap(url: string): Promise<string> {
  try {
    const response = await fetch(url);
    const xmlData = await response.text();
    return xmlData;
  } catch (error) {
    throw new Error(`Error fetching sitemap: ${error}`);
  }
}

function parseUrlsFromSitemap(xmlData: string): string[] {
  let urls: string[] = [];
  parseString(xmlData, (err, result) => {
    if (err) {
      throw new Error(`Error parsing XML: ${err}`);
    }

    urls =
      result?.urlset?.url?.map((urlObject: any) => urlObject.loc[0].trim()) ||
      [];
  });
  return urls;
}

function filterUniquePages(urls: string[]): string[] {
  const filteredUrls: string[] = [];

  const formattedUrls = urls.map(url => {
    const urlWithoutRootDomain = url.replace(LIVE_DOMAIN, '');
    const urlSections = urlWithoutRootDomain.split('/');
    const parentPath = urlSections.slice(0, urlSections.length - 1).join('/');

    return { url, parentPath };
  });

  formattedUrls.forEach((obj, idx) => {
    // If is not under any parent path
    if (!obj.parentPath) {
      filteredUrls.push(obj.url);
      return;
    }

    // If is first page with this parent path
    if (obj.parentPath !== formattedUrls[idx - 1]?.parentPath) {
      filteredUrls.push(obj.url);
    }
  });

  return filteredUrls;
}

function sortUrlsAlphabetically(urls: string[]): string[] {
  return urls.sort();
}

function replaceDomainsInUrls(
  urls: string[],
  oldDomain: string,
  newDomain: string,
) {
  return urls.map(url => url.replace(oldDomain, newDomain));
}

function writeUrlsToYamlFile(urls: string[]) {
  const yamlContent = yaml.dump(urls);
  fs.writeFileSync('snapshots.yml', yamlContent, 'utf8');
}

async function fetchAndParseSitemap() {
  const sitemapUrl = `${STAGING_DOMAIN}sitemap.xml`;

  try {
    const xmlData = await fetchSitemap(sitemapUrl);
    const urls = parseUrlsFromSitemap(xmlData);
    const sortedUrls = sortUrlsAlphabetically(urls);
    const filteredUrls = filterUniquePages(sortedUrls);
    const urlsWithStagingDomain = replaceDomainsInUrls(
      filteredUrls,
      LIVE_DOMAIN,
      STAGING_DOMAIN,
    );

    writeUrlsToYamlFile(urlsWithStagingDomain);
  } catch (error) {
    // eslint-disable-next-line no-console
    console.error(error);
  }
}

fetchAndParseSitemap();

Then, just add this script to your package.json scripts and run it whenever you want to run some visual tests: "test-visual": "ts-node-esm ./src/scripts/generateSnapshotsYaml.ts && percy snapshot ./snapshots.yml".

As always, make sure you have installed all the required dependencies such as node-fetch , xml2js , js-yaml , @percy/cli, ts-node, etc, using npm or yarn.