Visual Regression Testing with Apify and AI Vision Model

This rantir workflow is a proof-of-concept template exploring how we can work with multimodal LLMs and their multi-image analysis capabilities. In this demo, we compare two screenshots of a webpage taken at different timestamps and pass both to our multimodal LLM for a visual comparison of differences. rantir’s basic LLM node supports handling multiple binary inputs (i.e., images) in an AI request.

How it works

This template runs in two parts: first, to generate the base screenshots, and second, to run the visual regression test by capturing fresh screenshots.

 
• Starting with a list of webpages captured in a Google Sheet, base screenshots are taken for each using an external web scraping service called Apify.com (I prefer Apify, but feel free to use whichever web scraping service is available to you).
 
• These base screenshots are uploaded to Google Drive and will be referenced later when we run our testing.
 
• In Phase 2, a scheduled trigger will fire at a later time, using the same web scraping service to generate fresh screenshots of our desired webpages.
 
• Next, we re-download the base screenshots in parallel. With both the old and new captures, we pass them to our LLM node. In the LLM node’s options, we define two “user message” inputs with the type of binary (data) for our images.
 
• Finally, we prompt our LLM with our testing criteria and capture the regressions detected. Note that results will vary depending on which LLM you use.
 
• A final report is generated using the LLM's output and uploaded to Linear.

Requirements

 
• Apify.com API key for web screenshotting service
 
• Google Drive and Sheets access to store the list of webpages and captures

Customising this workflow

 
• Have your own preferred web screenshotting service? Feel free to swap out Apify with your service of choice.
 
• If the web screenshot is too large, it may prove difficult for the LLM to spot differences with precision. Try splitting the captures into smaller images instead.