Enrich Data from Google Sheet with OpenAI Agent and ScrapingBee

Automate the enrichment of company data with this workflow, designed to extract valuable information from a spreadsheet of company names and websites. It uses AI and web scraping to generate insights, including business area, value proposition, and customer profile, enhancing your company's data set for better decision-making.

Important notes

 
• Check legal regulations: Web scraping is involved, so ensure you comply with the legal regulations in your country before proceeding. It's always better to be safe than sorry!
 
• Mind those tokens: OpenAI tokens can accumulate quickly, so monitor usage to avoid unexpected charges. 💸

Main Workflow

Node 1 - Webhook

This node triggers the workflow via a webhook call. You can replace it with any other trigger, such as form submission, new row in Google Sheets, or a manual trigger.

Node 2 - Get Rows from Google Sheet

This node retrieves the list of companies from your spreadsheet. The columns in this Google Sheet include:

 
• Company: The name of the company
 
• Website: The website URL of the company
  These two fields are required at this step.
 
• Business Area: The business area deduced by OpenAI from the scraped data
 
• Offer: The offer deduced by OpenAI from the scraped data
 
• Value Proposition: The value proposition deduced by OpenAI from the scraped data
 
• Business Model: The business model deduced by OpenAI from the scraped data
 
• ICP: The Ideal Customer Profile deduced by OpenAI from the scraped data
 
• Additional Information: Information related to the scraped data, including:    
       
  • Information Sufficiency:        
             
    • Description: Indicates if the information was sufficient to provide a full analysis.
             
    • Options: "Sufficient" or "Insufficient"
           
         
       
  • Insufficient Details:        
             
    • Description: If labeled "Insufficient," specifies what information was missing or needed to complete the analysis.
           
         
       
  • Mismatched Content:        
             
    • Description: Indicates whether the page content aligns with that of a typical company page.
           
         
       
  • Suggested Actions:        
             
    • Description: Provides recommendations if the page content is insufficient or mismatched, such as verifying the URL or searching for alternative sources.
           
         
     
   

Node 3 - Loop Over Items

This node ensures that the website in the "extra workflow input" corresponds to the row being processed. You can delete this node, but ensure the "query" sent to the scraping workflow corresponds to the website of the specific company being scraped.

Node 4 - AI Agent

This AI agent is configured with a prompt to extract data from the content it receives. The node has three sub-nodes:

 
• OpenAI Chat Model: The model used is currently gpt4-o-mini.
 
• Call rantir Workflow: This sub-node calls the workflow to use ScrapingBee and retrieves the scraped data.
 
• Structured Output Parser: This parser structures the output for clarity and ease of use, adding rows to the Google Sheet.

Node 5 - Update Company Row in Google Sheet

This node updates the specific company's row in Google Sheets with the enriched data.

Scraper Agent Workflow

Node 1 - Tool Called from Agent

This node triggers when the AI Agent calls the Scraper. A query is sent with:

 
• Company name
 
• Website: The URL of the website

Node 2 - Set Company URL

This node renames a field, which is useful for performing transformations on data received from the AI Agent.

Node 3 - ScrapingBee: Scrape Company's Website

This node scrapes data from the provided URL using ScrapingBee. You can use any scraper of your choice, but ScrapingBee is recommended as it allows you to configure scraper behavior directly. Once configured, copy the provided "curl" command and import it into rantir.

Node 4 - HTML to Markdown

This node converts the scraped HTML data to Markdown, which is then sent to OpenAI. The Markdown format generally uses fewer tokens than HTML.

Improving the Workflow

Here are some ways you can enhance this workflow:

 
• Handle potential errors more effectively.
 
• Configure the scraper tool to scrape other pages on the website. Although this will cost more tokens, it can be useful (e.g., scraping "Pricing" or "About Us" pages in addition to the homepage).
 
• Instead of Google Sheets, connect directly to your CRM to enrich company data.
 
• Trigger the workflow from form submissions on your website and send the scraped data about the lead to a Slack or Teams channel.