Transcribing Bank Statements To Markdown Using Vision AI

This rantir workflow offers an effective approach to parsing bank statement PDFs using multimodal LLMs, which outperforms traditional OCR by providing accurate data extraction, especially for tables and complex layouts.

Advantages of Multimodal Parsing over Traditional OCR:

   
• Reduces complexity and overhead by avoiding text pre-processing before sending to the LLM.
   
• Handles non-standard PDF formats that may produce errors with traditional OCR.
   
• Is significantly more cost-effective than premium OCR models, which often require further cleanup.

How it Works

   
• The bank statement PDF is imported from Google Drive. This example uses a mock statement with complex 5-column tables that OCR struggles with.
   
• Since multimodal LLMs do not accept PDFs directly, the PDF is converted to images using Stirling PDF. This tool is self-hostable, ensuring privacy for sensitive data.
   
• Stirling PDF returns the PDF as a series of JPGs (one per page) in a zip file. The rantir workflow decompresses and sorts these images in the correct order.
   
• Each image is resized with the Edit Image node for optimal balance between resolution and processing speed.
   
• The resized images are passed to the Basic LLM node, which uses the multimodal LLM (e.g., Gemini 1.5 Pro). A "user message" of binary type is added as input to process each image.
   
• The prompt instructs the LLM to transcribe each page to markdown for clarity. Alternatively, you can prompt for specific data points directly.
   
• The markdown version of each page can then be analyzed by another LLM node to extract data, such as deposit line items.

Requirements

   
• Google Gemini API for multimodal LLM processing.
   
• Google Drive for document storage.
   
• Stirling PDF for PDF-to-image conversion.

Customizing the Workflow

   
• Gemini 1.5 Pro is optimal for text document parsing, but other multimodal LLMs like OpenAI GPT or Anthropic Claude can also be used.
   
• For faster results, skip markdown formatting and directly request data extraction from the LLM.
   
• This template is versatile and can be adapted for invoices, inventory lists, contracts, legal documents, and more.