Automatic Information Extraction from (semi-)structured documents: such as scanned images or PDFs containing tables, graphics, and other elements, remains ahead when using large language models (LLMs). This discrepancy stems from the loss of layout information when converting (e.g., Optical Character Recognition - OCR) documents to LLM-readable format (i.e., plain texts). Retrieval Augmented Generation (RAG) has been instrumental in combining the Retrieval of relevant information with generative models (i.e., LLMs) to improve chat-based question-answering systems. We propose a novel method that integrates layout information within the Retrieval Augmented Generation (RAG) framework. This approach leverages Intelligent Document Processing (IDP) to empower chat-based agents for effective document interaction. By incorporating layout cues alongside retrieved text, our RAG system improved question-answering capabilities for document-based chats.
Original languageEnglish
Publication statusPublished - 16 Oct 2023
EventAI Flanders Research Days - Van Beethovenstraat 8/10, Mechelen, Antwerp, Belgium
Duration: 16 Oct 2023 → …


WorkshopAI Flanders Research Days
Period16/10/23 → …


Dive into the research topics of 'A Layout-informed Retrieval Augmented Generation Framework for Document-based Chat'. Together they form a unique fingerprint.

Cite this