PP-DocBee - A Multimodal Large Model for Document Image Understanding Launched by Baidu's PaddlePaddle

What is PP-DocBee?

PP-DocBee is a multimodal large model focused on document image understanding, introduced by Baidu's PaddlePaddle team. Based on the ViT+MLP+LLM architecture, it possesses robust capabilities for parsing Chinese documents, efficiently handling various types of document content such as text, tables, and charts. PP-DocBee has achieved SOTA (State of the Art) levels in authoritative academic evaluations for models with similar parameter counts and performs exceptionally well in internal business scenarios involving Chinese. The inference performance of PP-DocBee has been optimized for faster response times while maintaining high-quality output. Suitable for scenarios like document Q&A and complex document parsing, PP-DocBee supports multiple deployment methods, providing efficient and intelligent solutions for document processing.

Main Features of PP-DocBee

Document Content Understanding: PP-DocBee accurately recognizes and understands elements such as text, tables, and charts in document images, supporting multimodal inputs including text and images.

Document Q&A: Generates accurate answers to questions based on the content of the document, utilizing the information contained within.

Structured Information Extraction: Converts information from documents (such as tables and charts) into structured data, facilitating further analysis and processing.

Technical Principles of PP-DocBee

Architecture Design: Based on the ViT (Vision Transformer) + MLP (Multilayer Perceptron) + LLM (Large Language Model) architecture, it combines the strengths of visual and language models to achieve end-to-end document understanding.

Data Synthesis and Preprocessing: Addresses the shortcomings in Chinese document understanding by designing intelligent document data production solutions, including combining OCR small models with LLM large models and generating image data based on rendering engines. During training, larger resize thresholds are set, and images are proportionally enlarged during inference to capture more comprehensive visual features.

Training Optimization: Mixes various document understanding data (such as general VQA, OCR, charts, mathematical reasoning, etc.), sets data ratio mechanisms to balance the quantity differences between different datasets. Based on OCR post-processing assistance, the text recognition results from OCR are used as prior information to enhance the model's understanding ability on images with clear text.

Project Address of PP-DocBee

GitHub Repository:  https://github.com/PaddlePaddle/PaddleMIX/tree/develop/deploy/ppdocbee

Online Demo: https://aistudio.baidu.com/application/detail/60135

Application Scenarios of PP-DocBee

Financial Sector: Parses financial reports, invoices, and other documents to extract key data, assisting in financial analysis and auditing.

Legal Sector: Processes contracts, regulations, and other documents to quickly locate clauses, supporting legal compliance reviews.

Academic Sector: Extracts text and chart information from papers, aiding in literature retrieval and research analysis.

Enterprise Document Management: Extracts and structures internal document content, optimizing document retrieval and management processes.

Educational Sector: Parses textbooks and exam papers, assisting in the development of teaching resources and personalized learning.