Document classification with AI: methods, models and use cases
Almost every enterprise operation team is pushing automation into their document process. And that’s right, as the market demands speed. Manual data entry can be a slow, costly and tedious process. In addition, incoming documents can be very complex. It could include multi-page loan applications, jumbled medical bills. With this in mind, raw business documents can look like a very messy intern’s desk after a triple espresso.
AI document classification will help you sort your incoming documents into pre-defined categories such as invoices/receipts, contracts, claim forms and bank statements. It is achieved using 5 different methods. Rule based, traditional ML, fine-tuned language models, document aware models and large language models. The method you choose depends on how diverse your documents are, how many labelled examples you have available for training and where your data can reside.
What is document classification?
The process involves identifying what class of document each incoming file belongs to. An AI classifier will return what class the file belongs to with a confidence score (based on the probability distribution of the classification head) and the page range of the document if it is a multi-page PDF. It means the file can be immediately routed to the correct workflow. E.g. invoice, 0.97, pages 1-2.
Depending on your business logic, document type classification handles either a single-label classification setup (where every page belongs to one exact category) or a multi-label classification setup (where a single file triggers multiple tags). To clear up any confusion right off the bat: we’re talking about operational document classification here, not security classification and not archival records classification (building a long-term file plan).
Document classification vs. OCR, data extraction and IDP
It is common to confuse document AI terminology, but every single stage solves a completely different part of the pipeline:
| Step | Question it answers | Output | Example |
| OCR | What text is on the page? | Machine-readable text with bounding boxes coordinates | A scanned invoice turned into raw text strings and layout positions |
| Document classification | What type of document is this? | Class label, confidence score, and page range | “invoice”, confidence: 0.97, pages 1–2 |
| Data extraction | Which specific values does it contain? | Named entity fields | Invoice number, issue date, total amount, vendor name |
| Intelligent document processing (IDP) | How is it processed end-to-end? | Validated, structured data delivered directly to downstream enterprise systems | Receipt capture → OCR → classification → extraction → validation → ERP system |
Intelligent document classification sits right in the middle between raw OCR scanning and deep field extraction: pulling fields out of forms is the job of form processing services; classification figures out which form template applies in the first place before you accidentally feed a tax form to an invoice parser.
How automated document classification works
Automated document classification and automatic document classification work by taking an incoming file, converting it into machine-readable text, layout geometry, and visual features, running those signals through a trained model to predict the class and confidence score, and passing the output straight to routing rules.
How does document classification work?
Intake: Files are continually ingested from various channels.
Preprocessing and OCR: Scanned pages go through image cleanup and text recognition. Teams without their own engine rely on OCR data services to get clean OCR text alongside precise bounding boxes coordinates.
Extract features: Textual Tokens are extracted, 2D Bounding Box (layout) features and Visual Page Patch features.
Prediction: The model will run through inference and predict what type of document it is.
Routing: The business logic will check the confidence score against a confidence threshold. If the confidence score is high it will be sent to straight through processing. If the confidence score is low it will be sent to a Human in the Loop queue for review. If the layout is strange it will be sent to the other class.
Feedback & retraining: Any edge cases that have been fixed by a human reviewer will be saved. This increases the diversity of the data set when retraining the model at scheduled times. It prevents model drift.
Text, layout and image: what the classifier looks at
To tell tricky documents apart, AI document classification models inspect three core signal types:
Words (Text): OCR-extracted tokens and vocabulary lists processed via tokenizers.
Positions (Layout): Spatial coordinates (bounding boxes) mapped to multi-dimensional embeddings showing where headers, tables, and footers live.
Appearance (Visual): Page image pixel data capturing logos, gridlines, stamps, and signatures.
Tax Forms can be an example of how the semantic vocabulary is very similar between a W-2 and 1099 form, but the grid layout and image characteristics are completely different. This leads to a significant difference in performance between BERT-base (89.81%) and LayoutLMv3-base (95.44%) when using the RVL-CDIP benchmark.
Confidence scores/ human review
Cloud based classifiers and neural networks will output the confidence scores producing a value between 0 and 1. A common issue in deployment is to just take the generic “95% confidence threshold” mentioned in a generic tech blog.
It is important to tune your confidence thresholds using your own validation dataset (either using reliability diagrams or temperature scaling). When performing this you will find that any below threshold documents should be sent to a human in the loop review queue. For OOD documents it is best practice to use a hard confidence threshold or “other” class (as Microsoft recommend), to ensure the model does not guess wildly at unknown layouts.
Multi page documents:
In many cases the documents you will face in business will not be nicely separated. Loans applications, bulk scans and inbox attachments may have multiple forms within one multi page PDF.
Modern technology allows for this by classifying and separating the pages.
Azure Document Intelligence: Can be set to split pages using the splitMode parameter (none perPage auto) the default setting in v4.0 is none.
Google Document AI: Using the Custom Splitter processor.
AWS: Bedrock Data Automation (files up to 3000 pages with sub documents up to 20 pages). Amazon Textract Analyze Lending for mortgage packs.
Document classification methods: from rules to LLMs
Selecting document classification methods and document classification techniques comes down to five main types of document classification algorithms, ranging from zero-maintenance rules to heavy vision-language foundation models. In production, engineering teams usually stack them into hybrid pipelines.
| Method | How it works | What it needs | Strengths | Limits |
| Rule-based | Keywords, regular expressions, templates, sender metadata, barcodes | Rules written manually for each document class | Highly transparent, zero training data needed | Fragile; breaks instantly when wording or layout shifts |
| Classic machine learning | TF-IDF features mapped to sparse vectors, evaluated by Naive Bayes, SVM, or logistic regression | Labeled training samples for every document class | Blazing fast, cheap, runs on a CPU, super easy to explain | Ignores spatial word order and visual layout geometry |
| Deep learning on text | Fine-tuned BERT-style encoder processing raw OCR token sequences | Labeled dataset, GPU instance for training | Deeply understands context and complex wording | Ignores layout structure; long documents require chunking |
| Layout-aware & image models | Multimodal cross-attention over text tokens, 2D bounding boxes, and visual page images | Labeled dataset, GPU training setup | Highest published accuracy on complex scanned documents | Heavier resource footprint; several prominent weights have non-commercial licenses |
| LLMs & vision-language models | Prompt-driven zero-shot or few-shot classification with class descriptions | Class descriptions, optional 5–10 reference examples | Instant support for new classes without full retraining | Lower baseline accuracy without tuning, high cost per page, latency, privacy risks |
Under the hood, machine learning document classification is a supervised task where the objective function will learn to minimize the cross-entropy loss function by finding optimal separations between patterns within labelled historical data. If you have no labels you can start with an unsupervised approach by clustering your data to find natural groupings. Semi-supervised approaches can be very powerful if you have limited labelled examples. You can also use zero shot prompting to gain instant cold start classifications by providing natural text descriptions of classes.
Rule based classification
Rule based engines will process your files by looking for exact matches of key phrases, regular expressions, anchors, barcodes and sender domains.
Zero training data required and completely transparent. However this can be very brittle. Simple changes to formatting or text will break the process. Rule based approaches should only really be used for static standardised formats or as a very cheap initial filter before a smart model.
Classic machine learning: TF-IDF with Naive Bayes, SVM or logistic regression
Traditional machine learning approaches will convert your documents into numeric features using TF-IDF. These features will then be processed by linear models..
Naive Bayes, SVM, and logistic regression are common classification algorithms in machine learning; for documents, they operate on TF-IDF features using a Naive Bayes classifier or support vector machine. Lightweight, fast and works well on cheap CPUs. This approach does not account for the order of words or geometry of the page, only the bag-of-words approach.
Deep Learning on text
Fine tune a BERT-like model. Document classification using nlp and deep learning. Pretrained encoder models such as BERT (Apache 2.0) or RoBERTa (MIT) are fine tuned on your own collection of documents through adding an additional classification layer on top of a pretrained transformer.
The model has great capabilities at understanding the semantic context and domain specific language through self-attention. However the text encoders do not take into account any visual bounding boxes so can struggle with documents that are heavily visual in nature and also there is an upper limit to the token length so long documents need to be broken down. Choosing between a fine-tuned transformer and a generative API is a classic natural language processing consulting question when scoping enterprise architecture.
Modern multimodal architectures aim to replicate human visual reading patterns by jointly pre-training text, geometric and pixel-based features via objectives such as masking language modelling and image-text matching.
Layout-aware and image-based models: LayoutLMv3, UDOP, DiT and Donut
LayoutLMv3 & UDOP: Combine text tokens, 2D bounding boxes, and visual patch features to deliver state-of-the-art performance on complex business documents. Notably, LayoutLMv3 is released under a non-commercial license (CC BY-NC-SA 4.0), making it ideal for research and prototyping rather than commercial deployment.
DiT & Donut: Focus primarily on visual pipelines. Donut operates as an OCR-free model that feeds raw page images directly into an encoder-decoder transformer, eliminating traditional text-extraction overhead.
Together, these multimodal architectures provide end-to-end capabilities for efficiently processing visual documents, structured forms, and unformatted textual layouts.
LLMs and vision-language models: zero-shot and few-shot classification
Using an LLM for document classification enables teams to provide raw text or page images to a vision language model with a prompt specifying what classes can be predicted. This allows for instant predictions through zero-shot prompting or few-shot examples. There is no need to train via backpropagation.
However, looking at the ICPR 2024 benchmark data we can see there are sacrifices to make. A zero shot GPT-4 vision model achieved 69.9% accuracy on subsets of RVL-CDIP. When using just raw OCR text this drops to 61.8% (GPT-4). Training an open source Mistral-7B model with classification head on 100 samples per class increased subset accuracy to 83.4% and 66.6% on the complete test set. This still falls short to specialised layout models that have been trained on large datasets (95-96%).
For zero-shot non generative models, small natural language inference models such as bart-large-mnli (MIT) can be used to convert the classification task into an NLI task. This provides fast inference without token generation issues. In the cloud, Google provides zero-shot pre-trained classifier v1.5 with Gemini 2.5 Flash. Issues with using only LLMs include hallucinations and high token costs per page, latency issues, privacy restrictions and unpredictable outputs.
Ready to make work easier for thousands of employees?
Book a call with our team. We’ll identify automation opportunities and show you where AI can make the biggest difference.
Document classification models compared
Let’s cut to the chase: there is no single best model for document classification for every scenario. On the standard RVL-CDIP benchmark test set, layout- and vision-aware models hit 95–96% accuracy, text-only transformers reach roughly 90%, and zero-shot LLM prompts hover around 70%.
Here is how prominent document classification models and document classification model options stack up on public benchmarks:
| Model | Developer, year | Reads | RVL-CDIP accuracy | Parameters | Weights license |
| BERT-base | Google, 2018 | OCR text | 89.81% | 110M | Apache 2.0 |
| RoBERTa-base | Facebook AI, 2019 | OCR text | 90.06% | 125M | MIT |
| LayoutLM (base) | Microsoft, 2020 | text + layout (+ image) | 91.78% (94.42% with image) | 113M (160M) | MIT |
| LayoutLMv2 (base / large) | Microsoft, 2021 | text + layout + image | 95.25% / 95.64% | ≈200M / ≈426M | CC BY-NC-SA 4.0 (non-commercial) |
| LayoutLMv3 (base / large) | Microsoft, 2022 | text + layout + image | 95.44% / 95.93% | 133M / 368M | CC BY-NC-SA 4.0 (non-commercial) |
| DiT (base / large) | Microsoft, 2022 | page image only | 92.11% / 92.69% | 87M / 304M | MIT (pre-trained weights) |
| Donut | NAVER Clova, 2022 | page image (no OCR) | 95.30% | 143M | MIT |
| LiLT (base) | SCUT, 2022 | text + layout (image features used) | 95.68% | — | MIT |
| UDOP | Microsoft, 2023 | text + layout + image | 96.00% | 794M | MIT |
| DocFormer (base) | Amazon, 2021 | text + layout + image | 96.17% | 183M | No public weights |
| GPT-4 Vision (zero-shot) | OpenAI (ICPR 2024 test) | page image + prompt | 69.9% ± 2.0%* | — | Commercial API |
| Mistral-7B (fine-tuned, 100/class) | Mistral AI (ICPR 2024 test) | OCR text | 83.4% ± 4.3%* (66.6% full test) | 7B | Apache 2.0 |
*Tested on 5 subsets of 160 test documents (10 per class), not the full test set. RVL-CDIP test set total: 40,000 documents. Figures as reported in cited papers, as of November 2026.
Why benchmark accuracy won’t match your documents
If you expect your production model to match published paper metrics out of the box, prepare for a reality check. Research by Larson et al. an estimated 8.1% of test labels are wrong. Furthermore, at least 32% of test documents overlap with templates or exact duplicates found in the training split.
When evaluated against out-of-distribution documents and real-world datasets (such as RVL-CDIP-N), model scores plummet: DiT-base drops from 93.3% to 78.6%, while LayoutLMv2 crashes from 88.7% down to 55.6% in the study authors’ own training run, not the 95.25% reported in the original paper. High document classification accuracy on leaderboards means very little until you test it against a held-out test set built from your own production data.
Licensing: which models you can use in a commercial product
Before deploying an open model into production, check its license carefully—open weights don’t automatically mean you can use them commercially.
| License Type | Models Included | Commercial Use Implications |
| MIT / Apache 2.0 | BERT, RoBERTa, LayoutLM (base), DiT (pre-trained), Donut, LiLT, UDOP, bart-large-mnli, Mistral-7B-v0.1 | Fully permitted for commercial products under standard attribution rules. |
| CC BY-NC-SA 4.0 | LayoutLMv2, LayoutLMv3 (code and model checkpoints) | Strictly non-commercial. Requires explicit dual-licensing permissions for commercial platforms. |
| No License / Restricted | DiT checkpoints fine-tuned on RVL-CDIP, LayoutLM (large) | Ambiguous legal standing. Avoid using directly in commercial software without legal review. |
| Not Released | DocFormer | Proprietary research model; unavailable for external deployment. |
P.S. This breakdown is for technical guidance and does not constitute formal legal advice.
How to choose a document classification approach
Choosing the right technical path comes down to five operational factors: document variety, required inference speed, available labeled training samples, security boundaries, and budget limits.
| If your organization needs to… | Start with… | Key Reason |
| Process fixed forms from known senders | Rule-based engine (keywords, templates, barcodes) | Completely deterministic, zero training data overhead. |
| Classify text-heavy documents (NDAs, letters) | TF-IDF baseline ➔ Fine-tuned BERT-style encoder | Cheap text baseline; transformer semantic depth if needed. |
| Sort scanned forms where layout matters | Commercial-friendly visual/layout model (Donut, UDOP) | Visual grid geometry separates lookalike document layouts. |
| Support new document classes instantly | Zero-shot LLM / Vision-Language Model | Launches immediately from natural language descriptions. |
| Split multi-document packet drops | Managed cloud classifier with auto-splitting | Generates clean page-range bounds per document. |
| Maintain strict on-premise data privacy | Self-hosted open model (Apache 2.0 / MIT) | Keeps sensitive files entirely inside local network perimeters. |
| Process massive volumes accurately | Fine-tuned encoder + confidence queues | Predictable cost per page with human review safety nets. |
Cloud classifiers or a custom model
For most engineering teams, the first major choice is whether to use a managed cloud API or train a custom open-source model locally.
| Managed Cloud Classifier | Training Data Required | Supported Classes | Multi-Document Splitting | Confidence Output |
| Azure Document Intelligence (Custom Classifier) | ≥ 5 samples per class (max 100) | 2 to 1,000 | splitMode: none, perPage, auto | 0.0 to 1.0 per document |
| Google Document AI (Custom Classifier) | Zero-shot (v1.5) or ≥ 10 train / 2 test samples | ≥ 2 labels | Dedicated Custom Splitter | [0.0, 1.0] score |
| Amazon Comprehend (Custom Classification) | ≥ 50 samples per class (CSV multi-class) | 2 to 1,000 | Not documented; a page number in each result | Score distribution per class |
| Amazon Bedrock Data Automation | Blueprint matching (no training) | Blueprint-based | Files up to 3,000 pages, documents up to 20 pages (off by default) | — |
Managed cloud classifiers handle text recognition out of the box, so choosing a cloud vendor usually means choosing OCR as a service as well. Cloud APIs are fantastic for standard formats, rapid prototyping, and non-sensitive workloads. On the flip side, building a custom self-hosted model makes sense for weird layouts, strict security needs, extreme scale where API fees add up fast, and total pipeline control.
How much training data you need
Cloud engines set low entry barriers: Azure requires just 5 samples per class, Google recommends 10, and Amazon Comprehend asks for 50. But don’t let marketing numbers fool you. ICPR 2024 testing proved that training a model on 100 samples per class yields 66.6% to 76.4% accuracy, compared to 95–96% for models trained on 320,000 documents.
In production, quality beats quantity every single time.
Document classification use cases by industry
Document classification use cases and document classification examples sit right at the front door of every enterprise workflow, transforming unorganized mixed document batches into clean, routed queues for indexing and archiving.
| Industry | Target Documents | Operational Trigger & Action |
| Accounts Payable | Invoices and receipts, credit notes, purchase orders | Triggers key field extraction and automated three-way matching; after classification, invoices move to processing—see invoice OCR with machine learning. |
| Banking & Lending | Loan applications, bank statements, pay stubs, W-2s, KYC documents | Routes mortgage packets to underwriting queues and powers NLP for financial documents. |
| Insurance | First notice of loss (FNOL), claim forms, repair quotes, policies | Drives rapid claims triage, automated routing, and fraud checks as core AI in insurance use cases. |
| Healthcare | Patient referrals, lab tests, discharge summaries, insurance cards | Indexes incoming medical records directly into electronic health record (EHR). |
| Legal & Compliance | Master service agreements, NDAs, amendments, court filings | Triggers automated contract review workflows, clause tagging, and retention schedules. |
| Logistics | Bills of lading, commercial invoices, customs declarations | Automates shipment tracking updates and cross-border customs filing clearance. |
| Human Resources | Resumes, employment contracts, identity verification cards | Feeds candidate management pipelines and automates employee onboarding checks. |
| Mailroom Operations | Mixed inbound mail scans, generic email attachments | Categorizes unstructured messages and dispatches them to appropriate department queues. |
High-volume email streams and customer service tickets follow the exact same classification architecture, evaluated purely on natural language text features rather than spatial coordinates.
How to implement AI document classification: from pilot to production
Building a reliable document pipeline takes more than plugging in an API key. Here is the practical roadmap:
Audit document types and volumes: Map all incoming file formats, sources, daily page counts, and operational SLAs.
Define the taxonomy: Establish clear, mutually exclusive categories within your document taxonomy, configure a mandatory “other” class safety net, and choose single-label or multi-label logic.
Collect and label representative samples: Gather authentic scanned documents and real-world edge cases while using professional data labeling and annotation services to ensure clean ground truth.
Build a baseline and strong candidates: Train a fast baseline (like TF-IDF with logistic regression) before moving to heavy multimodal models.
Evaluate on a held-out internal set: Test performance using a held-out test set of real operational files instead of public benchmark leaderboards.
Set confidence thresholds and human review: Establish cutoff scores and route uncertain outputs to a human-in-the-loop review queue.
Monitor drift and retrain: Track accuracy over time, log low-confidence edge cases, and schedule regular retraining to fight model drift.
How to measure classification accuracy
Relying on raw overall accuracy is dangerously misleading when dealing with imbalanced data (like 10,000 standard invoices mixed with 50 rare credit memos suffering from class imbalance).
Instead, evaluate pipelines using class-specific metrics like precision and recall, the F1 score, and a clear confusion matrix:
Precision = TP / (TP + FP); Recall = TP / (TP + FN); F1 = 2 × Precision × Recall / (Precision + Recall)
Always calculate macro-averaging (treating every class equally) and weighted-averaging (factoring in volume), and plot a complete confusion matrix to see which document pairs confuse your model.
Preparing and labeling training documents
Labeling data requires clear guidelines for tricky edge cases—like separating lookalike documents such as W-2s vs 1099s or ambiguous claim forms. Training sets must mix files from different years, scanners, and departments to prevent overfitting.
Scrub duplicate templates between train and test splits to prevent data leakage, and ensure sensitive personal data is minimized before model ingestion.
How Data Science UA builds document classification solutions
With years of deep hands-on experience under our belts,even though we fully embrace automation (hey, we are an AI-first company after all). As part of our specialized optical character recognition services, we build custom architectures tailored to complex enterprise pipelines.
When companies need a breakthrough, rigid off-the-shelf templates don’t cut it. Our team designs production pipelines blending computer vision.
Building real document classification algorithms takes more than parsing keywords. It takes actual human expertise. Here is how that translates to real-world impact:
Our consulting work for Uklon, which included multilingual customer feedback classification, resulted in a 20% decrease in negative (1–2-star) feedback.
In logistics automation, our custom engine for container number recognition on edge devices optimized lightweight computer vision models to run locally on resource-constrained hardware, cutting verification time by 70%.
Let’s have a quick call and create a smart document pipeline for your industry.
FAQ
What is document classification machine learning?
A typical example is Accounts Payable processing. Inboxes typically receive a range of documents in PDF format including invoices, receipts, credit notes and purchase orders. These can be classified by a classification model to determine the correct document type to allow for automation. Invoices can be sent through for extraction whilst unknowns can be flagged. One common example is the RVL-CDIP benchmark which has 16 classes of business documents.
Is document classification supervised or unsupervised?
Document classification NLP is generally a supervised learning task. This means that models are trained using historical data with pre-assigned labels. To help improve accuracy, unsupervised clustering can be used to pre-cluster the un-labelled documents to help find natural clusters. Zero shot LLMs can also be used to help classify documents when only provided with plain text descriptions of the classes.
What is the best model for document classification?
There is no one best model for document classification deep learning. Using the RVL-CDIP benchmark, multimodal models such as LayoutLMv3, UDOP and DocFormer achieve the highest results with an accuracy of 95-96%. If you are looking for models with permissive licensing then Donut, LiLT and UDOP (MIT) are great options. If you don’t have training data, zero shot prompts with LLMs are a great starting point.
Can ChatGPT or other LLMs be used for document classification?
Yes, ChatGPT can be used for Zero Shot Document Classification if it has been provided with appropriate class definitions within the prompt. Using ICPR 2024 benchmark data, GPT-4 Vision achieves an accuracy of 69.9% on subsets of the RVL-CDIP dataset. With fine tuning of an open source Mistral 7B model with 100 examples per class it achieves an accuracy of 83.4% on the same subsets (66.6% on the full test set). LLMs are great for quick prototyping, new classes and auto-labelling.
How do I automatically classify PDFs?
If the PDF is native digital then the text will be extracted from the digital text. If the PDF is scanned then OCR will be performed first. The classifier will process the text and bounding box co-ordinates. If there are multiple documents within one PDF they can be split using automated tools.
What is the difference between classification and categorization?
Most vendors will consider these terms to be the same. However if you are looking to be more specific then classification will assign a document to one category. Categorisation (Tagging) will assign multiple topics to the document. Clustering will automatically group un-labelled documents.
How accurate is document classification using LLM?
Accuracy will depend on how complex the documents are and how clear the scans are. Top multimodal models achieve an accuracy of 90-96% on public benchmarks such as RVL-CDIP. However accuracy will vary in production due to drift. There will also be noise in benchmarks with an estimated 8% error rate in labelling. Accuracy should always be measured using precision, recall and F1 score using a hold out test set.





