Document classification with AI: methods, models and use cases

Almost every enterprise operation team is pushing automation into their document process. And that’s right, as the market demands speed. Manual data entry can be a slow, costly and tedious process. In addition, incoming documents can be very complex. It could include multi-page loan applications, jumbled medical bills. With this in mind, raw business documents can look like a very messy intern’s desk after a triple espresso.

AI document classification will help you sort your incoming documents into pre-defined categories such as invoices/receipts, contracts, claim forms and bank statements. It is achieved using 5 different methods. Rule based, traditional ML, fine-tuned language models, document aware models and large language models. The method you choose depends on how diverse your documents are, how many labelled examples you have available for training and where your data can reside.

What is document classification?

The process involves identifying what class of document each incoming file belongs to. An AI classifier will return what class the file belongs to with a confidence score (based on the probability distribution of the classification head) and the page range of the document if it is a multi-page PDF. It means the file can be immediately routed to the correct workflow. E.g. invoice, 0.97, pages 1-2.

Depending on your business logic, document type classification handles either a single-label classification setup (where every page belongs to one exact category) or a multi-label classification setup (where a single file triggers multiple tags). To clear up any confusion right off the bat: we’re talking about operational document classification here, not security classification and not archival records classification (building a long-term file plan).

Document classification vs. OCR, data extraction and IDP

It is common to confuse document AI terminology, but every single stage solves a completely different part of the pipeline:

StepQuestion it answersOutputExample
OCRWhat text is on the page?Machine-readable text with bounding boxes coordinatesA scanned invoice turned into raw text strings and layout positions
Document classificationWhat type of document is this?Class label, confidence score, and page range“invoice”, confidence: 0.97, pages 1–2
Data extractionWhich specific values does it contain?Named entity fieldsInvoice number, issue date, total amount, vendor name
Intelligent document processing (IDP)How is it processed end-to-end?Validated, structured data delivered directly to downstream enterprise systemsReceipt capture → OCR → classification → extraction → validation → ERP system

Intelligent document classification sits right in the middle between raw OCR scanning and deep field extraction: pulling fields out of forms is the job of form processing services; classification figures out which form template applies in the first place before you accidentally feed a tax form to an invoice parser.

How automated document classification works

Automated document classification and automatic document classification work by taking an incoming file, converting it into machine-readable text, layout geometry, and visual features, running those signals through a trained model to predict the class and confidence score, and passing the output straight to routing rules.

How does document classification work?

Intake: Files are continually ingested from various channels.

Preprocessing and OCR: Scanned pages go through image cleanup and text recognition. Teams without their own engine rely on OCR data services to get clean OCR text alongside precise bounding boxes coordinates.

Extract features: Textual Tokens are extracted, 2D Bounding Box (layout) features and Visual Page Patch features.

Prediction: The model will run through inference and predict what type of document it is.

Routing: The business logic will check the confidence score against a confidence threshold. If the confidence score is high it will be sent to straight through processing. If the confidence score is low it will be sent to a Human in the Loop queue for review. If the layout is strange it will be sent to the other class.

Feedback & retraining: Any edge cases that have been fixed by a human reviewer will be saved. This increases the diversity of the data set when retraining the model at scheduled times. It prevents model drift.

Text, layout and image: what the classifier looks at

To tell tricky documents apart, AI document classification models inspect three core signal types:

Words (Text): OCR-extracted tokens and vocabulary lists processed via tokenizers.

Positions (Layout): Spatial coordinates (bounding boxes) mapped to multi-dimensional embeddings showing where headers, tables, and footers live.

Appearance (Visual): Page image pixel data capturing logos, gridlines, stamps, and signatures.

Tax Forms can be an example of how the semantic vocabulary is very similar between a W-2 and 1099 form, but the grid layout and image characteristics are completely different. This leads to a significant difference in performance between BERT-base (89.81%) and LayoutLMv3-base (95.44%) when using the RVL-CDIP benchmark.

Confidence scores/ human review

Cloud based classifiers and neural networks will output the confidence scores producing a value between 0 and 1. A common issue in deployment is to just take the generic “95% confidence threshold” mentioned in a generic tech blog.

It is important to tune your confidence thresholds using your own validation dataset (either using reliability diagrams or temperature scaling). When performing this you will find that any below threshold documents should be sent to a human in the loop review queue. For OOD documents it is best practice to use a hard confidence threshold or “other” class (as Microsoft recommend), to ensure the model does not guess wildly at unknown layouts.

Multi page documents:

In many cases the documents you will face in business will not be nicely separated. Loans applications, bulk scans and inbox attachments may have multiple forms within one multi page PDF.

Modern technology allows for this by classifying and separating the pages.

Azure Document Intelligence: Can be set to split pages using the splitMode parameter (none perPage auto) the default setting in v4.0 is none.

Google Document AI: Using the Custom Splitter processor.

AWS: Bedrock Data Automation (files up to 3000 pages with sub documents up to 20 pages). Amazon Textract Analyze Lending for mortgage packs.

Document classification methods: from rules to LLMs

Selecting document classification methods and document classification techniques comes down to five main types of document classification algorithms, ranging from zero-maintenance rules to heavy vision-language foundation models. In production, engineering teams usually stack them into hybrid pipelines.

MethodHow it worksWhat it needsStrengthsLimits
Rule-basedKeywords, regular expressions, templates, sender metadata, barcodesRules written manually for each document classHighly transparent, zero training data neededFragile; breaks instantly when wording or layout shifts
Classic machine learningTF-IDF features mapped to sparse vectors, evaluated by Naive Bayes, SVM, or logistic regressionLabeled training samples for every document classBlazing fast, cheap, runs on a CPU, super easy to explainIgnores spatial word order and visual layout geometry
Deep learning on textFine-tuned BERT-style encoder processing raw OCR token sequencesLabeled dataset, GPU instance for trainingDeeply understands context and complex wordingIgnores layout structure; long documents require chunking
Layout-aware & image modelsMultimodal cross-attention over text tokens, 2D bounding boxes, and visual page imagesLabeled dataset, GPU training setupHighest published accuracy on complex scanned documentsHeavier resource footprint; several prominent weights have non-commercial licenses
LLMs & vision-language modelsPrompt-driven zero-shot or few-shot classification with class descriptionsClass descriptions, optional 5–10 reference examplesInstant support for new classes without full retrainingLower baseline accuracy without tuning, high cost per page, latency, privacy risks

Under the hood, machine learning document classification is a supervised task where the objective function will learn to minimize the cross-entropy loss function by finding optimal separations between patterns within labelled historical data. If you have no labels you can start with an unsupervised approach by clustering your data to find natural groupings. Semi-supervised approaches can be very powerful if you have limited labelled examples. You can also use zero shot prompting to gain instant cold start classifications by providing natural text descriptions of classes.

Rule based classification

Rule based engines will process your files by looking for exact matches of key phrases, regular expressions, anchors, barcodes and sender domains.

Zero training data required and completely transparent. However this can be very brittle. Simple changes to formatting or text will break the process. Rule based approaches should only really be used for static standardised formats or as a very cheap initial filter before a smart model.

Classic machine learning: TF-IDF with Naive Bayes, SVM or logistic regression

Traditional machine learning approaches will convert your documents into numeric features using TF-IDF. These features will then be processed by linear models..

Naive Bayes, SVM, and logistic regression are common classification algorithms in machine learning; for documents, they operate on TF-IDF features using a Naive Bayes classifier or support vector machine. Lightweight, fast and works well on cheap CPUs. This approach does not account for the order of words or geometry of the page, only the bag-of-words approach.

Deep Learning on text

Fine tune a BERT-like model. Document classification using nlp and deep learning. Pretrained encoder models such as BERT (Apache 2.0) or RoBERTa (MIT) are fine tuned on your own collection of documents through adding an additional classification layer on top of a pretrained transformer.

The model has great capabilities at understanding the semantic context and domain specific language through self-attention. However the text encoders do not take into account any visual bounding boxes so can struggle with documents that are heavily visual in nature and also there is an upper limit to the token length so long documents need to be broken down. Choosing between a fine-tuned transformer and a generative API is a classic natural language processing consulting question when scoping enterprise architecture.

Modern multimodal architectures aim to replicate human visual reading patterns by jointly pre-training text, geometric and pixel-based features via objectives such as masking language modelling and image-text matching.

Layout-aware and image-based models: LayoutLMv3, UDOP, DiT and Donut

LayoutLMv3 & UDOP: Combine text tokens, 2D bounding boxes, and visual patch features to deliver state-of-the-art performance on complex business documents. Notably, LayoutLMv3 is released under a non-commercial license (CC BY-NC-SA 4.0), making it ideal for research and prototyping rather than commercial deployment.

DiT & Donut: Focus primarily on visual pipelines. Donut operates as an OCR-free model that feeds raw page images directly into an encoder-decoder transformer, eliminating traditional text-extraction overhead.

Together, these multimodal architectures provide end-to-end capabilities for efficiently processing visual documents, structured forms, and unformatted textual layouts.

LLMs and vision-language models: zero-shot and few-shot classification

Using an LLM for document classification enables teams to provide raw text or page images to a vision language model with a prompt specifying what classes can be predicted. This allows for instant predictions through zero-shot prompting or few-shot examples. There is no need to train via backpropagation.

However, looking at the ICPR 2024 benchmark data we can see there are sacrifices to make. A zero shot GPT-4 vision model achieved 69.9% accuracy on subsets of RVL-CDIP. When using just raw OCR text this drops to 61.8% (GPT-4). Training an open source Mistral-7B model with classification head on 100 samples per class increased subset accuracy to 83.4% and 66.6% on the complete test set. This still falls short to specialised layout models that have been trained on large datasets (95-96%).

For zero-shot non generative models, small natural language inference models such as bart-large-mnli (MIT) can be used to convert the classification task into an NLI task. This provides fast inference without token generation issues. In the cloud, Google provides zero-shot pre-trained classifier v1.5 with Gemini 2.5 Flash. Issues with using only LLMs include hallucinations and high token costs per page, latency issues, privacy restrictions and unpredictable outputs.

Ready to make work easier for thousands of employees?

Book a call with our team. We’ll identify automation opportunities and show you where AI can make the biggest difference.

Document classification models compared

Let’s cut to the chase: there is no single best model for document classification for every scenario. On the standard RVL-CDIP benchmark test set, layout- and vision-aware models hit 95–96% accuracy, text-only transformers reach roughly 90%, and zero-shot LLM prompts hover around 70%.

Here is how prominent document classification models and document classification model options stack up on public benchmarks:

ModelDeveloper, yearReadsRVL-CDIP accuracyParametersWeights license
BERT-baseGoogle, 2018OCR text89.81%110MApache 2.0
RoBERTa-baseFacebook AI, 2019OCR text90.06%125MMIT
LayoutLM (base)Microsoft, 2020text + layout (+ image)91.78% (94.42% with image)113M (160M)MIT
LayoutLMv2 (base / large)Microsoft, 2021text + layout + image95.25% / 95.64%≈200M / ≈426MCC BY-NC-SA 4.0 (non-commercial)
LayoutLMv3 (base / large)Microsoft, 2022text + layout + image95.44% / 95.93%133M / 368MCC BY-NC-SA 4.0 (non-commercial)
DiT (base / large)Microsoft, 2022page image only92.11% / 92.69%87M / 304MMIT (pre-trained weights)
DonutNAVER Clova, 2022page image (no OCR)95.30%143MMIT
LiLT (base)SCUT, 2022text + layout (image features used)95.68%—MIT
UDOPMicrosoft, 2023text + layout + image96.00%794MMIT
DocFormer (base)Amazon, 2021text + layout + image96.17%183MNo public weights
GPT-4 Vision (zero-shot)OpenAI (ICPR 2024 test)page image + prompt69.9% ± 2.0%*—Commercial API
Mistral-7B (fine-tuned, 100/class)Mistral AI (ICPR 2024 test)OCR text83.4% ± 4.3%* (66.6% full test)7BApache 2.0

*Tested on 5 subsets of 160 test documents (10 per class), not the full test set. RVL-CDIP test set total: 40,000 documents. Figures as reported in cited papers, as of November 2026.

Why benchmark accuracy won’t match your documents

If you expect your production model to match published paper metrics out of the box, prepare for a reality check. Research by Larson et al. an estimated 8.1% of test labels are wrong. Furthermore, at least 32% of test documents overlap with templates or exact duplicates found in the training split.

When evaluated against out-of-distribution documents and real-world datasets (such as RVL-CDIP-N), model scores plummet: DiT-base drops from 93.3% to 78.6%, while LayoutLMv2 crashes from 88.7% down to 55.6% in the study authors’ own training run, not the 95.25% reported in the original paper. High document classification accuracy on leaderboards means very little until you test it against a held-out test set built from your own production data.

Licensing: which models you can use in a commercial product

Before deploying an open model into production, check its license carefully—open weights don’t automatically mean you can use them commercially.

License TypeModels IncludedCommercial Use Implications
MIT / Apache 2.0BERT, RoBERTa, LayoutLM (base), DiT (pre-trained), Donut, LiLT, UDOP, bart-large-mnli, Mistral-7B-v0.1Fully permitted for commercial products under standard attribution rules.
CC BY-NC-SA 4.0LayoutLMv2, LayoutLMv3 (code and model checkpoints)Strictly non-commercial. Requires explicit dual-licensing permissions for commercial platforms.
No License / RestrictedDiT checkpoints fine-tuned on RVL-CDIP, LayoutLM (large)Ambiguous legal standing. Avoid using directly in commercial software without legal review.
Not ReleasedDocFormerProprietary research model; unavailable for external deployment.

P.S. This breakdown is for technical guidance and does not constitute formal legal advice.

How to choose a document classification approach

Choosing the right technical path comes down to five operational factors: document variety, required inference speed, available labeled training samples, security boundaries, and budget limits.

If your organization needs to…Start with…Key Reason
Process fixed forms from known sendersRule-based engine (keywords, templates, barcodes)Completely deterministic, zero training data overhead.
Classify text-heavy documents (NDAs, letters)TF-IDF baseline ➔ Fine-tuned BERT-style encoderCheap text baseline; transformer semantic depth if needed.
Sort scanned forms where layout mattersCommercial-friendly visual/layout model (Donut, UDOP)Visual grid geometry separates lookalike document layouts.
Support new document classes instantlyZero-shot LLM / Vision-Language ModelLaunches immediately from natural language descriptions.
Split multi-document packet dropsManaged cloud classifier with auto-splittingGenerates clean page-range bounds per document.
Maintain strict on-premise data privacySelf-hosted open model (Apache 2.0 / MIT)Keeps sensitive files entirely inside local network perimeters.
Process massive volumes accuratelyFine-tuned encoder + confidence queuesPredictable cost per page with human review safety nets.

Cloud classifiers or a custom model

For most engineering teams, the first major choice is whether to use a managed cloud API or train a custom open-source model locally.

Managed Cloud ClassifierTraining Data RequiredSupported ClassesMulti-Document SplittingConfidence Output
Azure Document Intelligence (Custom Classifier)≥ 5 samples per class (max 100)2 to 1,000splitMode: none, perPage, auto0.0 to 1.0 per document
Google Document AI (Custom Classifier)Zero-shot (v1.5) or ≥ 10 train / 2 test samples≥ 2 labelsDedicated Custom Splitter[0.0, 1.0] score
Amazon Comprehend (Custom Classification)≥ 50 samples per class (CSV multi-class)2 to 1,000Not documented; a page number in each resultScore distribution per class
Amazon Bedrock Data AutomationBlueprint matching (no training)Blueprint-basedFiles up to 3,000 pages, documents up to 20 pages (off by default)—

Managed cloud classifiers handle text recognition out of the box, so choosing a cloud vendor usually means choosing OCR as a service as well. Cloud APIs are fantastic for standard formats, rapid prototyping, and non-sensitive workloads. On the flip side, building a custom self-hosted model makes sense for weird layouts, strict security needs, extreme scale where API fees add up fast, and total pipeline control.

How much training data you need

Cloud engines set low entry barriers: Azure requires just 5 samples per class, Google recommends 10, and Amazon Comprehend asks for 50. But don’t let marketing numbers fool you. ICPR 2024 testing proved that training a model on 100 samples per class yields 66.6% to 76.4% accuracy, compared to 95–96% for models trained on 320,000 documents.

In production, quality beats quantity every single time.

Document classification use cases by industry

Document classification use cases and document classification examples sit right at the front door of every enterprise workflow, transforming unorganized mixed document batches into clean, routed queues for indexing and archiving.

IndustryTarget DocumentsOperational Trigger & Action
Accounts PayableInvoices and receipts, credit notes, purchase ordersTriggers key field extraction and automated three-way matching; after classification, invoices move to processing—see invoice OCR with machine learning.
Banking & LendingLoan applications, bank statements, pay stubs, W-2s, KYC documentsRoutes mortgage packets to underwriting queues and powers NLP for financial documents.
InsuranceFirst notice of loss (FNOL), claim forms, repair quotes, policiesDrives rapid claims triage, automated routing, and fraud checks as core AI in insurance use cases.
HealthcarePatient referrals, lab tests, discharge summaries, insurance cardsIndexes incoming medical records directly into electronic health record (EHR).
Legal & ComplianceMaster service agreements, NDAs, amendments, court filingsTriggers automated contract review workflows, clause tagging, and retention schedules.
LogisticsBills of lading, commercial invoices, customs declarationsAutomates shipment tracking updates and cross-border customs filing clearance.
Human ResourcesResumes, employment contracts, identity verification cardsFeeds candidate management pipelines and automates employee onboarding checks.
Mailroom OperationsMixed inbound mail scans, generic email attachmentsCategorizes unstructured messages and dispatches them to appropriate department queues.

High-volume email streams and customer service tickets follow the exact same classification architecture, evaluated purely on natural language text features rather than spatial coordinates.

How to implement AI document classification: from pilot to production

Building a reliable document pipeline takes more than plugging in an API key. Here is the practical roadmap:

Audit document types and volumes: Map all incoming file formats, sources, daily page counts, and operational SLAs.

Define the taxonomy: Establish clear, mutually exclusive categories within your document taxonomy, configure a mandatory “other” class safety net, and choose single-label or multi-label logic.

Collect and label representative samples: Gather authentic scanned documents and real-world edge cases while using professional data labeling and annotation services to ensure clean ground truth.

Build a baseline and strong candidates: Train a fast baseline (like TF-IDF with logistic regression) before moving to heavy multimodal models.

Evaluate on a held-out internal set: Test performance using a held-out test set of real operational files instead of public benchmark leaderboards.

Set confidence thresholds and human review: Establish cutoff scores and route uncertain outputs to a human-in-the-loop review queue.

Monitor drift and retrain: Track accuracy over time, log low-confidence edge cases, and schedule regular retraining to fight model drift.

How to measure classification accuracy

Relying on raw overall accuracy is dangerously misleading when dealing with imbalanced data (like 10,000 standard invoices mixed with 50 rare credit memos suffering from class imbalance).

Instead, evaluate pipelines using class-specific metrics like precision and recall, the F1 score, and a clear confusion matrix:

Precision = TP / (TP + FP); Recall = TP / (TP + FN); F1 = 2 × Precision × Recall / (Precision + Recall)

Always calculate macro-averaging (treating every class equally) and weighted-averaging (factoring in volume), and plot a complete confusion matrix to see which document pairs confuse your model.

Preparing and labeling training documents

Labeling data requires clear guidelines for tricky edge cases—like separating lookalike documents such as W-2s vs 1099s or ambiguous claim forms. Training sets must mix files from different years, scanners, and departments to prevent overfitting.

Scrub duplicate templates between train and test splits to prevent data leakage, and ensure sensitive personal data is minimized before model ingestion.

How Data Science UA builds document classification solutions

With years of deep hands-on experience under our belts,even though we fully embrace automation (hey, we are an AI-first company after all). As part of our specialized optical character recognition services, we build custom architectures tailored to complex enterprise pipelines.

When companies need a breakthrough, rigid off-the-shelf templates don’t cut it. Our team designs production pipelines blending computer vision.

Building real document classification algorithms takes more than parsing keywords. It takes actual human expertise. Here is how that translates to real-world impact:

Our consulting work for Uklon, which included multilingual customer feedback classification, resulted in a 20% decrease in negative (1–2-star) feedback. 

In logistics automation, our custom engine for container number recognition on edge devices optimized lightweight computer vision models to run locally on resource-constrained hardware, cutting verification time by 70%.

Let’s have a quick call and create a smart document pipeline for your industry.

FAQ

What is document classification machine learning?

A typical example is Accounts Payable processing. Inboxes typically receive a range of documents in PDF format including invoices, receipts, credit notes and purchase orders. These can be classified by a classification model to determine the correct document type to allow for automation. Invoices can be sent through for extraction whilst unknowns can be flagged. One common example is the RVL-CDIP benchmark which has 16 classes of business documents.

Is document classification supervised or unsupervised?

Document classification NLP is generally a supervised learning task. This means that models are trained using historical data with pre-assigned labels. To help improve accuracy, unsupervised clustering can be used to pre-cluster the un-labelled documents to help find natural clusters. Zero shot LLMs can also be used to help classify documents when only provided with plain text descriptions of the classes.

What is the best model for document classification?

There is no one best model for document classification deep learning. Using the RVL-CDIP benchmark, multimodal models such as LayoutLMv3, UDOP and DocFormer achieve the highest results with an accuracy of 95-96%. If you are looking for models with permissive licensing then Donut, LiLT and UDOP (MIT) are great options. If you don’t have training data, zero shot prompts with LLMs are a great starting point.

Can ChatGPT or other LLMs be used for document classification?

Yes, ChatGPT can be used for Zero Shot Document Classification if it has been provided with appropriate class definitions within the prompt. Using ICPR 2024 benchmark data, GPT-4 Vision achieves an accuracy of 69.9% on subsets of the RVL-CDIP dataset. With fine tuning of an open source Mistral 7B model with 100 examples per class it achieves an accuracy of 83.4% on the same subsets (66.6% on the full test set). LLMs are great for quick prototyping, new classes and auto-labelling.

How do I automatically classify PDFs?

If the PDF is native digital then the text will be extracted from the digital text. If the PDF is scanned then OCR will be performed first. The classifier will process the text and bounding box co-ordinates. If there are multiple documents within one PDF they can be split using automated tools.

What is the difference between classification and categorization?

Most vendors will consider these terms to be the same. However if you are looking to be more specific then classification will assign a document to one category. Categorisation (Tagging) will assign multiple topics to the document. Clustering will automatically group un-labelled documents.

How accurate is document classification using LLM?

Accuracy will depend on how complex the documents are and how clear the scans are. Top multimodal models achieve an accuracy of 90-96% on public benchmarks such as RVL-CDIP. However accuracy will vary in production due to drift. There will also be noise in benchmarks with an estimated 8% error rate in labelling. Accuracy should always be measured using precision, recall and F1 score using a hold out test set.