Essential IDP Evaluation Playbook | Knowledge Base | Definition
How to Conduct an IDP Evaluation
Intelligent Document Processing is essentially software that can classify, locate and extract data from documents and these documents can be born-digital (such as email or PDF) or they can be hard copy such as (scanned or faxed documents) for use in business systems, internal applications, RPA and other automation platforms across the organization. IDP software can leverage simple technologies such as OCR to very advanced machine learning or deep learning.
Let’s start with key factors that drive IDP evaluations. There are two factors and these are in order of priority. The first one is the performance of intelligent capture. Now it’s quite possible through a simple demo to review a list of system capabilities. But intelligent capture is quite different because the value proposition of intelligent capture is about optimizing the amount of structured information out of unstructured documents. That is, it’s about maximizing how much data output or results that you can get at the highest levels of accuracy. The biggest single factor that is in the success or failure of any kind of intelligent capture project is whether the system can actually deliver accurate data results consistently in the dynamic production environment.
For an evaluation, first ask your vendors to provide their configurations. The reason being is because even though the systems are designed to tackle the same problem, they all do it in a different way. It’s impractical for you to try to learn every nuance of the software, to the level of proficiency where you can successfully gauge the true performance of a system. So don’t expect to configure the system yourself.
Secondly, do your own measurements. Don’t take the vendor’s word for those measurements. Everybody talks about 99% accuracy. What vendors don’t tell you is what that really means. What accuracy really means and what you really get are explored in other webinars. When you’re talking automating data entry where you’ve got staff looking at the data on document pages and entering them into a system of record. Each one of those fields represents a task. If you’re going to measure at the page level, what percentage of pages are a hundred percent accurate? Meaning, you’re getting a hundred percent automation. It’s more important to look at the tasks because the tasks represent the actual effort, the labor, the cost savings, not at the page-level, but at the field-level. Use your own data, split your sample data into configurations and training sets. When you ask a vendor to configure a system for a POC or an evaluation, provide them with one data set that they can use to configure the system, but don’t provide them with the data set you’re going to use to test the performance, because there always ways to game the system, right? For the next four steps, it’s easiest to watch our on-demand webinar above or do a really deep-dive by downloading the eBook below that does a step-by-step on how to conduct a proof of concept.
Ultimately, the key is to focus on having really good sample sets and then evaluate the output of those systems. How quickly it’s configured and how easily the system is to maintain in production. It is nice to have an easy UI, but the engine behind that UI must be capable of classifying documents, locating, extracting and verifying your data so that the output is consistent and highly accurate in order to achieve real straight through processing.