Quick, which animal is smarter?
If you selected the newborn baby. you’re right! And if you selected the newborn Gazelle, you’re right!
Ok, both assertions cannot possibly be correct…can they? After all. the human will ultimately be able to communicate and do things that the Gazelle could never do. But the Gazelle, as a newborn, can do things that the newborn human could never do either. So both are smarter at different things and at different times.
Pushing for Quicker, Better Performance
I’ve written about this concept before when discussing machine learning, especially as it concerns solutions that are pre-trained providing “out-of-the-box” capabilities vs. systems that require training onsite using an organization’s own curated data. The upshot is that pretrained systems are “smarter at birth” than systems requiring onsite training. But systems that can be trained onsite eventually contribute much more value. Let’s use the example of mortgage document classification.
Given that many IDP vendors promote the variety of pre-trained capabilities for document types ranging from invoices to bills of lading, it is not that surprising that companies involved with mortgage loan documentation would prefer something pretrained on their range of documents. An interesting thing about mortgage loan documents is that they can range in number from 50 to over 1000 depending a number of factors. While it isn’t out of the realm of possibility to achieve accuracy rates greater than 90%, getting to that level is not trivial.
We recently had a prospective client that processes over 700 types of documents seeking to increase overall system accuracy; they weren’t happy with the low-90% accuracy they are achieving. And they wanted to do it with a system that provides this capability out-of-the-box. But is it reasonable to expect to significantly improve performance with a pre-trained system?
Out-of-the-Box Can Miss the Mark
The complete and overly-detailed answer is somewhat murky but the quick answer is “no, it is not reasonable”, especially if the system in-place already achieves a good amount of performance. The explanation lies within the differences between the smartness of a baby Gazelle and a human baby. It is certainly possible to pre-train a system on a sizable number of documents and achieve a good level of performance, even up to 90% system accuracy of the training set is sizeable and reflects, to a significant degree, the same types of documents and attributes reflected in real-world production; even on over 700 document types.
But as we continue to try and push the performance higher and higher, we start to encounter the limits of that pretrained system. The reason is that this system was not trained on the client’s specific documents but on another set; even with a lot of effort and expense to make the training data as representative as possible, it is almost certain that any single organization’s own documents will have slight (or even major) differences that affect performance.
Just as you cannot take a baby Gazelle out of the environment suited to its pre-programming and expect it to easily adapt, you cannot take a pre-trained system and expect the same level of performance as what can be attained with training on a client’s specific documents.
A Hybrid Approach is Best
The answer is to provide support for both pre-trained functions along with onsite training. The pre-trained system can provide more-immediate benefits while onsite training can gradually fit a system to an organization’s own document-based information and requirements.
The upshot: don’t be lulled into the market hype and promises of pre-trained systems; they often show well in demonstrations but fall well-short of what organizations really want. Instead, opt for a solution that has some built-in capabilities that allows you to easily customize it to your own needs.