Response to LinkedIn post by Michele Webb:
https://www.linkedin.com/feed/update/urn:li:activity:7465608020119478274
Why might it take 2 years to update an LLM model after a standards change.
At a recent NAACCR conference a data analyst who built LLMs for case identification for a hefty organisation made the claim to me “all you need is sufficient data and you will get everything correct”. I mused to myself, “little did he understand language usage or its computational processing”
Let’s take the simplest case: a change in the morphology code for a given histological description.
This person cannot update their model until sufficient examples of the changed standard permeates into their data source in sufficient volume to effect a change in their model. I speculate that this would be up to two years for obscure changes.
While as you say vendors watch the changes in standards. I have never heard or seen how any vendor reacts to those changes. While NAACCR and NCRA mount very significant campaigns to inform and educate registrars about changes to standards, there is no comparable campaign to understand how vendors react to the changes.
It is my personal opinion that vendors get off the hook on this issue. They don’t advertise what standards their training or test corpora are selected for. Two years ago I submitted to NAACCR a paper titled “Rapid adaptation to standards for conformant NLP for Reportability and Coding – Building trust in NLP”. It was not considered important enough to be given an oral presentation and was presented as a poster. So NAACCR referees don’t think it is important to understand how vendors make their software standards compliant and how well they conform. In that poster we identified 30 different rules for reportability alone, extracted from 10 different source documents. Any machine learning model of any type, whether to be any LLM or not, needs to have representative examples of each of these rules both for the training and test sets.
Now if you want to update your LLM model thoroughly for compliance to a change in standards you need to do a number of things. Using the example above:
a. search your millions of records for the codes that have changed;
b. change their gold standard values to the updated standards value;
c. do a text search for possible variations in language of the new standard to make sure you haven ’t missed anything;
d. if you don’t have enough records of the new standard then you have to build synthetic records to add to your training and test sets, or;
e. wait until your clients have supplied sufficient new examples to add to your corpus, maybe two years.
In this scenario you will always be two years behind the standards.
The diagram below illustrates the characteristic of trustable results from a Machine Learning classifier. Only the sample in the TEST Set uses report types that are reliably verified and therefore trustable. In ML practice it is typical for the test set to be about 10% of the total corpus available for training. The secret in selecting the Training corpus is to select reports that cover all the values of all the variables you think significant to correctly classifying a report. There are statistical methods to help with that selection but they are not definitive. The human intuition (read experience) of the problem is equally important in developing a good classifier.
The Universe of Trustable Reportability
When determining sample sizes I use the Law of Large Numbers as a guide which says statistical coherence is emerging with volumes of 20 data points. So to give me a safety margin I prefer to have a minimum of 30 documents conformant to a standard for training and 5 for testing, that is, for each value of each variable represented in the model.