I Still See The Same Mistakes with LLMs
Here’s an example of the type of error I still see with LLM coding that I saw 2 years ago.
Example: I ask the model to write a block of code. I deploy the code in a test environment. It is obvious there is a bug. I describe the bug to the model and share error logs. The model acknowledges the error and re-writes the code block. The new code block has the same error, or, the original error has been replaced by a new error.
I ask the model to iterate again and the new code block is error free. However, sometimes the model will iterate for a third time and replace the new error from the second iteration with the error from the first iteration. This type of error tree still occurs. Perhaps it occurs less frequently today than 6 months or a year ago, but frequently enough where you say to yourself “the models are not performing at a level that is congruent with their respective capital investment”.
There is lots of work to be done in terms of improving real-world LLM coding performance. Perhaps advances in inference will fix these errors. Perhaps LLMs need to be augmented with external tools and data to avoid making such mistakes. Regardless, the type of promised ROI will not become reality until this LLM error type is resolved.