Thoughts on building a LLM based product

In the past few months, if you haven't heard words like LLMs, Generative AI, AI, and how it is taking our jobs away, you are probably at some fancy cozy remote island enjoying life and not caring about what I'm writing up here anyway.

As I was not at some fancy cozy remote island in the past couple of months, I had the opportunity of building a product around LLM and I have to say that my feelings are mixed. So here are some of those "yes but" feelings I had about working with LLMs:

• It feels like you can quickly solve any problem with it but it's so hard to guarantee that the problem is solved. First time when it fails solving it, it may create its own problem to solve, it loves hallucinating sometimes.
• It can take you a few seconds to do what you need through chat UI, which any of the LLMs offer these days, but it may be almost impossible to do that using their API. That can easily lead you to underestimate the effort needed to achieve the desired solution. Most of the time, you'll face a bunch of other issues that you'll need to take care of which can be very time-consuming (e.g. UI easily accepts PDFs but API of it can just not offer that, you start experiencing timeouts from time to time as sometimes processing takes too long, etc).
• It can produce any answer you can think of but it can be very hard to have consistent responses. With that said, if you need such a solution, most likely LLMs are not what you are looking for. It may require a lot of prompt engineering, post-processing and still, you won't be able to guarantee how consistent the answers are.
• It often happens that today you are not able to solve something but tomorrow there is suddenly a new fancy model that takes care of exactly what you need. That can be very frustrating for software engineers as not many things rely on you but you hope that someone else will do the job for you.

Let's put it this way. In some magic world, you are walking across green fields, enjoying the sun and the sound of birds, having only your magic wand called LLM with you, and feeling powerful. Out of the blue, you've heard some strange noise and noticed that you are being ambushed by bad guys. You are not scared at all, why would you, you have your magic wand with you, just tell magic words, and problem solved. You are raising a magic wand in the air and telling magic words "Please make me invisible!". Immediately, a huge puff of smoke erupts from the wand. The wand grants your wish, you are now invisible, but there is also one more deal. It makes everything else invisible to you, leaving you blind and helpless, stumbling around. In the end, you did manage to escape bad folks but next time, you might consider being more precise in what you are prompting your magic wand to do.

Conclusion

Building a LLM based product is a mix of fun and frustration. You will experience a lot of interesting engineering problems to solve but also a lot of annoying stuff like tweaking a prompt for hours to get better results. It may be challenging to fully depend on LLM's responses for building a reliable product but it does provide a solid foundation and most likely you will need to do last mile delivery yourself.