Our Hopes Were Dashed: Why Isn't TurboQuant Working?#
As you know, the local AI world throws something new at us every day. Last night, I decided to take a look at the famous TurboQuant thing. You know, that technology from Google DeepMind that reduces the KV cache to 3 bits and literally sucks up VRAM... I have an RTX 4060 Ti 16GB card, and I thought, "Man, now I can sail into massive context windows with this." But as it turns out, everything wasn't as rosy as it looked on paper. 🤯
I did the installation, everything is in place. But when I clicked on that famous llama-server.exe file (or fired it up from the terminal), what did I see? Nothing! Literally a silent resistance. No error code, no "this file is missing" warning... I click it, it closes instantly. This situation makes you a nervous wreck; I felt like throwing the computer out the window for a moment. Anyway, back to the subject.
Behind the Scenes of the Silent Crash: STATUS_DLL_NOT_FOUND#
Interestingly, it didn't give any output even when I ran it from the terminal. It just silently moves to the next line. While wondering, "Maybe the processor couldn't handle it (does it want AVX-512?)," I found myself on GitHub Issues pages. It turns out this isn't a "defect," it's literally a story of forgetfulness.
In the Windows CUDA 12.4 version (tqp-v0.1.1) of the TheTom/llama-cpp-turboquant fork, someone forgot to package two very critical files. It's exactly like building a house but forgetting the key inside. 🤦♂️
Warning! ⚠️
If the program closes for you without any error, it probably means OpenSSL files are missing from your system. This error usually happens silently in the Windows command line.
The Solution: OpenSSL 3 (LTS) to the Rescue#
The issue was this: The application needs OpenSSL libraries, namely libssl-3-x64.dll and libcrypto-3-x64.dll, to run. If these aren't installed on your computer (which they usually aren't), the application says "I'm leaving" as soon as it starts.
So I immediately jumped in with winget. First I tried the newest version (OpenSSL 4.0), but it turns out our finicky llama-server insisted on version "3". Now you have to start all over again... Luckily, the LTS version came to our rescue.
Tip ⚡
To solve this problem, paste this command into your terminal:
winget install ShiningLight.OpenSSL.LTS.Light
This command will provide your system with those missing DLL files it needs.
Conclusion: Let the Massive Context Enjoyment Begin!#
After installing OpenSSL version 3.x, that silently closing terminal suddenly came to life. It saw my RTX 4060 Ti card and greeted it so well... Now there's no more worry about running out of VRAM while writing code with 128K context. So sometimes the solution isn't inspecting code for hours, but just finding those two missing files and putting them in place. 😎
The bottom line is; technology sometimes plays little games like this with us. But when you don't give up and see the output of that final --version command on the screen, you feel relieved, as if the world exists. Now it's time to enjoy TurboQuant! 🚀
Comments