I had been using Qwen3.6 and it was good enough for me, it can perform at Sonnet levels, which was my daily-coding-driver. I used the 35b variant or Gemma for my AI agents, because they are fast, very fast. Qwen3.8 was released and it's great, but it hurt for a few reasons.
Q4_K_M had issues with looping, despite lesser quants with 3.6 working just fine. So I ended up using Q4_K_L, which takes away from my usable context. That wouldn't be much of an issue, but 3.8 is so strong because it's reasoning is... exhaustive. It's a very chatty model. I found I could easily use 100k+ of context just from the reasoning it produces, when working on a hard problem. I've actually settled on 150k, because I have broken 100k, too. The native context size is only 262k, so that leaves me with 112k for inputs! Ouch. Not ideal when it needs to read a large portion of a code base.
As for the speed, I tried everything: I was an LM Studio user, but it's MTP implementation was broken for the model, I was getting very low acceptance rates. So off to Unsloth! It worked alright, but there were claims of better performance with projects like vLLM, SGLang, and ninfer. They all had their own quirks, ninfer having the most around tool calls. Qwen tool calls have historically been a challenge on all platforms and ninfer's approach is to be very strict, which produces the worst results. Ninfer's speeds were great though! Unsloth had the best tool calling, but it still wasn't perfect.
So I found a way to get the best of all worlds.
I found a fork of ninfer that somewhat fixes tool calls and introduced YaRN. I wanted to experiment with a pure NVFP4 KV cache anyway, and that could get me well beyond the 262k model's limit, with better speed and tool calling?! Sign me up. I found the fork's tool calls to still have issues and YaRN had bugs. So I made my own. I forked ninfer and introduced those two new options (YaRN and tool call fixes). I wanted a way to better maintain a fork, and the one I found changed too much, which would introduce merge conflicts later down the road. If I add features via command line flags, I can maintain it more easily. Work on ninfer isn't done, so I wanted to be able to pull in the core updates on a regular basis.
To the fun part. After a couple of rounds of debugging, I was able to fix the YaRN implementation and fix the remaining tool calling issues. It was working great and I could get a 400k context! Much more room for Qwen's inner dialog. I wrote a small script to validate the extended context was working correctly aanndd... error. I found it was an unaddressed bug in the main project and related to their new NVFP4 KV cache implementation. It was a pretty small bug, so I fixed it up in no time, and pointed the author to my change, in case he wants to pull in the change.
It was working great. No issues with recall, no loops, it was fast and tool calls always worked. But we can't accept great solution, can we? I had one more idea. Unsloth will automatically offload vision to the CPU, if there's not enough VRAM, so I wanted the same! It took a little bit of work, but I got it all working. Vision on a CPU isn't too bad either. It's not lightning fast, but it runs at a very reasonable speed.
So where did I end up? 100-120 tokens per second in decoding (much faster than Claude!), a great model, MTP (which uses up VRAM), and vision. The best part is I now have a 456k context! That's massive to me. I used to work within 320k or 160k with Claude. I am extremely happy with the results.
If you have an RTX 5090 and want to try it, you can find it here. If I come up with some other ideas that can speed things up or increase context I will, as long as I can do them in an additive way, so I can keep the project up to date with the base project.