Cheap Models

So I've been doing some personal investigation and analyzing the cost-to-compute for various models, and what I've learned is that traditional ops for my site here are probably stupid and expensive for the kind of operations I need. This site isn't complex per-se but running everything from scratch on a small VPS requires me to be a bit more economical and discerning when it comes to technical dependencies, so I've been reviewing my own usage with Deepseek v4 flash and comparing in-house options. It's difficult to beat chinesium subsidy, but also selling my internal details to a third party is

  1. not the point of my site
  2. a liability
  3. did I mention branding?

So I'm left considering options, and the in-house options do not look great. I currently have a 6 year old 2070 Super which still kicks butt, but with only 8gb of VRAM I'm kind of stuck running micro-models like Gemma e2b on the card, which also takes away my ability to use it for other, video-game-related activities. So, purchasing options include:

  • Machines I cannot afford like the 5090.
  • Machines I can no longer find on the used market like the 3090.
  • Machines that generally lack effective vLLM support or Cuda magic like the Arc B70 Pro.
  • Machines that are purpose-built with VRAM bandwidth in mind like the Thinkstation PGX Spark, and which are completely out of my price bracket.

I'm also just a cheapskate, so spending a lot of money on something like this irks me. I'd prefer to use what I have or be thrifty, so obviously my ideal has to be some secret alternative option that nobody has found yet! Well, almost nobody, some folks are totally hogging DDR4 ram sticks for this, but you can technically run LLMs slowly through the CPU so long as you have a decent RAM bus that can transmit RAM into the CPU, which is what most enterprise-grade servers are built for. And guess what you can buy for rather cheap on second hand markets!

If you think about where LLM utility comes from, most of the value comes from long-running tasks that can utilize multimodal intelligence to perform work through tool calls and agent harnesses. While having ~60-80 tokens per second is nice for chat GPT, it's rather pointless if the model is spending most of it's time gathering context through tool calls and selecting best-fit against MTP predicted tokens for software outputs. Having high-speed, long-context, cheap agents is a nice dream, but you get to pick 2 these days, so why not opt for bulk and parallel operations over raw speed?

Imagine for a moment: A used Epyc series chip running on a mobo with well over 200gb/s over the bus, with access to well over 256gb glorious DDR4 RAM. You could very easily stand up 4 Qwen 3.8 27b coder models that run in parallel with MTP at very-long contexts (YaRN-compatible I hear), depending on the chip series and ram it's not unbelievable to assume you could hit ~7-10 tps, and if you just design your work items ahead of time it can chug in the background against your verification CI/CD and review steps autonomously, chewing through your deliverables and bringing your given projects to life, all for the low, low cost of ~$1500 worth of used, enterprise-grade components.

It's absurd but guys seriously I can't find a 3090 for less than $1,500 this is a plea for help.

← All notes