It depends!

Exllamav2 was pretty fast on AMD, exllamav3 is getting support soon. Vllm is also fast AMD. But its not easy to setup; you basically have to be a Python dev on linux and wrestle with pip. Or get lucky with docker.

Base llama.cpp is fine, as are forks like kobold.cpp rocm. This is more doable without so much hastle.

The AMD framework desktop is a pretty good machine for large MoE models. The 7900 XTX is the next best hardware, but unfortunately AMD is not really interested in competing with Nvidia in terms of high VRAM offerings :'/. They don’t want money I guess.

And there are… quirks, depending on the model.

I dunno about Intel Arc these days, but AFAIK you are stuck with their docker container or llama.cpp. And again, they don’t offer a lot of VRAM for the $ either.

NPUs are mostly a nothingburger so far, only good for tiny models.

Llama.cpp Vulkan (for use on anything) is improving but still behind in terms of support.

A lot of people do offload MoE models to Threadripper or EPYC CPUs, via ik_llama.cpp, transformers or some Chinese frameworks. That’s the homelab way to run big models like Qwen 235B or deepseek these days. An Nvidia GPU is still standard, but you can use a 3090 or 4090 and put more of the money in the CPU platform.

You wont find a good comparison because it literally changes by the minute. AMD updates ROCM? Better! Oh, but something broke in llama.cpp! Now its fixed an optimized 4 days later! Oh, architecture change, not it doesn’t work again. And look, exl3 support!

You can literally bench it in a day and have the results be obsolete the next, pretty often.

Depends. You’re in luck, as someone made a DWQ (which is the most optimal way to run it on Macs, and should work in LM Studio): https://huggingface.co/mlx-community/Kimi-Dev-72B-4bit-DWQ/tree/main

It’s chonky though. The weights alone are like 40GB, so assume 50GB of VRAM allocation for some context. I’m not sure what Macs that equates to… 96GB? Can the 64GB can allocate enough?

Otherwise, the requirement is basically a 5090. You can stuff it into 32GB as an exl3.

Note that it is going to be slow on Macs, being a dense 72B model.

One last thing: I’ve heard mixed things about 235B, hence there might be a smaller, more optimal LLM for whatever you do.

For instance, Kimi 72B is quite a good coding model: https://huggingface.co/moonshotai/Kimi-Dev-72B

It might fit in vllm (as an AWQ) with 2x 4090s. It and would easily fit in TabbyAPI as an exl3: https://huggingface.co/ArtusDev/moonshotai_Kimi-Dev-72B-EXL3/tree/4.25bpw_H6

As another example, I personally use Nvidia Nemotron models for STEM stuff (other than coding). They rock at that, specifically, and are weaker elsewhere.

Ah, here we go:

https://huggingface.co/ubergarm/Qwen3-235B-A22B-GGUF

Ubergarm is great. See this part in particular: https://huggingface.co/ubergarm/Qwen3-235B-A22B-GGUF#quick-start

You will need to modify the syntax for 2x GPUs. I’d recommend starting f16/f16 K/V cache with 32K (to see if that’s acceptable, as then theres no dequantization compute overhead), and try not go lower than q8_0/q5_1 (as the V is more amenable to quantization).

Qwen3-235B-A22B-FP8

Good! An MoE.

Ideally its maxium context lenght of 131K but i’m willing to compromise.

I can tell you from experience all Qwen models are terrible past 32K. What’s more, going over 32K, you have to run them in a special “mode” (YaRN) that degrades performance under 32K. This is particularly bad in vllm, as it does not support dynamic YaRN scaling.

Also, you lose a lot of quality with FP8/AWQ quantization unless it’s native FP8 (like deepseek). Exllama and ik_llama.cpp quants are much higher quality, and their low batch performance is still quite good. Also, VLLM has no good K/V cache quantization (its FP8 destroys quality), while llama.cpp’s is good, and exllama’s is excellent, making it less than ideal for >16K. Its niche is more highly parallel, low context size serving.

My current setup is already: Xeon w7-3465X 128gb DDR5 2x 4090

Honestly, you should be set now. I can get 16+ t/s with high context Hunyuan 70B (which is 13B active) on a 7800 CPU/3090 GPU system with ik_llama.cpp. That rig (8 channel DDR5, and plenty of it, vs my 2 channels) should at least double that with 235B, with the right quantization, and you could speed it up by throwing in 2 more 4090s. The project is explicitly optimized for your exact rig, basically :)

It is poorly documented through. The general strategy is to keep the “core” of the LLM on the GPUs while offloading the less compute intense experts to RAM, and it takes some tinkering. There’s even a project to try and calculate it automatically:

https://github.com/k-koehler/gguf-tensor-overrider

IK_llama.cpp can also use special GGUFs regular llama.cpp can’t take, for faster inference in less space. I’m not sure if one for 235B is floating around huggingface, I will check.

Side note: I hope you can see why I asked. The web of engine strengths/quirks is extremely complicated, heh, and the answer could be totally different for different models.

Be specific!

• What models size (or model) are you looking to host?

• At what context length?

• What kind of speed (token/s) do you need?

• Is it just for you, or many people? How many? In other words should the serving be parallel?

In other words, it depends, but the sweetpsot option for a self hosted rig, OP, is probably:

• One 5090 or A6000 ADA GPU. Or maybe 2x 3090s/4090s, underclocked.

• A cost-effective EPYC CPU/Mobo

• At least 256 GB DDR5

Now run ik_llama.cpp, and you can serve Deepseek 671B faster than you can read without burning your house down with H200s: https://github.com/ikawrakow/ik_llama.cpp

It will also do for dots.llm, kimi, pretty much any of the mega MoEs de joure.

But there’s all sorts of niches. In a nutshell, don’t think “How much do I need for AI?” But “What is my target use case, what model is good for that, and what’s the best runtime for it?” Then build your rig around that.

They need to give it to the current marketing team. And save some for me.

Where’s all this hate against Wikipedia from?

From:

Most of them from are from the west want to shape their own version of history…

AKA Wikipedia writing something specific people don’t want to hear, therefore the Wikipedia system must be fundamentally broken and biased.

Wikipedia is far from perfect, but they are cognisant of that. That’s kinda the point. English Wikipedia undoubtedly has a Western slant too, but that doesn’t nullify their system that’s held up reasonably well as the internet burns down around them.

…They don’t use it over API?

Rimworld’s DLCs are kinda assumed purchases for the modding scene, too. I feel like this drives a lot of their sales TBH.

Coffee Stain’s another good example on the bigger end.

It does seem like there’s a danger zone behind a certain size threshold. It makes me worry for Warhorse (the KCD2 dev), which plans to expand beyond 250.

Funny thing is Teslas already have something more sophisticated. They could pipe the FSD’s diagnostics to a HUD as a more polished, standard ‘overlay’ for the driver, literally run with the car’s own hardware. You’d think Tesla execs would know about that since it’s literally their business, and predates the LLM craze.

…But no.

This is so stupid.

To me, “AI” in a car would be like highlighting pedestrians in a HUD, or alerting you if an unknown person messes with the car, or maybe adjusting mood lighting based on context. Or safety features.

…Not a chatbot.

I’m more “pro” (locally hostable, task specific) machine learning than like 99% of Lemmy, but I find the corporate obsession with cloud instruct textbots bizarre. It would be like every food corp living and breathing succulents. Cacti are neat, but they don’t need to be strapped to every chip bag, every takeout, every pack of forks.

Not everyone’s a big kb/mouse fan. My sister refuses to use one on the HTPC.

Hence I think that was its non-insignificant niche; couch usage. Portable keyboards are really awkward and clunky on laps, and the steam controller is way better and more ergonomic than an integrated trackpad.

Personally I think it was a smart business decision, because of this:

It doesnt have 2 joysticks so I just buy an Xbox one instead.

No one’s going to buy a steam-branded Xbox controller, but making it different does. And I think what killed it is that it wasn’t plug-and-play enough, eg it didn’t work out of the box with many games.

With respect, this doesn’t make any sense. If you want a joystick controller, just buy an Xbox controller that everything’s compatible with anyway?

The trackpads shine when one needs to emulate a mouse/kb in non-controller games; a nightmare with joysticks.

My sister still has a working one that she treats like a religious artifact, as it’s the best way to play mouse/KB games from the sofa.

I see why they discontinued them though. They need custom configs for most games, and I think most people don’t like that much tweaking.

A lot, but less than you’d think! Basically a RTX 3090/threadripper system with a lot of RAM (192GB?)

With this framework, specifically: https://github.com/ikawrakow/ik_llama.cpp?tab=readme-ov-file

The “dense” part of the model can stay on the GPU while the experts can be offloaded to the CPU, and the whole thing can be quantized to ~3 bits average, instead of 8 bits like the full model.

That’s just a hack for personal use, though. The intended way to run it is on a couple of H100 boxes, and to serve it to many, many, many users at once. LLMs run more efficiently when they serve in parallel. Eg generating tokens for 4 users isn’t much slower than generating them for 2, and Deepseek explicitly architected it to be really fast at scale. It is “lightweight” in a sense.

…But if you have a “sane” system, it’s indeed a bit large. The best I can run on my 24GB vram system are 32B - 49B dense models (like Qwen 3 or nemotron), or 70B mixture of experts (like the new Hunyuan 70B).

DeepSeek, now that is a filtered LLM.

The web version has a strict filter that cuts it off. Not sure about API access, but raw Deepseek 671B is actually pretty open. Especially with the right prompting.

There are also finetunes that specifically remove China-specific refusals. Note that Microsoft actually added saftey training to “improve its risk profile”:

https://huggingface.co/microsoft/MAI-DS-R1

https://huggingface.co/perplexity-ai/r1-1776

That’s the virtue of being an open weights LLM. Over filtering is not a problem, one can tweak it to do whatever you want.

Grok losing the guardrails means it will be distilled internet speech deprived of decency and empathy.

Instruct LLMs aren’t trained on raw data.

It wouldn’t be talking like this if it was just trained on randomized, augmented conversations, or even mostly Twitter data. They cherry picked “anti woke” data to placate Musk real quick, and the result effectively drove the model crazy. It has all the signatures of a bad finetune: specific overused phrases, common obsessions, going off-topic, and so on.

…Not that I don’t agree with you in principle. Twitter is a terrible source for data, heh.

Nitpick: it was never ‘filtered’

LLMs can be trained to refuse excessively (which is kinda stupid and is objectively proven to make them dumber), but the correct term is ‘biased’. If it was filtered, it would literally give empty responses for anything deemed harmful, or at least noticably take some time to retry.

They trained it to praise hitler, intentionally. They didn’t remove any guardrails. Not that Musk acolytes would know any different.

Also a crime. Not just a great game in their niche, but a long history of them.