Late to this post, but shoot for and AMD Strix Halo or Nvidia Digits mini PC.
Prompt processing is just too slow on Apple, and the Nvidia/AMD backends are so much faster with long context.
Otherwise, your only sane option for 128K context in a server with a bunch of big GPUs.
Also… what model are you trying to use? You can fit Qwen coder 32B with like 70K context on a single 3090, but honestly its not good above 32K tokens anyway.
I wonder if those other “spammy” adblockers do precisely this. Insert affiliate links.
Doesn’t Brave already swap some ads for their own?
It’s probably not big if it’s included in the driver download and run in real-time so quickly. Not big enough to worry about anyway.
That’s so corporate. Everyone’s big idea boils down to being a service instead of a seller (because that’s theoretically more profitable), but it sucks.
He’ll probably show up at Ubisoft or EA, lol.
Or maybe even back at Microsoft to lead their handheld launch.
If the alternative is nothing, that’s fine.
And if you need Facebook for work, fine.
But there is no reason to endure being on hellish social media. We lived without it for thousands of years, and you are (IMO) deluding yourself if you think participating brings even an ounce of change. The biggest impact you can make is not being on there and robbing Facebook (or whoever) of the engagement boost you give it.
Off topic, but I am jealous of that handle.
The “piracy database” in question is: https://en.m.wikipedia.org/wiki/Library_Genesis
Screw scholarly journals and giant publishers for squeezing cash out of already-thin academics while being shitty gatekeepers, letting complete trash through while blocking others (among other things). They’re greedy, stagnant, exploitive middlemen.
Doesn’t make what Zuck\Facebook did OK, but I have zero sympathy for the monopolistic “victims” here, especially since it’s going to open weight models one can use for free.
Unfortunately Nvidia is, by fair, the best choice for local LLM coder hosting, and there are basically two tiers:
Buy a used 3090, limit the clocks to like 1400 Mhz, and then host Qwen 2.5 coder 32B.
Buy a used 3060, host Arcee Medius 14B.
Both these will expose an OpenAI endpoint.
Run tabbyAPI instead of ollama, as it’s far faster and more vram efficient.
You can use AMD, but the setup is more involved. The kernel has to be compatible with the rocm package, and you need a 7000 card and some extra hoops for TabbyAPI compatibility.
Aside from that, an Arc B570 is not a terrible option for 14B coder models.
It’s not functional yet.
Yep. I didn’t mean to process shame you or anything, just trying to point out obscure but potentially useful projects most don’t know about :P
Unfortunately that’s not really relevant to LLMs beyond inserting things into the text you feed them. For every single word they predict, they make a pass through the multi-gigabyte weights. Its largely memory bound, and not integrated with any kind of sane external memory algorithm.
There are some techniques that muddy this a bit, like MoE and dynamic lora loading, but the principle is the same.
A1111
Eh, this is a problem because the “engine” is messy and unoptimized. You could at least try to switch to the “reforged” version, which might preserve extension compatibility and let you run features like torch.compile.
Oh you should be able to batch the heck out of that on a 4080. Are you not using HF diffusers or something?
I’d check out stable-fast if you haven’t already:
https://github.com/chengzeyi/stable-fast
VoltaML is also old at this point, but it has really fast AITemplate implementation for SD 1.5: https://github.com/VoltaML/voltaML-fast-stable-diffusion
Oh, 16GB should be plenty for SDXL.
For flux, I actually use a script that quantizes it down to 8 bit (not FP8, but true quantization with huggingface quanto), but I would also highly recommend checking this project out. It should fit everything in vram and be dramatically faster: https://github.com/mit-han-lab/nunchaku
You don’t want it to anyway, as “automatic” spillover with an LLM painfully slow.
The RAM/VRAM split is manually configurable in llama.cpp, but if you have at least 10GB VRAM, generally you want to keep the whole model within that.
Try a new quantization as well! Like an IQ4-M depending on the size of your GPU, or even better, an 4.5bpw exl2 with Q6 cache if you can manage to set up TabbyAPI.
Depends which 14B. Arcee’s 14B SuperNova Medius model (which is a Qwen 2.5 with some training distilled from larger models) is really incrtedible, but old Llama 2-based 13B models are awful.
No, all the weights, all the “data” essentially has to be in RAM. If you “talk to” a LLM on your GPU, it is not making any calls to the internet, but making a pass through all the weights every time a word is generated.
There are system to augment the prompt with external data (RAG is one word for this), but fundamentally the system is closed.
Decep is right.
You’re basically implying the bad outweighs the good to the point that it won’t give them any ideas.
The bad may be very amplified now, but I think the TikTok crowd in particular and US history skews towards “majority good”