Which GPU is best for training large language models?

Just sharing my experience: I went through this exact same dilemma last year. I was basically torn between the consumer hype and those massive enterprise buffers. Honestly, I started out trying to be super budget-conscious and picked up a used NVIDIA GeForce RTX 3090 24GB for around $700 on eBay. It’s actually been a beast for PEFT and QLoRA runs with stuff like Mistral.

But then I hit a wall where 24GB just wouldn't cut it for what I wanted to do, so instead of dropping $4k on an A6000, I went the "franken-server" route. I found a deal on two NVIDIA Tesla P40 24GB cards for like $170 each. They're older and slower (GDDR5, ugh), but having 48GB of total VRAM for under $400 was a total game changer for experimentation. It kinda depends on your patience—the 3090 is way faster, but those cheap older enterprise cards let me load way bigger models without OOM errors, right? Anyway, that's just how I saved some cash while learning the ropes. gl!

> I've been eyeing the RTX 3090/4090 because of the 24GB VRAM, but I'm worried about whether that's enough for parameter-efficient fine-tuning (PEFT) or if I should look into enterprise options like the A6000.

Sooo I've been messing around with local LLMs for a few years now, and honestly? I've had a bit of a love-hate relationship with my setup. I started out thinking I could get away with cheaper cards, but quickly realized that VRAM is literally EVERYTHING when it comes to training.

In my experience, here is how the options actually stack up for PEFT:

NVIDIA GeForce RTX 3090 24GB vs NVIDIA GeForce RTX 4090 24GB vs NVIDIA RTX A6000 48GB

Basically, the NVIDIA GeForce RTX 3090 24GB is still the king of price-to-performance for training. It has the same 24GB VRAM as the 4090 but costs way less on the used market. But honestly, even with 24GB, you're gonna hit a wall fast if you wanna fine-tune Meta Llama 3 70B or even the larger Mistral variants without heavy quantization. I tried running some 4-bit LoRA training on a single 3090 and it was... okay? But slow.

If you have the budget, the NVIDIA RTX A6000 48GB is a beast because that extra VRAM lets you use larger batch sizes, which is SO important for stability. Unfortunately, I had issues with the price tag on that one—it's just hard to justify for a hobbyist.

Wait no, I take that back... if you're serious, look for a used NVIDIA RTX 3090 24GB (or two!) and link them up. It's kinda annoying to set up, but having 48GB total across two consumer cards is highkey better than overpaying for a single enterprise card imo. But yeah, if you just want one card that "just works" and fits in a normal case, the NVIDIA GeForce RTX 4090 24GB is the fastest, even if the VRAM limit is still a bummer. Good luck with the build! 👍

Quick question - what's your actual budget limit? honestly, i was disappointed with single-card setups for anything bigger than 7B models. basically, would you consider dual NVIDIA GeForce RTX 3090 24GB cards instead of one expensive NVIDIA RTX A6000 48GB? id love to help but need to know your cap first!

tbh i have been running my rig for about half a year now and the one thing i wish i knew before starting is how much the power bill and noise actually matter for long term ownership if ur gonna be training every night... i actually started with a consumer setup but i recently moved to a NVIDIA RTX 5000 Ada Generation and it has been a total game changer for me. even though i am still kinda new to this whole machine learning thing i noticed that 32GB of vram is like the perfect sweet spot for fine tuning stuff like Google Gemma 2 9B or Microsoft Phi-3.5-mini-instruct without hitting those annoying out of memory errors you get on 24GB cards. plus it only uses like 250 watts so it stays way cooler than a 4090 which is great if u dont want ur office to feel like a desert lol. i have been looking at my logs and here is what i found out after 6 months of doing this:

• consumer cards like the 4090 are fast but the fans can get reallyyy loud during 12 hour training runs which gets old fast
• the 32GB on the 5000 ada lets u use slightly bigger batch sizes or longer context windows which i think helps the model learn better but i am still trying to figure out all the math on that
• enterprise drivers seem way more stable for long training sessions compared to the game ready ones i was using before basically i havent had a single crash since i switched just something to think about if ur budget allows for it because it might save u some headaches later on especially if ur worried about long term reliability and keeping ur room cool!

🙌

> I've been eyeing the RTX 3090/4090 because of the 24GB VRAM, but I'm worried about whether that's enough for parameter-efficient fine-tuning (PEFT) or if I should look into enterprise options like the A6000.

yo just saw this thread and honestly?? i feel u on the dilemma. i remember when i first started fine-tuning things like Mistral and some other open-weights models, i was sooo hyped to just plug in the fastest consumer card i could find and let it rip... but i actually almost fried my motherboard cuz i didnt realize how much heat these things dump during a long training run. basically i think you gotta be really careful about the power draw and reliability aspects before you drop a few grand.

i mean i'm not 100% sure if you strictly *need* those super expensive enterprise cards, but iirc those pro-grade cards are way safer for 24/7 workloads. they're built to handle the constant thermal stress without melting a connector, you know? i've heard some horror stories about the power cables on those top-tier consumer cards if they aren't seated perfectly.

if youre gonna stick with the consumer stuff for the price-to-performance, just PLEASE make sure you have a massive, high-quality power supply and like... literally all the fans. seriously. i would suggest looking into the workstation versions though if you can find a deal, even if the raw speed is a bit lower. the peace of mind knowing your house wont burn down while you sleep is worth a lot tbh!! plus they usually have way more vram which helps with those larger models anyway... just be careful and maybe talk to someone who builds servers before you commit. gl with the training runs!! peace

This ^

Regarding what #3 said about 🙌 - honestly, me too. I have been stuck in this exact same loop for months now and its actually exhausting. I am really chasing that high-end performance for training but i am just so paranoid about the stability of consumer hardware for these massive LLM runs. One day i think i have it figured out, then i read a post about thermal throttling and i am back to square one. I am really trying to figure out if the performance gains on the top consumer cards are worth the risk of them crashing mid-way through a training session. Before i dive into what i have seen, are you planning to run these training jobs back-to-back or just occasionally? I want to know if you are as worried about the long-term hardware reliability as i am or if you are just looking for raw speed.

Any updates on this?

Facts.