Should I choose an RTX 4090 or A100 for deep learning?

Question

Should I grab an RTX 4090 or hunt down a used A100 for my deep learning setup? I'm starting my masters project here in London next month and I'm gonna be training some heavy NLP models. Budget is around 2500 quid which puts me right in the middle. I read that the 4090 has faster clock speeds but then people say the A100 is better because of the 40gb vram and actual stability for long runs. Im worried the 24gb on the 4090 will bottle neck me pretty fast but the A100s I see on ebay are all used and sketchy. Is the enterprise driver support actually worth the extra headache for a solo researcher or should I just stick to the consumer stuff?

FeralFjord · Accepted Answer

I have been quite satisfied using the NVIDIA GeForce RTX 4090 24GB GDDR6X for my own research projects lately. Like someone mentioned, going with new consumer hardware is usually the more practical path for a student. TL;DR: Buy the 4090. Its much faster and the 24GB limit can be managed with modern software optimizations.

NVIDIA GeForce RTX 4090 24GB GDDR6X: Very high clock speeds, easier to fit in a standard case, and comes with a warranty.

NVIDIA A100 40GB PCIe: Larger memory pool for big batches, but used units are risky and cooling them in a home setup is a nightmare. Honestly, the 4090 works well for heavy NLP if you use bfloat16 or 4-bit quantization. I have no complaints about the stability during long runs. Its just a better value for your 2500 quid.

MysticMarmot · Answer

> Im worried the 24gb on the 4090 will bottle neck me pretty fast but the A100s I see on ebay are all used and sketchy. Sketchy is the right word lol. Most used A100s are passive cooled, so they wont work in a normal PC without serious fan mods. Honestly, the NVIDIA GeForce RTX 4090 24GB GDDR6X is a decent option since it just works. Youll lose VRAM compared to the A100, but you avoid the reliability headaches and power delivery issues of enterprise gear.

MoonlitMantis · Answer

Coming back to this after seeing a few people talk about used gear... honestly id stick with the new consumer stuff. I went through this exact dilemma when I was setting up my home lab for a similar NLP project. Ended up getting a brand new NVIDIA GeForce RTX 4090 24GB GDDR6X and honestly, I couldnt be more satisfied with the stability. A buddy of mine tried the used enterprise route with an old A100 from a reseller and the thing arrived with weird coil whine and eventually died three months later with no warranty to back him up. Total nightmare for a student budget. If you really think you will hit a wall with VRAM, maybe look into getting two NVIDIA GeForce RTX 3090 24GB GDDR6X cards instead of one sketchy A100. Works well for larger batches and you still get 48gb total. Quick tip: use 4-bit quantization to fit larger models. Also, stick to cards with a local warranty since youre in London.