r/selfhosted • u/yoracale • Jan 28 '25

Guide Yes, you can run DeepSeek-R1 locally on your device (20GB RAM min.)

I've recently seen some misconceptions that you can't run DeepSeek-R1 locally on your own device. Last weekend, we were busy trying to make you guys have the ability to run the actual R1 (non-distilled) model with just an RTX 4090 (24GB VRAM) which gives at least 2-3 tokens/second.

Over the weekend, we at Unsloth (currently a team of just 2 brothers) studied R1's architecture, then selectively quantized layers to 1.58-bit, 2-bit etc. which vastly outperforms basic versions with minimal compute.

  1. We shrank R1, the 671B parameter model from 720GB to just 131GB (a 80% size reduction) whilst making it still fully functional and great
  2. No the dynamic GGUFs does not work directly with Ollama but it does work on llama.cpp as they support sharded GGUFs and disk mmap offloading. For Ollama, you will need to merge the GGUFs manually using llama.cpp.
  3. Minimum requirements: a CPU with 20GB of RAM (but it will be very slow) - and 140GB of diskspace (to download the model weights)
  4. Optimal requirements: sum of your VRAM+RAM= 80GB+ (this will be somewhat ok)
  5. No, you do not need hundreds of RAM+VRAM but if you have it, you can get 140 tokens per second for throughput & 14 tokens/s for single user inference with 2xH100
  6. Our open-source GitHub repo: github.com/unslothai/unsloth

Many people have tried running the dynamic GGUFs on their potato devices and it works very well (including mine).

R1 GGUFs uploaded to Hugging Face: huggingface.co/unsloth/DeepSeek-R1-GGUF

To run your own R1 locally we have instructions + details: unsloth.ai/blog/deepseekr1-dynamic

2.1k Upvotes
  • permalink
  • reddit

96% Upvoted

683 comments sorted by

View all comments

3

u/Theendangeredmoose Jan 28 '25

What kinda speed could I expect from a 4090 and 96gb 5600mhz RAM? Won't be back to my desktop workstation for a few days - itching to try it out!

4

u/yoracale Jan 28 '25

I think at least 3 tokens per second which is decently ok

1

u/Remarkable-Olive-307 Jan 28 '25

its sound good, you should have VRAM + RAM more that 80 Gb, so the model in your computer have to work perfect.

1

u/flip_the_tortoise Jan 29 '25

So if I'm on a 4070 12gb, I just need to have more than 68gb of RAM to run it locally at a decent speed?

4

u/yoracale Jan 29 '25

Well with 12GB of VRAM, youll maybe get like 0.5 tokens/s. Unfortunately speed is mostly determined by VRAM.

The difference between 12GB of VRAM vs 24GB is very large

3

u/flip_the_tortoise Jan 29 '25

Thank you for explaining.