Private & Local A.I. Tools (LLM)

CCarter

CCarter

Final Boss ®
Moderator
BuSo Pro
Boot Camp
Digital Strategist
Joined
Sep 15, 2014
Messages
4,372
Likes
8,897
Degree
8
If you like A.I. but don't want your queries going to random servers and being saved for who knows how long this is for you.

The biggest problem with A.I./LLM like ChatGPT is the privacy aspect, since all the data gets sent to ChatGPT/Microsoft and their servers. What happens afterwards is anyone's guess.

That's where Private / Local LLM Tools come into play. You download the open source LLM libraries or whatever within your desktop APP and can run queries privately. They all have Linux, Mac OS X, and Microsoft Windows versions.

Note: AnythingLLM - this one lets you upload your own PDFs and documents and then run questions and queries against the documents. The tests I ran have been extremely successful so that's something to note.

I actually know very little about each model, like LLAMA or DeepSeek. I'm not an expert in any of this but as I continue to explore I'll post updates.

These are some of the LLM for Desktop that you can use to run your queries locally and privately. Most of the credit for this goes to @SmokeTree:

--

AnythingLLM: https://anythingllm.com/

This one allows you to upload PDFs and other documents then ask the A.I. questions against the documents.

w8ouQoG.png

--

LM Studio: https://lmstudio.ai/

AF2xf2d.png

--

Msty: https://msty.app/

l2cQD4U.png

--

Ollama: https://ollama.com/

(I haven't tried this one)

dBiZtge.png

--

Since they all run locally and privately, none of the above can tell you what time it is in London at the moment unless you enable them. So far I have not been successful in enabling that for any of them, but I haven't tried that hard.

They all can code pretty decently as well.
 
I tested Ollama a few months ago, it's pretty cool to run LLMs locally but keep in mind that you need a very powerfull computer to run large models.

Smaller models will run just fine but they won't compare with what you are used to on ChatGPT, Claude, etc.
 
Last edited:
infotech said:
What are the RAM/GPU requirements for running locally?
Click to expand...
If you want as close to the live examples you would get from ChatGPT or Claude...

A ton, more then even I want to afford.

Or you could get a small quantified model... which wont be the same result.. BUT MAY WORK for you needs.

Only testing will let you know which model size you need
 
infotech said:
What are the RAM/GPU requirements for running locally?
Click to expand...
Depends on the model. Take a look at DeepSeek for example, you have several options to choose from:

b5DWk30.png


The actual model is the one with 671 billion parameters. Makes no sense to try to run that locally. Then you have smaller models that they "distilled" from the original like they say.

To give you an idea:

Memory requirements​

  • 7b models generally require at least 8GB of RAM
  • 13b models generally require at least 16GB of RAM
  • 70b models generally require at least 64GB of RAM

 
Smith said:
would you ideally run a separate device?
Click to expand...
Me personally, yes.

I see builds on YouTube where they have 4-5 5090 GPU cards running.. but I also know the M3 Ultra and M4 apples chips just got released on Mac Studios too which might be fine to run.. but the Mac Studio you'd need with M3 Ultra to run this well would be north of $10k

To clarify, Im talking about running the 671b model
 
You must log in or register to reply here.
Back