Skip to content
AI Stack Nest
BeginnerLesson 1 of 3 · 1 min read

What is a local AI model?

Understand tokens, weights, open-weight models and why you might run a model on your own computer.

Models are files

An AI language model is a very large file full of numbers, called weights, produced by training on large amounts of text. When you ask a question, a program loads those weights into memory and uses them to generate an answer one small piece at a time. Those pieces are called tokens. A token is often part of a word.

Cloud and local

With a cloud assistant, your text travels to the company's servers and the answer comes back. With a local model, the weights are on your own disk and the answer is generated on your own hardware.

Local models are useful when you want privacy, offline use, or no per-request fees. The trade-offs are that you need enough memory and disk space, and the models that fit on a laptop are smaller than the biggest cloud models.

Open-weight is not always open source

Companies publish some models as open-weight, which means you can download them. Licenses differ. For example, Mistral 7B Instruct v0.3 uses Apache 2.0, while Llama 3.1 and Gemma use their own custom terms. Always read the license on the model card before using a model in a product.

What runs the model

You need a program to load the weights and talk to them. Ollama is one such program. It downloads models, runs them, and offers a local API on your computer. In the next lessons you will check how much memory you need and run your first model.

Quick check

What does an open-weight model let you do?

Quick check

What is a token?

0 of 3 lessons done in this track