CHOOSE YOUR FIRST LOCAL AI MODEL
MATCH THE MODEL TO YOUR TASK AND HARDWARE
Start with a clear task and a model your computer can comfortably run. This guide focuses on choosing a local text model for work such as drafting, summarizing, brainstorming, or assisting with code.
You’ll learn what to check before downloading, how to compare models using your own test prompts, and when to try a different size or model family.
If you haven’t run a local model yet, begin with Run Your First Local AI Model, then return here to explore your next choice.
START WITH YOUR TASK
DEFINE WHAT A USEFUL ANSWER LOOKS LIKE
Choose one task you want help with, such as rewriting a paragraph, summarizing notes, brainstorming ideas, or explaining a short piece of code.
Write down what a successful response should do. You might need it to follow a specific format, preserve important details, use a particular tone, or produce code you can test.
Prepare three short prompts based on that task. Include one straightforward example, one with specific instructions, and one that previously gave you trouble.
Keep these prompts so you can compare models using the same inputs. Judge the results against your requirements, and check factual claims or generated code before using them.
CHECK YOUR HARDWARE
PLAN AROUND AVAILABLE MEMORY
Record your system RAM, GPU model and VRAM, and free disk space. Leave room for your operating system and the other applications you expect to use alongside AI.
A model’s download size tells you how much storage its files need. Running it also requires memory for the model, conversation context, and supporting operations, so download size alone won’t tell you whether it will fit.
Check your model runner’s GPU compatibility documentation. Depending on your hardware and the runner, a model may use GPU memory, system memory, or both. Performance will vary with that setup.
Start with a small model and short prompts, then check memory use and response speed before moving to a larger model or longer conversations.
UNDERSTAND THE MODEL NAME
CHECK THE FAMILY, SIZE, AND VARIANT
Model names often identify a family, a generation, and a size. In the Ollama name gemma3:1b, gemma3 identifies the Gemma 3 family, while 1b selects a variant with approximately one billion parameters.
Parameters are numerical values learned during training. Parameter count helps describe a model’s scale, but it doesn’t tell you exactly how much memory it needs or how well it will perform on your task.
Other names may include labels for instruction tuning, coding, reasoning, or quantization. Naming conventions vary, so read the model’s description rather than relying on its name alone.
Use the complete model name and tag from the model page when downloading. Record that name with your test results so you know which variant you evaluated.
READ THE MODEL PAGE
CONFIRM WHAT YOU’RE DOWNLOADING
Before downloading, check the model’s intended tasks, supported inputs, available variants, and any requirements for your model runner.
For a first conversation, choose a model designed for chat or instruction following. If you need image understanding, confirm that the specific variant supports images; models in the same family may have different capabilities.
Check the context limit, which describes how much input and output the model can handle in one request. Your runner’s configured limit may be lower, and longer context can require more memory.
Read the license and usage terms, especially if you plan to use the model for commercial work or redistribute it. Follow links to the original publisher’s model documentation when the download page leaves an important detail unclear.
CHOOSE A STARTING SIZE
ESTABLISH A BASELINE BEFORE GOING LARGER
Begin with a small model that runs comfortably on your computer. If you completed our first walkthrough, use Gemma 3 1B as your baseline and save its responses to your test prompts.
When those responses fall short, try a larger variant or another model suited to your task. Compare answer quality, response speed, and memory use before deciding whether to keep it.
A larger parameter count doesn’t guarantee a better result across different model families. Training, specialization, and the model’s stored precision also affect the comparison.
Choose the smallest model that meets your requirements at an acceptable speed. Leave enough memory available for the other tools in your workflow.
CHECK FORMAT AND QUANTIZATION
CHOOSE A VARIANT YOUR RUNNER SUPPORTS
Confirm that your model runner supports the model’s architecture and file format. For your first comparisons, choosing an entry from the runner’s own model library can simplify setup.
Quantization reduces the numerical precision used to store model values. It can lower storage and memory requirements, though its effect on answer quality and speed varies by model and implementation.
Labels such as Q4 or Q8 often indicate quantization choices. Read the variant’s details rather than treating those labels as a complete description of quality or hardware requirements.
Record the quantization alongside the model name when comparing results. If a model doesn’t fit comfortably, a supported quantized variant may be worth testing before moving to a smaller model.
COMPARE WITH THE SAME PROMPTS
TEST QUALITY AND SPEED TOGETHER
Run your three test prompts on each model. Start a fresh conversation for each independent test so earlier messages don’t influence the comparison.
Keep your instructions and settings as consistent as possible. Record the full model name, quantization, runner version, and any settings you changed.
Compare whether the responses follow instructions, preserve important details, and meet your task requirements. Note how long you wait for an answer and whether your other applications remain responsive.
Repeat any surprising result before drawing a conclusion. Responses can vary between runs, so choose based on several useful tests rather than one impressive answer.
KEEP A WORKING CHOICE
UPGRADE WHEN IT IMPROVES YOUR WORK
Once a model meets your needs, keep its name, settings, and test results with your project notes. Use it for real tasks to see whether it remains useful beyond your initial tests.
A newer release is a reason to compare again, but you don’t need to replace a working model immediately. Test the new option against the same prompts and look for improvements that matter to your workflow.
Revisit your choice when your tasks change, your hardware changes, or the model repeatedly falls short. Check current availability and compatibility before rebuilding a setup from older notes.
Keep a working baseline while experimenting so you can return to a setup you understand.
SOURCES & VERIFICATION
OFFICIAL DOCUMENTATION
Hugging Face — Quantization Concepts
Last checked against official documentation: October 5, 2026.
Model availability, hardware support, and runner behavior may change. Check the current documentation for the specific model and variant you plan to use.