Skip to content
AI Stack Nest

Interview Q&A

Everything to pick, install and run open models on your own computer.

Interview Q&A

Questions people are asked about open models and running AI locally, with short model answers, plus a short quiz to test yourself. Read the answer only after you have tried to answer in your own words, because explaining an idea out loud is what makes it stick.

Common questions

What is a token?
A small piece of text, often part of a word, that a language model reads and writes. Models have limits and prices measured in tokens.
What is a context window?
The maximum amount of text, in tokens, a model can consider at once, including your prompt and its own answer.
What does an open-weight model mean?
The trained model files are published so you can download and run them. It does not always mean the license is a standard open-source license, so read the terms.
What is quantization?
Storing a model's numbers with fewer bits, for example 4-bit instead of 16-bit, so the file is smaller and uses less memory, usually at some cost in quality.
What is a hallucination?
A confident answer that is false or invented. Check important facts against a reliable source.
What is prompt injection?
Text in a document or web page that tries to give the model new instructions. Treat model input from untrusted sources as untrusted.
Why run a model locally?
Privacy, offline use, no per-request fees and control. The trade-offs are hardware needs and usually weaker models than the largest cloud ones.
What is retrieval-augmented generation?
Searching your own documents for relevant passages and giving them to the model along with the question, so answers are based on that material.
What is a transformer architecture and why do most modern language models use it?
A transformer is a neural network architecture that processes all words in an input simultaneously by computing relationships between every pair of positions. This parallel processing makes training much faster than older sequential architectures that read text one word at a time. The attention mechanism at the core of transformers lets the model weigh how relevant each word is to every other word when generating output. Transformers scaled well as more data and computing power became available, which is why they became the dominant architecture for language tasks.
What is the difference between pretraining and fine-tuning?
Pretraining is the initial large-scale training phase where a model learns general patterns from a broad dataset covering many topics and writing styles. The model develops its core language ability and general knowledge during this phase. Fine-tuning is a subsequent, smaller training process that adapts the pretrained model for a specific task, style, or set of behaviours using a narrower dataset. Pretraining is expensive and slow; fine-tuning is relatively quick and cheap by comparison. Together they allow a single base model to serve many different specialised applications.
What is reinforcement learning from human feedback (RLHF) and why is it used?
RLHF is a training technique where human raters evaluate pairs of model responses and indicate which is better. These preferences are used to train a reward model that predicts what humans will prefer. The language model is then trained using reinforcement learning to generate outputs that the reward model scores highly. RLHF is used because it is difficult to describe in explicit rules what a good response looks like; human preferences are easier to capture than hand-written rules. It has been central to making chat assistants more helpful and less likely to produce harmful content.
What is retrieval-augmented generation (RAG) and what problem does it solve?
RAG is a technique that connects a language model to an external collection of documents at query time. When a question is asked, the system first retrieves the most relevant documents from the collection, then passes those documents along with the question to the model. The model uses the retrieved content to generate a more accurate, grounded answer. RAG solves the problem of models having outdated or missing knowledge, since the external document store can be updated without retraining the model. It is widely used for knowledge-base chatbots and document question-answering systems.
What are embeddings and how are they used in AI systems?
Embeddings are numerical representations of text, images, or other data as lists of floating-point numbers, where items with similar meanings end up close together in that numerical space. They are generated by a model trained to capture semantic similarity. Embeddings are used for semantic search, recommendation systems, and as the retrieval layer in RAG systems, where they allow fast lookup of relevant documents. They are also used internally within language models to represent words and sentences as the model processes them. The key property is that the distance between two embeddings reflects how related the underlying items are.
What does the temperature setting control in a language model?
Temperature controls how random the model's word selection is at each generation step. At a low temperature, the model almost always picks the highest-probability next token, making output predictable and consistent. At a high temperature, lower-probability tokens are selected more often, making output more varied and sometimes more creative, but also less coherent at extremes. A temperature of zero makes the output deterministic: the same input always produces the same output. Most practical applications use a temperature tuned for the specific task, often lower for factual tasks and slightly higher for creative ones.
What is the difference between a base model and an instruction-tuned model?
A base model is the result of pretraining on a large corpus of text. Its primary ability is to predict and continue text in the style of its training data, not to follow instructions or answer questions helpfully. An instruction-tuned model has gone through additional training, often using RLHF or supervised fine-tuning on instruction-response pairs, to make it respond usefully to requests. Most consumer AI chat products are instruction-tuned models. The base model provides the knowledge and language ability; the instruction tuning shapes the behaviour into something useful for conversation.
What is overfitting and how is it addressed?
Overfitting occurs when a model learns the specific patterns in its training data so well that it fails to generalise to new, unseen data. The model performs well on training examples but poorly on anything outside that set. It is a problem because the goal of training is to produce a model that works on new inputs, not just the ones it has already seen. Overfitting is more likely with small datasets or very large models relative to the data available. Common countermeasures include regularisation techniques, dropout during training, and always evaluating the model on a held-out validation set that it did not train on.
What is a vector database and what role does it play in AI systems?
A vector database is a specialised database that stores data as numerical vectors and supports efficient similarity search, finding stored vectors that are closest to a query vector in the mathematical space. It is a key component in RAG systems, where text passages are converted to embeddings and stored; at query time, the most relevant passages for a given question are retrieved by finding the nearest vectors. Standard relational databases are not built for this kind of similarity search and are too slow for the typical scale of document retrieval. Vector databases make it practical to search large document collections by meaning rather than by keyword.
What is the difference between supervised and unsupervised learning?
In supervised learning, the model is trained on data where each example has a label, and the goal is to predict the correct label for new examples. Classification and regression are typical supervised tasks. In unsupervised learning, the model receives data without labels and must find structure on its own, such as grouping similar items or identifying underlying patterns. Language model pretraining is often called self-supervised because the model predicts the next word using the existing text as its own label, without requiring human annotation. The boundary between these categories is not always sharp, and many modern systems combine elements of both.
What is safety alignment and why is it a separate concern from capability?
Safety alignment is the process of training an AI model to follow guidelines, refuse inappropriate requests, and avoid producing harmful or policy-violating content. Alignment shapes what the model will and will not do, as distinct from its raw capability to produce fluent and accurate text. A highly capable but misaligned model can produce accurate but dangerous content. Common alignment techniques include RLHF and constitutional AI, which guide the model toward helpful, honest, and harmless behaviour. Getting alignment right remains an active area of research because defining what counts as acceptable behaviour is itself a complex problem.
What is model distillation and when is it useful?
Distillation is a process where a smaller model, called the student, is trained to reproduce the behaviour of a larger one, called the teacher. The student learns not just from the original training data but from the teacher's output probabilities, which carry more information than hard labels alone. The result is a smaller, faster model that retains much of the teacher's performance. Distillation is useful when you need to deploy capable models in resource-constrained environments, such as on mobile devices, in browser-based applications, or in systems where low latency is essential. It allows the benefits of a large model to reach contexts where the full model would be impractical.
What is a system prompt and how does it affect a conversation?
A system prompt is a set of instructions given to an AI model before the user's conversation begins. It is typically invisible to the end user but shapes how the model behaves throughout the entire conversation. A system prompt might define the model's role, set a specific tone, restrict certain topics, or provide background information the model needs to do its job. Developers use system prompts to customise a general-purpose model for a specific application without additional training. The model treats the system prompt as authoritative guidance and tries to follow it consistently, though it can still be overridden by certain prompt-based techniques.
What is chain-of-thought prompting and why does it improve results on some tasks?
Chain-of-thought prompting is a technique that asks the model to reason through a problem step by step before giving a final answer. Rather than jumping straight to a conclusion, the model writes out its reasoning process, which often leads to more accurate results on tasks involving multi-step logic, mathematics, or complex decision-making. The technique works because the intermediate steps help the model maintain correct reasoning across multiple operations and catch errors before committing to a final answer. It was observed that simply including a phrase like 'let us think step by step' could improve accuracy on reasoning tasks without other changes to the prompt.
What is the purpose of a model card?
A model card is a document published alongside an AI model that describes its intended uses, training methodology, evaluation results, known limitations, and potential risks. It is a transparency practice that helps users and developers understand what the model is suitable for and what it is not. Model cards originated as a way to make AI development more accountable, particularly for models used in high-stakes applications. They help downstream users make informed decisions about whether to use a model and how. Publishing a model card does not guarantee the model is safe, but the absence of one is a signal that less is known about the model's properties and behaviour.
What is the difference between closed-source and open-weight models?
A closed-source model is one where the developer does not release the model's weights or architecture. Users interact with it only through a product interface or programming interface, and the provider controls all access. An open-weight model releases the trained weights publicly, allowing others to download, run, and in many cases modify the model. Open-weight models offer more flexibility and privacy since they can be run locally. Closed-source models are often developed with more resources and may have additional safety layers, while open-weight models give researchers and developers more direct access and control.
What is grounding and why does it matter for reliability?
Grounding means connecting a model's output to real, verifiable information rather than relying solely on what the model learned during training. A grounded response is supported by specific evidence, such as a retrieved document, a database lookup, or a tool call result. Ungrounded responses are generated purely from the model's internal knowledge, which may be outdated, incomplete, or simply wrong. Grounding is important in applications where accuracy matters, such as answering questions about current regulations, specific product details, or recent events. It does not guarantee correctness, but it provides a traceable basis for the answer that can be checked.
What is the agentic loop and what are its main risks?
The agentic loop is the repeated cycle in which an AI agent takes an action, observes the result, and decides what action to take next, continuing until the task is complete. Each step involves the model reasoning about the current state, selecting a tool or action, running it, and processing the result to determine the next step. This enables agents to accomplish multi-step tasks that a single prompt cannot complete. The main risk is that errors can compound across steps: a wrong assumption early in the loop can send the agent down an unproductive path, and actions taken during the loop may be hard to undo. Agents require monitoring and human checkpoints for consequential tasks.
What is synthetic data and what are its limitations?
Synthetic data is artificially generated data used to train or evaluate AI models when real data is scarce, expensive, or sensitive. For example, synthetic conversations can be generated to train a customer service model when real conversations cannot be shared for privacy reasons. Synthetic data can also be used to cover edge cases that occur rarely in real data. The main limitation is representational accuracy: if the synthetic data does not match the real distribution closely, the model trained on it may not perform well on real inputs. Generating high-quality synthetic data is itself a non-trivial task, and the quality of the resulting model depends heavily on the quality of the synthetic data.
What is the purpose of annotation in machine learning?
Annotation is the process of labelling data with human judgments so that a model can learn from them. This might involve marking text as positive or negative in sentiment, drawing boxes around objects in images, or rating which of two AI responses is more helpful. The labels give the model a target to learn toward during training. Annotation quality directly affects model quality: inconsistent or incorrect labels lead to a model that learns the wrong patterns. Annotation is often the most labour-intensive and expensive part of building a specialised AI system, and large amounts of it are required to train models reliably.

Mini quiz

Quick check

1. What is a token in a language model?

Quick check

2. Which Ollama command shows whether a model is using the GPU or CPU?

Quick check

3. What is the default address Ollama listens on?

Quick check

4. What does quantization do?

Quick check

5. Which proxy variable does the Ollama FAQ recommend for downloads?

Quick check

6. What does a language model do when you send it a message?

Quick check

7. What is fine-tuning?

Quick check

8. What does the temperature setting control in a language model?

Quick check

9. What is a system prompt?

Quick check

10. What is the main advantage of retrieval-augmented generation (RAG)?

Quick check

11. What are embeddings used for in AI systems?

Quick check

12. What is RLHF?

Quick check

13. What is a vector database best suited for?

Quick check

14. What is distillation in machine learning?

Quick check

15. What is a training cutoff?

Quick check

16. What is chain-of-thought prompting?

Quick check

17. What is overfitting?

Quick check

18. What is a model card?

Quick check

19. What is the main difference between a base model and an instruction-tuned model?

Quick check

20. What is semantic search?

Quick check

21. What is the agentic loop?

Quick check

22. What is synthetic data?

Quick check

23. What is safety alignment?

Quick check

24. What is a multimodal model?

Quick check

25. What is grounding in AI systems?

Interview Q&A: open-source AI hub | AI Stack Nest