Lexicon / CONCEPT
Inference
The process of running input data through a trained AI model to generate predictions or outputs, as opposed to training the model.
Inference is when a trained model does its job — you send it a prompt, and it generates a response. Every time you chat with an AI, you’re running inference.
Inference vs. Training
- Training: The model learns from data (expensive, done once)
- Inference: The model applies what it learned (fast, done per request)
What Affects Inference Cost
- Input tokens — how much text you send
- Output tokens — how much text the model generates
- Model size — larger models cost more per token
- Streaming — delivers tokens as they’re generated instead of waiting for the full response