Lecture 2 - Ingredients of intelligence
| ← Back to AICS | Next: Lecture 3 → |
This lecture unpacks how key-value memory works, how it is implemented in the transformer, and how it relates to fast associative memory in the brain. It explains how large neural networks allow abstractions to be formed, and discusses the distinction between in-weight and in-context learning, and how the latter may explain sample efficient learning in humans.
What you need to understand:
- Key-value memory and the transformer network
- What it means for language to be “grounded”
- Abstraction in neural representations (in LLMs or brains)
- The distinction between in-weight and in-context learning; sample-efficient learning
- Structure learning and neural scaffolds in brains and machines
Sample essay questions:
What are the computational principles that underpin modern AI’s success, and how do they mirror those of biological brains?
Discuss the difference between in-context and in-weight learning in transformer networks. Is this distinction useful for understanding natural intelligence?