Why Context Matters
The problem with static vectors
In the previous course, we learned that words can be represented as vectors. But there's a fundamental limitation: the same word always gets the same vector.
Consider the word "bank":
"I went to the bank to deposit money"
"I sat on the river bank watching fish"
Same word. Completely different meanings. With static word vectors, "bank" would have identical representation in both sentences. That's a problem.
See it for yourself
The "bank" Problem
Click on each sentence to see how context changes the meaning of "bank".
Context is everything
Humans naturally understand that "bank" means different things in different contexts. We look at surrounding words like "deposit", "money" (financial) vs "river", "fish" (nature) to determine meaning.
This is exactly what Attention does. It allows a model to look at all other words in a sentence and determine which ones are most relevant for understanding each word.
Key insight: The meaning of a word depends on its context. Attention is the mechanism that captures this context.
Key Takeaways
- Static word vectors give the same representation regardless of context
- The same word can have completely different meanings
- Attention solves this by considering surrounding words