Attention Mechanism
A neural network component that allows models to focus on relevant parts of input by computing weighted combinations.
In more detail
Attention computes relevance scores between query and key elements, uses them to weight values, producing outputs that emphasize important information. Self-attention attends to positions in the same sequence. Enables transformers to capture long-range dependencies efficiently.
Examples
- Self-attention in Transformers
- Cross-attention in Seq2Seq models
- Multi-head attention
- Positional attention in images
Related terms
Ready to Build Production AI Systems?
Join a small cohort learning directly from an engineer who's shipped AI at Atlassian and PhonePe scale.
20 weeks · Weekend live classes · 1:1 mentorship · 20+ projects.
20 Weeks
Live Weekend Classes
< 30 Seats
Small Batch
Next Soon
Get priority access
7-Day
Money-Back Guarantee
Taught by Debasish Maji · Senior AI Engineer · Ex-Atlassian (Rovo Agent) · Ex-PhonePe (550M+ users)
Get notified when the next batch opens + free AI resources