Positional Encoding

​

Positional Encoding Overview

Positional encoding adds information about the position of each token in the input sequence. This is crucial for models to understand the order of words, as Transformers do not inherently capture sequence information.

​

Techniques

• Sinusoidal Functions: A common method for generating positional encodings.
• Learned Embeddings: Trainable embeddings that capture positional information.