Sliding Window Attention

What is Sliding Window Attention?

Sliding Window Attention (SWA) restricts each token's attention to a fixed-size local window of neighboring tokens, reducing the quadratic cost of full self-attention while preserving local context useful for long sequences.

How it works in practice

Long-context transformers use SWA (sometimes with periodic global tokens) so models can scale to longer documents without attending to every past token at every layer.

Key points

• Trade global token interactions for compute and memory savings.
• Combine with sparse or dilated windows when longer range is needed.
• Validate quality on tasks that need distant dependencies.

Related Giskard articles

• A practical guide on AI security and LLM vulnerabilities

Secure long-context LLM apps

Long windows expand the attack surface for prompt injection in retrieved context—test with Giskard security suites. See practical LLM security.

Further reading: Longformer paper (sliding window attention).