Attention Mechanisms in Neural Networks

1. What is the purpose of masking in attention mechanisms?

To randomly disable neurons during training

To prevent the model from attending to future tokens in autoregressive tasks

To reduce the dimensionality of input embeddings

To apply batch normalization across attention heads

Submit

3. What role does the feedforward network play in a Transformer block?

It replaces the attention mechanism in some layers

It applies non-linear transformations after attention

It computes attention weights directly

It reduces the embedding dimension

Submit

6. What is the computational complexity of standard multi-head attention relative to sequence length?

O(n) linear in sequence length

O(n log n) logarithmic

O(n²) quadratic in sequence length

O(n³) cubic in sequence length

7. The scaling factor in scaled dot-product attention (typically 1/√d_k) prevents ____.

Exploding gradients from large dot products

Overfitting to training data

Memory allocation errors

Softmax saturation with very small gradients

Submit

9. Vision Transformers (ViTs) apply attention mechanisms to images by ____.

Treating image patches as sequence tokens

Replacing all convolutional layers with attention

Using separate attention for each pixel

Applying 3D attention across color channels

Submit

11. What is the primary function of an attention mechanism in neural networks?

To increase model size and parameter count

To allow the model to focus on relevant parts of input data

To replace all recurrent layers in a network

To reduce computational cost by removing hidden layers

Submit

13. Which of the following is a key advantage of attention mechanisms over RNNs?

Better handling of very long sequences due to parallelization

Lower memory requirements during training

Automatic feature extraction without learned parameters

Guaranteed convergence in fewer epochs

Submit

15. In multi-head attention, what is the benefit of using multiple attention heads?

It allows the model to attend to different representation subspaces

It eliminates the need for positional encoding

It reduces overfitting by random subsampling

It guarantees faster inference time

16. What is the Transformer architecture primarily based on?

Recurrent neural networks with gating mechanisms

Convolutional layers stacked in parallel

Multi-head self-attention and feedforward networks

Bidirectional LSTM encoders

17. Which operation is used to compute attention weights in the scaled dot-product attention?

Softmax applied to Query-Key dot products, scaled by dimension

Element-wise multiplication of Query and Value vectors

Cross-entropy loss between Query and Key

L2 normalization of attention vectors

Submit

19. In the attention mechanism, what do the Query, Key, and Value matrices represent?

Different layers of a convolutional network

Components used to compute attention weights and output

Input, hidden, and output states of an RNN

Regularization parameters for dropout

DataAI Attention Mechanism in Neural Networks Quiz

1. What is the purpose of masking in attention mechanisms?

2.

What first name or nickname would you like us to use?

2. Attention mechanisms have become fundamental in modern NLP because they enable models to ____.

3. What role does the feedforward network play in a Transformer block?

4. In the encoder-decoder attention layer of a Transformer, queries come from the decoder while keys and values come from ____.

5. Efficient attention variants like Linformer and Performer aim to reduce ____.

6. What is the computational complexity of standard multi-head attention relative to sequence length?

7. The scaling factor in scaled dot-product attention (typically 1/√d_k) prevents ____.

8. In BERT, bidirectional attention allows the model to ____.

9. Vision Transformers (ViTs) apply attention mechanisms to images by ____.

10. The attention weight for a query-key pair is computed using a softmax function primarily to ____.

11. What is the primary function of an attention mechanism in neural networks?

12. Cross-attention mechanisms are commonly used to ____.

13. Which of the following is a key advantage of attention mechanisms over RNNs?

14. Positional encoding in Transformers serves to ____.

15. In multi-head attention, what is the benefit of using multiple attention heads?

16. What is the Transformer architecture primarily based on?

17. Which operation is used to compute attention weights in the scaled dot-product attention?

18. Self-attention differs from standard attention in that it ____.

19. In the attention mechanism, what do the Query, Key, and Value matrices represent?