Transformer Architecture for Generative AI

1. What is the purpose of masking in transformer decoders during training?

To prevent attention to future tokens

To ignore low-confidence predictions

To reduce model size

To speed up inference

2. Which NVIDIA optimization technique quantizes model weights to reduce inference memory?

Quantization-aware training

Dropout regularization

Layer normalization

Attention masking

3. In generative AI pipelines, what does 'temperature' control during inference?

Randomness in token selection

Model training speed

Attention head count

Embedding dimension

4. What is the main benefit of using residual connections in transformer blocks?

Enables deeper models to train effectively

Reduces inference latency

Decreases model parameters

Simplifies attention computation

5. Which scaling factor is applied to attention scores before softmax in transformers?

1 / sqrt(d_k)

1 / d_model

Sqrt(d_model)

Number of heads

6. In transformer models, what does the query-key-value mechanism compute?

Attention scores for weighted aggregation

Loss function values

Gradient updates

Embedding vectors only

7. What is the primary advantage of using mixed precision training in NVIDIA systems?

Faster computation with reduced memory usage

Improved model accuracy only

Smaller vocabulary size

Simpler model architecture

8. Which technique improves transformer efficiency by reducing attention computation?

Sparse attention

Increased embedding dimension

More decoder layers

Higher learning rate

9. What does 'context window' refer to in large language models?

Maximum sequence length the model can process

Training batch size

Number of attention heads

Model parameter count

10. In NVIDIA's generative AI frameworks, which library is commonly used for transformer implementations?

TensorRT

CUDA

Megatron-LM

CuDNN only

11. What is the primary function of the attention mechanism in transformers?

To weight input sequences based on relevance

To reduce model parameters

To increase computational speed

To eliminate the need for embeddings

12. Which training objective is commonly used for large language models based on transformers?

Next token prediction

Image classification

Supervised classification only

Reinforcement learning exclusively

13. What is the computational complexity of self-attention with respect to sequence length?

O(n)

O(n log n)

O(n²)

O(2ⁿ)

14. In the context of generative AI, what does 'autoregressive' mean?

Predicting next token based on previous tokens

Using bidirectional context only

Training without labels

Parallel token generation

15. What role does the feed-forward network play in a transformer block?

Applies non-linear transformations to attention output

Computes attention weights

Reduces sequence length

Generates embeddings

16. Which normalization technique is typically used in transformer layers?

Batch normalization

Layer normalization

Instance normalization

Group normalization

17. What is the purpose of positional encoding in transformers?

To add sequence order information

To compress model weights

To normalize input data

To apply dropout regularization

18. In a transformer encoder-decoder architecture, what does the decoder receive as input?

Raw source text

Encoder output and target embeddings

Only target tokens

Attention weights only

19. What does 'multi-head attention' enable in transformer models?

Multiple attention patterns simultaneously

Faster inference only

Reduced model size

Single representation learning

20. Which component allows transformers to process sequences in parallel rather than sequentially?

Recurrent connections

Self-attention

Convolution layers