AWS ML Model Monitoring & Maintenance Quiz

1. How does SageMaker Clarify contribute to model monitoring and maintenance?

It automatically retrains models

It detects feature importance shifts and potential bias in predictions

It stores model artifacts

It manages endpoint scaling

2. What is the significance of maintaining version control for ML models in production?

It has no practical impact

Enables rollback to previous versions if new model performs poorly and tracks model evolution

Only required for compliance

Reduces endpoint latency

3. Which approach best mitigates the risk of deploying a model with high drift?

Ignore drift signals and continue serving predictions

Establish drift thresholds, trigger alerts, and implement automated retraining pipelines

Manually check metrics monthly

Deploy multiple models simultaneously

4. In SageMaker, what role does the Data Capture feature play in model monitoring?

Stores training data permanently

Records input features and predictions for offline analysis and drift detection

Encrypts model artifacts

Manages endpoint autoscaling

5. What is the primary limitation of only monitoring model accuracy metrics in production?

Accuracy is the only metric that matters

Accuracy alone doesn't detect data drift or feature changes affecting future predictions

Accuracy metrics are not available in CloudWatch

Accuracy monitoring is too expensive

6. How does SageMaker Model Monitor detect feature attribution drift?

By comparing model predictions only

By analyzing statistical differences in feature values and their distributions over time

By monitoring endpoint latency

By tracking model training iterations

7. What should be included in a model monitoring strategy for compliance and audit purposes?

Only prediction accuracy logs

Baseline configurations, drift metrics, retraining events, and decision logs

Training data only

Model file checksums

8. Which AWS service integrates with SageMaker for automated model retraining pipelines?

AWS Glue

Amazon SageMaker Pipelines and EventBridge

AWS CodeBuild only

Amazon S3 only

9. In model maintenance, what does 'concept drift' specifically refer to?

Changes in feature distributions

Changes in the relationship between features and target variable over time

Model file corruption

Endpoint configuration changes

10. What is the primary advantage of using canary deployments for model updates?

Faster training time

Gradual rollout to subset of traffic to detect issues before full deployment

Lower storage costs

Automatic model versioning

11. Which AWS service is primarily used to monitor data quality and model performance drift in production ML models?

Amazon CloudWatch

Amazon SageMaker Model Monitor

AWS Lambda

Amazon S3 Analytics

12. Which CloudWatch metric indicates potential performance issues with a SageMaker endpoint?

ModelLatency and ModelInvocations

ModelAccuracy only

ModelSize

ModelTrainingTime

13. What is the relationship between model bias and fairness monitoring?

They are unrelated concepts

Bias monitoring detects systematic errors favoring certain groups; fairness ensures equitable predictions

Fairness monitoring is deprecated in AWS ML services

They only apply to regression models

14. In SageMaker, how can you enable automatic model monitoring for a deployed endpoint?

Through CloudFormation templates only

By creating a Model Monitor job with a baseline and capture configuration

By setting environment variables on the endpoint

Through the SageMaker console without additional configuration

15. What metric combination helps identify prediction drift in classification models?

Precision and recall on current predictions

Training time and model size

Feature distributions and prediction label distributions

Endpoint memory usage and CPU utilization

16. Which of the following best describes model retraining strategy in production?

Always retrain models daily regardless of performance metrics

Retrain based on drift detection, performance degradation, or scheduled intervals

Never retrain after initial deployment

Retrain only when model accuracy drops below 50%

17. What is the primary purpose of CloudWatch alarms in ML model monitoring?

Automatically retrain models when drift is detected

Trigger notifications when monitored metrics exceed defined thresholds

Store model training artifacts

Manage model versioning

18. Which metric is most critical for detecting data drift in continuous production monitoring?

Model latency

Statistical distance between baseline and current data distributions

Endpoint invocation count

Model file size

19. In SageMaker Model Monitor, what does a baseline represent?

The minimum accuracy threshold required for deployment

A reference dataset capturing expected feature and label distributions

The initial model training dataset

The production endpoint configuration

20. What is model drift in the context of ML production systems?

Gradual degradation of model accuracy over time due to changing data patterns

Random fluctuations in prediction confidence

Unplanned model version updates

Network latency between endpoints

AWS ML Engineer Model Monitoring and Maintenance Quiz