A/B Testing for Business Decisions Quiz

1. How should you handle external factors that might confound your A/B test results?

Ignore them and proceed with analysis

Control for them through randomization and test design

Include them as additional variables

Run the test only during holidays

2. What is the 'confidence interval' in A/B test results and why is it important?

A range of values likely to contain the true effect; helps assess result reliability

The percentage of users who completed the test

The margin of error in your marketing budget

A measure of how long the test should run

3. Which best practice should you follow when designing an A/B test for a website checkout page?

Test multiple elements simultaneously to save time

Isolate one element per test to identify its specific impact

Run tests only during business hours

Use different traffic sources for control and variation

4. In A/B testing, what does 'statistical significance' guarantee about business impact?

The result will always be profitable

The difference is real, not due to chance; business value still requires evaluation

No further testing is needed

The winner will perform well in all markets

5. What is a 'multivariate test' and how does it differ from an A/B test?

It tests one variable while A/B tests multiple variables

It tests multiple variables simultaneously while A/B tests one variable

It is identical to an A/B test

It uses qualitative data instead of quantitative

6. How should you decide the duration of an A/B test?

Run it for exactly 7 days regardless of traffic

Base duration on sample size requirements and daily traffic volume

Continue until you see statistical significance

Stop after the first week of data collection

7. What is 'regression to the mean' and how does it affect A/B testing?

A statistical phenomenon where extreme values tend to move toward the average on retest

A requirement to use regression models in all tests

The decline in performance over time

An increase in sample variance during testing

8. Which of the following is a valid reason to stop an A/B test early?

You observe a positive trend in preliminary data

You reach your predetermined sample size and statistical significance

A competitor launches a similar feature

The test has been running for one week

9. When interpreting A/B test results, what does a p-value of 0.03 mean?

3% probability the result is due to chance

97% confidence in the new variation

The test should continue for 3 more days

The effect size is 3% of the baseline

10. What is the relationship between sample size and statistical power in A/B testing?

Larger sample size decreases power

Larger sample size increases power

Sample size has no effect on power

The relationship is nonlinear and unpredictable

11. What is the primary purpose of an A/B test in a business context?

To compare two versions and determine which performs better

To increase marketing budget without analysis

To replace all qualitative research methods

To guarantee profit margins will increase

12. In A/B testing, what does 'effect size' refer to?

The duration of the experiment

The magnitude of the difference between variations

The budget allocated to the test

The number of users in each group

13. What is 'peeking' in A/B testing and why is it problematic?

Checking results early; it increases Type I error risk

Reviewing competitor data; it's unethical

Examining individual user behavior; it's time-consuming

Previewing the test before launch; it's standard practice

14. Which metric is most appropriate for measuring conversion rate changes in an A/B test?

Chi-square test

T-test

ANOVA

Pearson correlation

15. What is the minimum recommended sample size for a valid A/B test?

10 users per group

50 users per group

Depends on effect size, variance, and power desired

1,000 users per group always

16. How should users be assigned to control and test groups in an A/B test?

Based on their geographic location

Randomly to minimize bias

Based on their past purchase history

Based on their device type

17. What does statistical power measure in an A/B test?

The probability of detecting a true effect when it exists

The cost of running the experiment

The number of variations being tested

The duration of the test

18. Which significance level is most commonly used in A/B testing?

α = 0.10

α = 0.05

α = 0.01

α = 0.20

19. What is a Type I error in A/B testing?

Failing to detect a true effect when it exists

Incorrectly concluding a difference exists when it doesn't

Using an insufficient sample size

Stopping the test too early

20. In A/B testing, what does the 'control group' represent?

The group that receives the new variation

The group using the existing version (baseline)

The group that is not measured

The group with the smallest sample size

DataX A/B Testing for Business Decisions Quiz