Use RPV for a/b testing? Watch out.
Big fluctuations vs CVR. You need MUCH more data - impractical for many sites.
Results skewed by outliers eg big spenders, large order.
Advanced calcs required for statsig. Best done manually; may not be as rosy as testing tool claims.
Big fluctuations vs CVR. You need MUCH more data - impractical for many sites.
Results skewed by outliers eg big spenders, large order.
Advanced calcs required for statsig. Best done manually; may not be as rosy as testing tool claims.
Comments
Can we still use it in combo with CVR and AOV?
Whats the alt? What if you trimmed the top and bottom 3% of spenders or used median instead of mean, climbing to 1% until you had enough data?
Yes, in combo with CVR & AOV.
If CVR is flat / negative vs positive RPV, I’d use CVR all things equal.
CVR may be better primary metric with RPV guardrail. Not ideal, but practical and valid.
No threshold for RPV reliance, depends on many variables eg fluctuation in AOV.