The reason is statistical, not operational. Your visibility for a question is a rate; each run is one Bernoulli draw. With a handful of draws the interval around that rate is very wide — wide enough that a jump from 30% to 45% may be indistinguishable from nothing happening. Adding calendar frequency without adding repeats just gives you more wide intervals.
So the first question is not "how often" but "how wide is my interval". If you want to detect a 10-point change, you need enough samples that your interval is narrower than 10 points. That is a sample-size decision you can compute up front, and it is the difference between measurement and vibes.
The second consideration is engine drift. Providers update models and retrievers without notice, and those updates move everyone's numbers at once. Comparing this week to last week without a control means you may credit your own work for a provider's change. The defense is to track a stable control set alongside your own questions: if everything moved together, it was the engine.
Around an intervention, cadence matters differently: measure a real baseline before shipping, then re-measure after, ideally with some prompts deliberately held out from the change so they act as a comparison group. Without that structure, "we fixed it and the number went up" is a correlation, not a result.
