I can help in this statistical analysis of the data as well. Sax, I agree with you that with a bench dyno and same diameter wood, data could be more uniform with less error due to human factor. Enough someone in measuring cutting time with a chronograph to introduce error. But, in the lack of a bench dyno, you could still make the test by increasing the number of cuts (repetitions) to decrease the influence of error caused by the human operator. Important is to use a uniform diameter wood, and a must to use the same bar size with a new chain for each saw and randomly select the succession of cuts for each saw. Before each cut saws should be weighted and set at the same initial weight, lets say 12 kg for example.
It's not a matter of reducing the influence of variation (I'd say error, but most people think "mistake" when they see the word error), but actually getting a good measure of the variation due to operator variation and wood consistency.
If I tell you I'm getting gas for $1.25 a gallon, you know that's cheap because you have multiple measures of gas prices. The gas prices vary, but you know how much they vary.
If I say I got a Shimano Stella FA10000 in excellent condition for $450, many wouldn't have a clue as to whether that was cheap, fair, or overpriced. They have not looked at enough Stella prices to know how much variation exists.
If you have a lot of cuts with each saw (preferably alternating cuts), you can tease apart how much variation is due to the cutting conditions, and how much variation is due to differences between the saws.
The important is that all this will accomplish is a more precise estimate of the difference between 2 (or more) individual saws.
It will not tell you what the actual differences are between models. It's reasonable to assume the same model saws should perform similarly, but not reasonable to assume they will perform identically. It's even less reasonable to assume that the same person modding 2 saws of the same model will produce 2 identical saws.
So...one could certainly produce a more reliable estimate of the differences observed by using the numbers generated from such a test in an appropriate fashion. I'm of the opinion that if you are going to provide numbers, you should make the effort to make them meaningful.
BUT it still is just the difference between individual saws, in a single type and size of wood, on one particular day. Your mileage most likely will vary.