Yeah I agree with that. I think maybe what works against Allen is a little like Whitaker in that he has some seasons where he missed 15-40 games so there are not as many seasons where he racks up the big numbers. Also, overall his career wasn't as long as some (only 7315 plate appearances). But I still think he should be in. His career took place when it was a lot harder to put up power numbers than it is today. He was leading the league in OPS before OPS existed.
He'd probably fare much better in MVP voting too if he had seasons like 1965, 1971, or 1967 in baseball's current era.