The fact that usually it works because most of the outputs were recently created is incredibly dangerous. If the ratio of best case to worst case performance gets bad enough the attacker just has to come along with a block spending outputs that weren't recently created, or otherwise picked in a way where retrieval happens to be slow, to knock slower miners offline.
Who gets to decide how slow is too slow?
Mining these days requires investing in ASIC hardware. Solo mining or running a pool will very soon require investing in a reasonably fast network connection and a machine with at least a few gigabytes of memory.
Knocking the slowest N% of solo miners/pools off the network every year (where N is less than 20 or so) is not a crisis. That is the way free-market competition works.