Even though the average temperature might be comfortable, if your left foot is in ice water and your right foot is in scalding water, you won't be.
Average time, or average memory use may not be the whole story.
Sometimes you need to be sure it's done within a set time, or within a set amount of memory.
A car that is faster most of the time, but occasionally crashes is unacceptable.
Time is highly dependant on implementation.
A 10 millisecond delay before sorting begins is probably acceptable if the sort runs 20 seconds faster.
10 milliseconds every time you sort a three element list is probably not acceptable.
There's not much that can be done about this problem other than being aware of it.
It will affect things, but how much and how important that is, is variable.
Certain algorithms might be unacceptable for your implementation - Your Mileage May Vary.