Another posibility: (with lenghty description)
Internet in Belgium sucks, or used to suck for a while. While pretty much the whole of (Western, and most of Eastern) Europe had 20, 40, 80 Mbps connections allready, we were stuck with a shitty 3Mbps line.
Anyway, when they FINALLY upgraded the speeds/bandwiths, apparently they changed the DSL mode from normal (I forgot the real name) to interleaving.
You never notice this, as your router is supposed to take care of this automatically. Turns out my router was too old to handle this new type of mode, and interleaving increases latency slightly.
Result: I had trouble maintaining connections at peak hours somehow, when I got through to a technical operator, he told me my DSL-router/modem was working at 90% of it's capacity. So when I started TF2-ing, the poor thing came to a near meltdown.
Bought other router, problem solved.
[edit] Bottom line Junta, could you get another router (perhaps a newer one) to test this ? I realize that not everyone just has a spare router liying around.