Also from an OS design standpoint: What were you expecting? Any modern day OS is designed with some form or other of swapping in mind. Meaning that the page fault interrupts the CPU _will_ throw (because let's face it, there'll be more stuff on your harddisk than you have RAM*) go unoptimized if you turn it off.
<'splain>
The logic that drives the memory system often runs along those lines:
- If it's in RAM, do nothing. (The CPU will wake me up if there's anything missing...)
- On page fault:
- check the swap page table if I have it somewhere. If yes, load that page from swap into ram.
- if no, go ask the filesystem to go find it.
Thing is, that in the page table, the OS basically knows where on the drive the page is (swapfile/partition having a defined size and location), and can just ask the drive to get it.
When swap is disabled, it _always_ has to go through through the file system to get that particular bit of data back into ram, which obviously involves more cpu cycles.
</'splain>
If you're switching between loads of programs, of course the overhead of not having a swap space is more pronounced. And loads of memory usually go along with loads of programs.
PS: Don't try to switch off swapping on a Linux OS ever (/proc/sys/vm/swappiness=0). ;-)