Hi, I'm a developer at VMware. It's not surprising that you're finding that VMware Player is faster than VMware Server. When you use VMware Server on Windows, interactive traffic (graphics, keyboard/mouse) uses our remote display path, even when you run the server console on the same machine as the client. This is part of what allows you to "detach" the console and run the VM in what we call "headless" mode. You can also run multiple consoles on separate machines and point them at the same VM at the same time.
The primary and intended use case for VMware Server is for running light production/test server environments and hence interactive graphics performance was not a priority; it's more important to us to make sure that it's possible to interact with the server remotely and to expose e.g. the scripting APIs and so on. That having been said, the next version of VMware Server should bring with it some improvements in graphics/interactive performance.
However, if you're primarily trying to run a Linux VM interactively e.g. in order to do some Linux development, or play around with a new Linux desktop release or whatever, then you should stick with VMware Player.
I'd also recommend you try out the latest version of VMware Workstation (6.0), although it is a for-pay product so that might deter you. Workstation brings with it some of the Server features like headless and remote display support, but also includes improved graphics and interactivity, for example the new ability to use multiple host monitors within your guest.