Traditionally we seldomly connect our KVM from the port directly to the server that consists of usb for keyboard and mouse, vga cable for display purposes. More advanced you may have an experience using the iKVM which connect all you kvm port from the internet where you can have a remote location as long as you have the internet connection. When technology evolve, physical server are becoming obsolete since we have the virtualization technology. Since that to access your virtual machine (vm) are more easier. Instead of using SSH, like a Xen hypervisor, the Xen itself can host as a KVM server and map to the vm on different port as configured from the config file.
Don't look your past, but look your future