A VNC Server can pretty easily be setup to accept connections only 
from specific IP addresses. If you're trying to connect from a 
different IP adddress you'll have trouble -- you're supposed to.

On a Windows VNC Server look for VNC registry keys such as:

HKEY_LOCAL_MACHINE\Software\ORL\WinVNC3\AuthHosts
or
HKEY_LOCAL_MACHINE\Software\ORL\WinVNC3\LoopbackOnly

See http://www.realvnc.com/winvnc.html#4 for details.

On a Linux VNC Server, check for entries in the "hosts.deny" file
or in the packet-filter scripts.