This page briefly outlines some typical Connection Refused errors as well as possible solutions.
Try correctly setting STUN on the device that's having problems (i.e. your IP phone)
For first-time installation, ensure your freeswitch server hostname is resolved to IP on the server. You need a DNS entry or /etc/hosts entry matching the HOSTNAME value in /etc/sysconfig/network file