From FreeSWITCH Wiki
This page briefly outlines some typical Connection Refused errors as well as possible solutions.
Try correctly setting STUN on the device that's having problems (i.e. your IP phone)
For first-time installation, ensure your freeswitch server hostname is resolved to IP on the server. You need a DNS entry or /etc/hosts entry matching the HOSTNAME value in /etc/sysconfig/network file