Open configuration file /etc/ssh/sshd_config of SSH server by using nano text editor and find next line: Port 22 or your current port.
Save file and reboot SSH:
Debian / Ubuntu
/etc/init.d/ssh restart
CentOS
/etc/init.d/sshd restart
FreeBSD
/etc/rc.d/sshd restart