I have used RSYNC to copy files between Linux servers a lot, but never knew how to use different port other than default port 22. So I looked it up and found the solution.
Here it is
rsync -avz -e "ssh -p $portNumber" user@remoteip:/remotepath/files/ /localpath/
Hope this help someone :)