
Generally, since you were not married, Dad would probably need to file a paternity action to set a hearing for custody/visitation. Without a court order for custody/visitation (since you were not married), you generally would not have to allow him to see the child.
This response is general in nature and should not be considered legal advice.