
Generally, when the parties are not married and have a child, the non-custodial parent (father in your case) really has no rights per se. He (or you) would be required to file a Paternity action (Complaint to Establish Parental Relationship) with the courts for the court to establish that legally he is the father (even though there may be no dispute about it). He or you would also set a hearing with the court (Order to Show Cause) to set up some temporary custody and visitation rights. Absent a court order you have no legal obligation to allow him visitation.
This response is general in nature and should not be considered legal advice.