The states recognized long ago that a parent might take the kids and move across the country in order to make things more difficult for the other parent. The laws are generally written to avoid that possibility. It's best to consult with a local attorney before moving.
This means, of course, that if custody is important to you, you should not leave your children with the other parent for any length of time (more than a night or two). And if you're afraid the children aren't safe alone with the other parent, you should consider whether it's safe to leave them with the other parent for any length of time.
A man who thinks he is the father of a child born out of wedlock normally shouldn't attempt to take custody of the child. If things go very badly, there is the potential for criminal charges.
This, alone, may not be enough to require a change in custody, but it doesn't help. If your ex has shown an intention to use your smoking against you, stop smoking around the child. Better yet, stop smoking.
Most states use some form of a "best interests of the child" standard to determine custody issues. Adultery may not have a direct effect on the custody decision, but may play an indirect role. Under the "best interests" standard, the court will consider your daughter's surroundings, which certainly includes her home life. Your child's relationship with the man with whom you're living, and how your relationship with him affects your daughter, will be relevant in the court's decision.
Only one thing is just about certain with regard to adultery and divorce - it is almost never a good thing. It can harm you, and will likely never help you.