If I have sole custody and court gives him visitation, do I have to comply?

Child's father has not seen her in four years. Suddenly he wants her. He refuses to sign divorce papers after 5 years of seperation because he claims he wants to see her. He is not in the US. If he contests and the court grants him custody, what's the worse that can happen if something comes up during the time he is supposed to have her. Also note that my trust is wary because I filed a Hague Convention before to get her back after he kept her for almost six months in his country. Do I just get a default divorce?