if the ball is in your court then,
if it were me, I wouldn't allow him to visit. Let him take you to court if he chooses to do that. Then you can demand he be drug free and have him tested regularly before he sees his son. wouldn't last long.
I don't know how it works where you are where adoption is concerned but the way it works here is that if the biological parent is alive, they have to sign for it. If they refuse to sign, then there's a battle in court. The court just might find him unfit enough to strip his rights away from him and grand your b/f legal rights.
Hubby and I waited until we were married before he adopted my daughter (oldest one) because that's what we were advised to do. But, her biological father was dead so there was no fighting it there. It costs us about 750.00 and we had to do nothing.