Your partner sounds like he's really looking out for you. Is it possible to have a relationship with your first born without bringing them into your home or the other childrens lives?

If my man said that I'd question the kind of person he is

I am in the same situation except I'm the partner who doesn't want a step teen near my children. I have told my husband he should have a relationship with his son, but we can't be part of it. My husband argues that the kids are half siblings and have a right to know each other, but it's not fair on the younger ones to have to deal with him. The younger kids have said they don't want to see him and my husband has to accept that. My husband sees his son outside the house, it's not regular but they have their own time together. There is nothing wrong with having your own separate relationship that doesn't involve the other members of your family. You need to consider what is right for everyone, there is no reason you can't still see them. I hope it works out x

Kids before men, every time.

Tricky, depending on the circumstances of the fallout between the first born and yourself.
If you are looking to rebuild the relationship with the first born can you do so without involving the other children initially?
Or does the first born want contact with the other siblings/not you?
In which case how would this affect your children? What damage has been done in the past? Are the children likely to exposed to danger etc seeing the older child? Perhaps write a pro and cons list. Can there perhaps be supervised contact in a safe setting at first, supervised by a third party? Mediation? Although if you had problems before it might not help.
Good luck.
Family dynamics are tricky

Does your eldest want to move back in with you, or do they just want to visit and your partner doesn't want that?

If they don't want to move back in, but you do want to see them, how about planning that you two (and only you two) will catch up somewhere, every X many weeks. For example the first Saturday of every month, your partner takes care of the other kids, and you and your eldest go out for morning tea, or see a movie and get an ice-cream after, or take a walk along the beach, or go to an arcade and play games (make sure you do stuff your teen is interested in so that they can share themselves with you, and you can get an insight into who they are these days).

Keep the agreed time and date as often as you can, unless it's an absolute emergency. So you, your partner and your teen will all need to work out a time that suits so you know your other kids are taken care of, and so your eldest knows you have specifically made time for them. If their resentment stems from feeling abandoned then hopefully when you consistently make them for them, always show up when you said you would, the activity is about just the two of you and so on, hopefully they'll come to see you do love them and want them to be part of your life (and you want to be part of theirs). It won't happen overnight, but little steps add up.

Your child trumps your partner always!