Suppose you have been in a relationship with your partner for several years (no marriage, no children).
Even though you still have strong emotional feelings for your partner (to the extent that you would claim to love her/him), you are no longer sexually attracted for him/her. While your partner can do without the physical aspects of your relationship, you feel to miss out on something important in your life.
Is it selfish to end the relationship, even though the breakup would be very hard for your partner and you don't want to hurt him/her? In other words: Is it immoral to choose sexual desires over friendship and mutual love?
It sounds to me as if what you need to do is to have a frank conversation with your partner about things. Sexual attraction for a partner can ebb and flow, and one option might be that some good communication would improve things between the two of you on that front. Alternatively, you could stay together in an "open" relationship, where the value of your partnership can be preserved but not at the cost of your sexuality. The point is that between the two of you, communicating well about what you have and what you (now) lack, there might be some creative problem-solving that would give a more optimal result than the options you are currently considering.