Bro..listen. I too was in the same position some years ago. Due to this imprinted on my mom's sis, I faced some losses. Believe me it's not that hard for me to leave her, so it's easy for you too.
Is it fair to have affair with other's wife? NO. Think if it's your wife or your girl child, will you do it? Could you face the world ? (I am sorry if I am harsh). But remember good words are always harsh.
Take more think about this and make a perfect decision.
My advice: go to the gym, make a passion about something like music, arts, higher studies, business etc.
Have a good life..