A sexual fantasy is a mental representation of any kind of sexual activity. It would be naive to believe that none of us fantasize during sex. At some time or the other we all fantasize either verbally or secretly. Otherwise sex would be mechanical and boring.
Fantasies can be positive or negative. While some fantasies can be fleeting, some can be elaborate. Some might find it pleasurable to fantasize about sex with their partners, other people's partners, animals, all sorts of combinations. Some people may think it's wrong to fantasize about sex at all, about certain kinds of sex, or about sex with certain people.
Men are much more likely to fantasize, or to admit to fantasizing, than women, since you have also noticed and admitted. Some men have sexual fantasies many times each day, while others can go for weeks without one. Fantasies, does not mean that one wants to carry out those fantasies in real life. It could be just to pep up your sex life and add a little spice to it. It on no way means that ones loves his/her partner any less when one fantasizes.
Your fear that fantasies can also lead to the next level are also true. Sex rarely stagnates and we want more. This is rightly your area of concern. You seem to be more worried about your wife asking for the real thing sometime later on in the relationship. This only you can judge and provide answers.
Rather than talk about lovers, what you can do is role play with masks and wigs and blind folds. Give rein to your imagination. Talk dirty during sex. Role playing is a great way to add variety, spice, and excitement to your sex life. Communication and talking about your fantasies or role playing ideas is a good starting point for you both.
One TIP. Do not talk about what you fantasize at other times. Leave it only for the bedroom and sex time.