For the sake of this discussion, we’re going to talk about two kinds or types of gay men*. There are many other kinds and there gay men who don’t fall into any types that we could talk about, but just for this topic, let’s focus on two types.
There are the gay men who are attracted to men. That is to say, no matter what gender or sex the person was assigned at birth, no matter what kind of genitals the person used to have or currently has, as long as the person identifies (and probably presents) as a man, these gay men may be attracted to him and enjoy sex with him. These gay men identify as gay because they are attracted to a similar (same, homo) gender.
Additionally, there are gay men (perhaps “gay males” would be a more appropriate term) who are attracted to people with penises. They can be attracted to men, women, genderqueers, and people of other genders, as long as the people have penises (and perhaps lack breasts). These gay men identify as gay because they are attracted to a similar (same, homo) set of genitals.
Although different, both of these types are referred to as gay. There isn’t a distinction in the terms, unless you start getting into the more specific, complicated labels, and frankly, most don’t bother with those.
However, this is the difference between bisexuals and pansexuals. Pansexuals are like the first type discussed, the ones who are attracted to people of all (pan) genders, regardless of genital type or sex assigned at birth. Bisexuals are like the second type discussed, the ones who are attracted to all (typically there are only two, bi) sexes assigned at birth, regardless of gender identification. Both bisexuals and pansexuals have the potential to be attracted to people of all genders and sexes—it’s how we see that attraction that tends to determine which label we use. It’s a subtle difference, yes, but still deserves respect.
It’s a question of whether sexuality deals with gender identification or types of genitals—other sexualities deal with the same question, but the non-monosexuals are the first to use different labels depending on the answer.
* “Gay men” refers to people who self-identify as gay men.