Most people with penises have a refractory period. It's normal, and healthy, so don't stress too much about it.
I'd be curious for you to ask yourself, why does it bother you so much? Can you potentially put more focus into foreplay or aftercare with your lover when this happens?
If you can take the pressure and focus off of your refractory period and your penis, you may enjoy a wider range of pleasurable activities without an erection.
To learn more about sex without an erection, and how to re-train your body to have more erections, download the Lover app for personalized courses.