I did not glue mine on until one day leaving Ely (NV) after the plane sat in the sun for most of the day, I took off and heard this horrible banging/drumming sound. Landed and it was the fairing seal on the side that had been facing the sun. Flew home without it in place (no big deal) and then used some clear silicone to glue it onto the fairing. No issues since then. Should be removable if I need to do so. Note that I did this after paint, however, since paint and silicone are pretty much incompatible.
There are some archival discussions on this.
And as Noah said, make sure it's not on upside down (or is it inside out?).
cheers,
greg