On my old airplane, the pants were trimmed very tight and sat extra low around the wheels. That experience taught me that large gaps are preferable for normal flying. My wheels pants always took a beating in the grass and trapped snow in the winter. The wheels and brakes built up a little more heat, etc. In the end, you probably won't notice a difference in speed... but I promise you will notice if your brakes are clogged with snow or the tire blows.
For racing, I like the idea of removable lower fairings. You can get a little edge during a race... with the option to remove them for normal flying. If you are determined to seal around the tire, I recommend trying some heavy-duty felt-like materials. I've had good luck sealing other aerodynamic problems with felt. I think this will stop more airflow than brissles.
Of course... there is always the Bede-style wheel pants that Lopresti is now making.