The problem could be that fillers may not stick as well to your primer as they would to fiberglass or epoxy. That said, if you use an epoxy primer and sand it before applying your next layer of filler or epoxy, you'd *probably* be OK.
Common practice is to sand away all of the rough areas, fill (with an epoxy/micro mix) any low areas, sand, and then apply 3 separate coats of epoxy squeegeed onto the surface to fill the pinholes.