You need to fill the holes because they are rough and because they can out-gas air if the bait is heated, bubbling the paint. Since PVC is a closed cell foam, you don't actually need a durable waterproof sealer to protect it as you would with wood. But the sealer needs to be compatible with the topcoat you are planning to use. If your topcoat is solvent-based, I'd probably choose epoxy as the sealer because it will not react with just about any topcoat. If your topcoat is epoxy, you have a wider range of sealers that will work, including regular sanding sealers, epoxy, etc.