I assume you mean the inside edges. It helps to have a longer arm for the deburring tool for some tighter spots (somewhat counterintuitive). I have also put a deburring bit in a right angle drill attachment to reach some difficult areas. It would help to know which deburring tool you are trying to use.
You don't need to use a countersink-style tool to deburr holes. Most of us are too agressive with those anyway, beveling the edge of the hole. Use a scotchbrite pad or a small wheel in a dremel tool. Don't try to work just over the hole or you will make a shallow divot. Instead, polish the entire flange surface.
If you can reach it with a piece of scotchbrite, generally a couple rubs will do the job. I also find that for tighter spots, wearing a leather work glove to rub the holes works well too.