It only seems counterintuitive to a non-pilot.
The practice comes from years of "rolling" the trim wheel forward to produce nose down trim. As you roll the wheel forward, your thumb goes forward toward the panel. It's as if you are "rolling" the trim switch forward.
Most planes have the top of the switch labeled "DOWN", like you have it. I know it seems counter intuitive, but the way that I remember it is that pushing the top of the control stick causes the nose to go down.
The practice comes from years of "rolling" the trim wheel forward to produce nose down trim. As you roll the wheel forward, your thumb goes forward toward the panel. It's as if you are "rolling" the trim switch forward.