Looking from the rear of the slide to the front of the slide, they come off Left to Right. They get installed Right to Left. Apparently there is a slight taper in the dovetail and if you try to take off/put on the opposite way you are fighting the taper.
And don't forget to loosen the allen screw on the rear sight before uninstalling! Otherwise, it didn't take as much force as I thought it would. It did help using the sight tool, probably a little tougher with a hammer and punch.
Oh, and there's a spring and metal piece under the rear sight, it'll fly off when you remove the sight. Remove the rear sight in a plastic bag if you can to catch the spring. It's pretty easy getting the spring back in. I started the new rear sight and installed it until about 30% of the hole is covered. Then I inserted the spring, used the smallest allen wrench I had to compress the spring, put the metal piece on and pushed the sight over them. If you look at a parts diagram you'll see what I'm talking about.
Adam