This was my same exact experience.
I initially tried with a punch and hammer but just marked the original steel sights. Then I used a Wheeler sight pusher. The rear sight took a lot of force with a Wheeler sight pusher, two loud pops and off it came. Front was simple. Noticed the rear plate that covers the striker is now slightly loose but it won't be coming off by itself as it's locked in by the rear sight.
There was no set screw on the original BG2.0 sights, installed the XS R3D 2.0 Night Sights which does have a set screw and used red thread locker on them per their instructions.
Will test fire it this Friday but both front/back appear to be on rock solid. I like the three tritium dots vs. fiber optics.
Attaching a photo of the originals (left) vs. the XS Sights R3D sights (right).