Same here. I finally bought a set of brass punches off amazon and used one of those and a hammer to install the front sights in the 4" slides I bought. One was issue free. The other one had either a bad front sight cut or a bad front sight in the set I bought. The dovetail was so loose on the right side the sight would fall out when it got near the center of the dovetail.
Lock the slide down tight to control movement, use a punch (brass is softer than steel and shouldn't damage/ding the side of the sight - I've used steel punches, dinged the sides of the slight and the pistol still shoots great, but brass won't ding the steel sight). And tap as hard as required to move the sight (start it in the left side of the dovetail if looking from the rear of the slide and drive it to the right side. You can eyeball it for center or work at measuring it.