Your choices for installation are to leave the aluminum frame in the wall and downsize the new windows for fit within that existing frame. Then brake metal (pre-finished aluminum) cover the perimeter to finish. The second choice involves either cutting the flange of the frame out as you describe (damage included) or remove the frames by cutting the nails that secure the frame to the wall framing under both the brick and the siding. Both ways of removal will likely damage the siding. The siding then gets cut back to expose the nailing fin of the existing prime windows, new windows get installed, and a wider trim board is installed and then covered with brake metal to finish. I suggest you make your decision based on how particular you are for the finish appearance. The wider boards sometime improve the exterior appearance and sometimes really detract from the appearance.