The near 40-year-old amenity was closed in 2019 after its roof was found to be in urgent need of costly repair.
Sport England had objected to the move as there are no firm plans to replace the current swimming pool with another one.
The council is proposing redeveloping the site to include a new leisure centre and housing.