Richmond Council says the current Elleray Hall “has reached the end of its useful life”, is beyond repair and too costly to heat.
The new development will still offer services for older residents, as well as host events, classes and activities aimed at all age groups.
The housing part of the scheme will deliver 16 residential units, all of them affordable.