The Government has already announced that it is to provide housing UK-wide for 20,000 people fleeing the Taliban.
Richmond says private housing is better suited because of the need to balance an established housing demand in the borough.
Council leader, Gareth Roberts, says the priority is to get refugees out of temporary accommodation like hotels.