The MP for Esher and Walton, Dominic Raab's said to have delegated the conversation - to start the evacuation of translators - to a junior minister.
But the Home Office says developments were too fast to set up a call.
Labour says Mr Raab should resign - the shadow security minister Conor McGinn says his position's "untenable".