The Tories lost what was an ultra-safe seat to Lib Dem Helen Morgan.
But Oliver Dowden says he still has "every confidence" in Boris Johnson's leadership.
Lib Dem leader and Kingston MP Sir Ed Davey doesn't agree, and thinks ministers should listen more closely to what people have been saying.