The England defender missed the 2-nil win at Aston Villa in the Premier League after suffering the injury in Tuesday's victory at AC Milan.
James is expected to be out for 8 weeks, meaning he won't be fit until after the group stage of the tournament in Qatar, which starts next month.
Potter expects the right back to recover well.