The coach defended his visit to the White House, which took place earlier this month and included his family, after a reporter brought up President Donald Trump’s “denigrating”