Biden made the remarks during his toast at the Summit of the Americas leaders dinner on Thursday, after shrugging off controversy