Fabio Wardley has ruled out one potential challenger as he prepares for the first defence of his WBO heavyweight title.