Naturally, Simon wasn't a fan