UK ministers isolate after Joyce contact