PM must take responsibility: Chalmers