Heat on PM as Speaker stands aside