A member of the government and a prominent MP said Sir Keir Starmer could be ousted unless Labour has a strong showing in May's elections.