Take Baby Steps Forward Even When You Fall Back