Login Node ssh-keygen Failure

[bluecrayon52]

From the login node:

[+ echo Generate a new keypair...
Generate a new keypair...
+ ssh-keygen -t rsa -b 4096 -q -f id_rsa -N ''
id_rsa already exists.
Overwrite (y/n)? Traceback (most recent call last):
  File "/tmp/sagemaker-lifecycle-07f7a580/src/lifecycle_script.py", line 255, in <module>
    main(args)
  File "/tmp/sagemaker-lifecycle-07f7a580/src/lifecycle_script.py", line 208, in main
    ExecuteBashScript("./utils/gen-keypair-ubuntu.sh").run()
  File "/tmp/sagemaker-lifecycle-07f7a580/src/lifecycle_script.py", line 31, in run
    result.check_returncode()
  File "/usr/lib/python3.10/subprocess.py", line 457, in check_returncode
    raise CalledProcessError(self.returncode, self.args, self.stdout,
subprocess.CalledProcessError: Command '['sudo', 'bash', './utils/gen-keypair-ubuntu.sh']' returned non-zero exit status 1.]

This appears to be coming from line 21 in gen-keypair-ubuntu.sh

ssh-keygen -t rsa  -b 4096 -q -f id_rsa -N ""

This might be due to a race condition where after checking for an existing key on line 9, another node creates a key before line 21 is executed.

This may be fixed by adding a -y flag

[amanshanbhag]

-y can fix this

[mhuguesaws]

Is this fixed?

[amanshanbhag]

Bundling as part of #703

[amanshanbhag]

Changing from -y to gracious error handling instead 2>/dev/null || true -- in the future, a majority of this script should probably run only from one node (egs. keygen can be done from a single node, since we're writing to a shared fs).

Tracking in #703

[bluecrayon52]

Resolved by @amanshanbhag in #703