Spark/Hadoop Cluster

Revision as of 06:28, 29 January 2024 by Bpopp (talk | contribs) (Created page with "To allow the spark user to ssh to itself and also the workers, you need ssh passwordless to be enabled. This can be done by logging into the spark user and doing: <pre>ssh-keygen -t rsa -P ""</pre> Once the key has been generated, it will be in /home/spark/.ssh/id_rsa (by default). Copy it to the authorized hosts file (to allow spark to ssh to itself): <pre>cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys</pre> Or, for each worker, do something like: <pre>ssh-copy-i...")
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)

To allow the spark user to ssh to itself and also the workers, you need ssh passwordless to be enabled. This can be done by logging into the spark user and doing:

ssh-keygen -t rsa -P ""

Once the key has been generated, it will be in /home/spark/.ssh/id_rsa (by default). Copy it to the authorized hosts file (to allow spark to ssh to itself):


cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

Or, for each worker, do something like:

ssh-copy-id -i ~/.ssh/mykey spark@localhost