Installing Apache Spark via Puppet - Revision history

Bpopp: /* Start the Services */

2020-10-15T18:22:29Z

Start the Services

← Older revision		Revision as of 18:22, 15 October 2020
Line 298:		Line 298:
	<pre>		<pre>
	sudo systemctl enable spark		sudo systemctl enable spark
	sudo systemctl hadoop ~~spark~~		sudo systemctl enable hadoop
	sudo systemctl jupyter ~~spark~~		sudo systemctl enable jupyter

	</pre>		</pre>

2020-10-15T06:58:05Z

Create Jupyter Service

← Older revision		Revision as of 06:58, 15 October 2020
Line 246:		Line 246:
	JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-amd64/		JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-amd64/
	PYSPARK_SUBMIT_ARGS=--master spark://spark1.lab.bpopp.net:7077 pyspark-shell		PYSPARK_SUBMIT_ARGS=--master spark://spark1.lab.bpopp.net:7077 pyspark-shell
			PYTHONPATH=/usr/local/spark/python:/usr/local/spark/python/lib/py4j-0.10.7-src.zip
	</pre>		</pre>

Line 277:		Line 278:

	</pre>		</pre>


	== Start the Services ==		== Start the Services ==

2020-05-01T22:00:08Z

Removing DNS entries from pfsense

← Older revision		Revision as of 22:00, 1 May 2020
Line 65:		Line 65:
	== Removing DNS entries from pfsense ==		== Removing DNS entries from pfsense ==

	Also ran into issues w/ pfsense caching DNS entries and foreman not being able to overwrite them. To remove them from the pfsense command line:		Also ran into issues w/ pfsense caching DNS entries and foreman not being able to overwrite them.

			1) Use the Status > DHCP Leases to remove the existing entry

			2) Remove them from the pfsense command line:

	<pre>unbound-control -c /var/unbound/unbound.conf flush spark4.lab.bpopp.net</pre>		<pre>unbound-control -c /var/unbound/unbound.conf flush spark4.lab.bpopp.net</pre>

	And then restart the unbound service from Status > Services > unbound		And then restart the unbound service from Status > Services > unbound.

	= Manual Configuration =		= Manual Configuration =

2020-02-15T21:22:14Z

Removing DNS entries from pfsense

← Older revision		Revision as of 21:22, 15 February 2020
Line 68:		Line 68:

	<pre>unbound-control -c /var/unbound/unbound.conf flush spark4.lab.bpopp.net</pre>		<pre>unbound-control -c /var/unbound/unbound.conf flush spark4.lab.bpopp.net</pre>

			And then restart the unbound service from Status > Services > unbound

	= Manual Configuration =		= Manual Configuration =

2020-02-15T21:21:00Z

Removing Existing DHCP connection

← Older revision		Revision as of 21:21, 15 February 2020
Line 62:		Line 62:
	remove		remove
	</pre>		</pre>

			== Removing DNS entries from pfsense ==

			Also ran into issues w/ pfsense caching DNS entries and foreman not being able to overwrite them. To remove them from the pfsense command line:

			<pre>unbound-control -c /var/unbound/unbound.conf flush spark4.lab.bpopp.net</pre>

	= Manual Configuration =		= Manual Configuration =

2020-01-23T02:53:41Z

Update Hosts

← Older revision		Revision as of 02:53, 23 January 2020
Line 44:		Line 44:

	Debian adds a host line that needs to be removed. Remove spark1.lab.bpopp.net from /etc/hosts		Debian adds a host line that needs to be removed. Remove spark1.lab.bpopp.net from /etc/hosts


			== Removing Existing DHCP connection ==

	Ran into a situation where existing connections were not correctly removed from the DHCP server and was getting :		Ran into a situation where existing connections were not correctly removed from the DHCP server and was getting :

2020-01-23T02:53:07Z

Removing An Existing Connection from DHCP

← Older revision		Revision as of 02:53, 23 January 2020
Line 40:		Line 40:

	= Removing An Existing Connection from DHCP =		= Removing An Existing Connection from DHCP =

			== Update Hosts ==

			Debian adds a host line that needs to be removed. Remove spark1.lab.bpopp.net from /etc/hosts

	Ran into a situation where existing connections were not correctly removed from the DHCP server and was getting :		Ran into a situation where existing connections were not correctly removed from the DHCP server and was getting :
Line 55:		Line 59:
	remove		remove
	</pre>		</pre>


	= Manual Configuration =		= Manual Configuration =

2020-01-23T01:18:16Z

SSH Setup

← Older revision		Revision as of 01:18, 23 January 2020
Line 64:		Line 64:

	<pre>		<pre>
			ssh spark@localhost
	ssh-keygen -t rsa		ssh-keygen -t rsa
	ssh-copy-id spark@spark1.lab.bpopp.net		ssh-copy-id spark@spark1.lab.bpopp.net

2020-01-23T00:56:38Z

@@ Line 38: / Line 38: @@
    include spark::frontend
 }</pre>
 = Manual Configuration =

2020-01-22T16:32:32Z

← Older revision		Revision as of 16:32, 22 January 2020
Line 1:		Line 1:
			= Puppet Setup =

	Install Java Module into Puppet		Install Java Module into Puppet