Getting Started with PySpark - Revision history

Bpopp: /* Date Functions */

2020-04-21T21:35:05Z

Date Functions

← Older revision		Revision as of 21:35, 21 April 2020
Line 180:		Line 180:

	display ( myDf.limit(5).toPandas() )</pre>		display ( myDf.limit(5).toPandas() )</pre>

			== Aggregating ==

			=== Rank Items Within a Group ===

			<pre>
			from pyspark.sql.window import Window
			from pyspark.sql.functions import percent_rank, rank, ceil

			stationRankDf = ( recentHistoryDf
			.groupBy ( "CountryCode", "STATION" )
			.agg(
			round(avg("PK GROSS INB"),1).alias("PK GROSS INB")
			)
			)
			w = Window.partitionBy(stationRankDf['CountryCode']).orderBy(stationRankDf['PK GROSS INB'].desc())
			stationRankDf = ( stationRankDf
			.select('*', rank().over(w).alias('rank'))
			.filter(col('rank') <= 10)
			.filter ( "CountryCode == 'US'")

			)

			stationRankDf.toPandas()</pre>

2020-01-06T18:21:16Z

Creating Columns from Rows using Multiple Aggregates

← Older revision		Revision as of 18:21, 6 January 2020
Line 157:		Line 157:
	.agg(count('excp_type_cd').alias('cnt'),F.first('evnt_crt_tmstp').alias('first'),F.last('evnt_crt_tmstp').alias('last'))		.agg(count('excp_type_cd').alias('cnt'),F.first('evnt_crt_tmstp').alias('first'),F.last('evnt_crt_tmstp').alias('last'))
	)</pre>		)</pre>

			== Date Functions ==

			=== Convert Time w/ Time Offset to Timestamp ===

			<pre>
			from dateutil import parser, tz
			from pyspark.sql.types import StringType
			from pyspark.sql.functions import col, udf

			# Create UTC timezone
			utc_zone = tz.gettz('UTC')

			# Create UDF function that apply on the column
			# It takes the String, parse it to a timestamp, convert to UTC, then convert to String again
			func = udf(lambda x: parser.parse(x).astimezone(utc_zone).isoformat(), StringType())

			myDf = ( scanDf
			.withColumn('event_timestamp', func(col("evnt_crt_tmstp")) )
			.select("evnt_crt_tmstp", "event_timestamp")
			)

			display ( myDf.limit(5).toPandas() )</pre>

2020-01-04T18:19:37Z

Plotting Bar Graph

← Older revision		Revision as of 18:19, 4 January 2020
Line 149:		Line 149:
	ggplot( country_totals.limit(10).toPandas() , aes(x='COUNTRY_CD',y='count' ) ) + geom_bar(stat='identity')		ggplot( country_totals.limit(10).toPandas() , aes(x='COUNTRY_CD',y='count' ) ) + geom_bar(stat='identity')
	</pre>		</pre>

			== Creating Columns from Rows using Multiple Aggregates ==

			<pre>scanLookupDf = ( scansDf
			.groupby('shp_trk_nbr')
			.pivot('excp_type_cd')
			.agg(count('excp_type_cd').alias('cnt'),F.first('evnt_crt_tmstp').alias('first'),F.last('evnt_crt_tmstp').alias('last'))
			)</pre>

2019-11-01T17:53:14Z

@@ Line 74: / Line 74: @@
+)
 country_totals.show()</pre>
 == Cool trick to display panda data frame ==

2019-10-31T15:51:20Z

← Older revision		Revision as of 15:51, 31 October 2019
Line 74:		Line 74:
	)		)
	country_totals.show()</pre>		country_totals.show()</pre>

			== Cool trick to display panda data frame ==
			<pre>from IPython.display import display, HTML

			display(HTML(country_totals.toPandas().to_html()))</pre>

	==Plotting Bar Graph==		==Plotting Bar Graph==

2019-10-31T15:27:00Z

Plotting Bar Graph

← Older revision		Revision as of 15:27, 31 October 2019
Line 81:		Line 81:
	<pre>		<pre>
	from plotnine import ggplot, geom_point, aes, stat_smooth, facet_wrap		from plotnine import ggplot, geom_point, aes, stat_smooth, facet_wrap
	~~from plotnine.data import mtcars~~
	~~%matplotlib inline~~

	ggplot( country_totals.limit(10).toPandas() , aes(x='COUNTRY_CD' ) ) + geom_bar()		ggplot( country_totals.limit(10).toPandas() , aes(x='COUNTRY_CD' ) ) + geom_bar()
Line 91:		Line 89:
	<pre>		<pre>
	from plotnine import ggplot, geom_point, aes, stat_smooth, facet_wrap		from plotnine import ggplot, geom_point, aes, stat_smooth, facet_wrap
	~~from plotnine.data import mtcars~~
	~~%matplotlib inline~~

	ggplot( country_totals.limit(10).toPandas() , aes(x='COUNTRY_CD',y='count' ) ) + geom_bar(stat='identity')		ggplot( country_totals.limit(10).toPandas() , aes(x='COUNTRY_CD',y='count' ) ) + geom_bar(stat='identity')
	</pre>		</pre>

2019-10-31T15:26:14Z

Plotting Bar Graph

← Older revision		Revision as of 15:26, 31 October 2019
Line 85:		Line 85:

	ggplot( country_totals.limit(10).toPandas() , aes(x='COUNTRY_CD' ) ) + geom_bar()		ggplot( country_totals.limit(10).toPandas() , aes(x='COUNTRY_CD' ) ) + geom_bar()

	</pre>		</pre>

Line 96:		Line 95:

	ggplot( country_totals.limit(10).toPandas() , aes(x='COUNTRY_CD',y='count' ) ) + geom_bar(stat='identity')		ggplot( country_totals.limit(10).toPandas() , aes(x='COUNTRY_CD',y='count' ) ) + geom_bar(stat='identity')

	</pre>		</pre>

2019-10-31T15:26:02Z

Plotting graph

@@ Line 75: / Line 75: @@
 country_totals.show()</pre>
-==Plotting graph==
+==Plotting Bar Graph==
 <pre>

2019-10-31T15:25:04Z

← Older revision		Revision as of 15:25, 31 October 2019
Line 74:		Line 74:
	)		)
	country_totals.show()</pre>		country_totals.show()</pre>

			==Plotting graph==

			<pre>
			from plotnine import ggplot, geom_point, aes, stat_smooth, facet_wrap
			from plotnine.data import mtcars
			%matplotlib inline

			ggplot( country_totals.limit(10).toPandas() , aes(x='COUNTRY_CD',y='count' ) ) + geom_bar(stat='identity')

			</pre>

2019-10-30T23:28:37Z

← Older revision		Revision as of 23:28, 30 October 2019
Line 35:		Line 35:
	subsetDF.show()</pre>		subsetDF.show()</pre>

	~~This isn't working, but the intent is to map a new column based on a custom UDF function. I think it's maybe failing because functions are not allowed inside~~ a UDF.		Categorize records using a User Defined Fucntion (UDF)

	<pre>		<pre>