How to change dataframe column names in PySpark?

Renaming with SQL Expressions and Alias

Rename multiple columns dynamically using selectExpr or alias:

df = df.selectExpr('old1 as new1', 'old2 as new2')
# Pro Programmer Tip: Think of it as the "I've got a new alias" command for columns!


Or using alias for a more SQL-like flavor:

from pyspark.sql.functions import col

df = df.select(col("old1").alias("new1"), col("old2").alias("new2"))
# "Alias" sounds cool, like you’re an SQL super-spy!


Both methods improve efficiency and maintain code cleanliness.

Column renaming with SQL

If you eat, breathe, and live SQL, PySpark offers sqlContext.sql to rename columns, assuming your DataFrame is registered as a temporary SQL table:

df.createOrReplaceTempView("temp_table")
df = sqlContext.sql("SELECT old_name AS new_name FROM temp_table")


Remember, SQL-based renaming's efficiency depends on the dataset size and available resources.

Renaming Columns: The Jedi Way

Some scenarios might require the force of a Jedi, not a blaster. Here are some Jedi secrets:

• Automate renaming using the Jedi mind trick (a dictionary):

  rename_dict = {'old1': 'new1', 'old2': 'new2'}
  for old, new in rename_dict.items():
      df = df.withColumnRenamed(old, new)
  # Execute the Jedi mind trick!
  

• Use Jedi quick reflexes (list comprehension) for efficient renaming:

  new_cols = [col_name.title() for col_name in df.columns]
  df = df.toDF(*new_cols)
  # Use the Yoda technique to quickly rename in one line!
  

• Harness the Jedi wisdom (mapper and lambda) for transformation-based renaming:

  mapper = lambda c: c.replace(" ", "_")
  df = df.toDF(*map(mapper, df.columns))
  # True power lies in understanding the power of lambda!
  

Keeping away from the Dark Side

Here are some Jedi moves to avoid the path to the dark side:

• Maintain order of columns when using toDF; it's just like keeping the force in balance.
• No crossing with reserved words by surrounding column names with backticks (`).
• Check for clones (duplicate names) after renaming, they lead to disturbances in the force.