How to remove duplicate files in Python?

Question

Aryan Kumar · Answer

Sure, here is a Python code that you can use to remove duplicate files:
Python
import os
import hashlib
def remove_duplicates(directory):
  """Removes duplicate files in the specified directory."""
  # Create a dictionary to store the hashes of all the files in the directory.
  file_hashes = {}
  for file in os.listdir(directory):
    file_path = os.path.join(directory, file)
    # Get the hash of the file.
    file_hash = hashlib.sha1(open(file_path, "rb").read()).hexdigest()
    # If the file hash is already in the dictionary, remove the file.
    if file_hash in file_hashes:
      os.remove(file_path)
    else:
      file_hashes[file_hash] = file_path
if __name__ == "__main__":
  directory = "/path/to/directory"
  remove_duplicates(directory)
This code first creates a dictionary to store the hashes of all the files in the directory. Then, it iterates over all the files in the directory. For each file, it gets the hash of the file and checks if the hash is already in the dictionary. If the hash is already in the dictionary, the file is removed. Otherwise, the hash of the file is added to the dictionary.
To use this code, you need to pass the path to the directory that you want to remove duplicate files from as the argument to the
remove_duplicates() function. For example, to remove duplicate files from the directory
/path/to/directory, you would run the following code:
Python
remove_duplicates("/path/to/directory")
This code will remove all duplicate files from the directory /path/to/directory.

forum

How to remove duplicate files in Python?

Revati S Misra

Can you answer this question?

1 Answers

Liked By