Ways to remove duplicates from list in Python

In this article, we'll learn several ways to remove duplicates from a list in Python. The simplest way to remove duplicates is by converting a list to a set.

Using set()

We can use set() to remove duplicates from the list. However, this approach does not preserve the original order.

a = [1, 2, 2, 3, 4, 4, 5]

# Remove duplicates by converting to a set
a = list(set(a))

print(a)

[1, 2, 3, 4, 5]

Note: Using set() removes duplicates but does not guarantee the order of elements.

Let's explore other different ways to remove duplicates from list:

Using For Loop

To remove duplicates while keeping the original order, we can use a loop (for loop) to add only unique items to a new list.

a = [1, 2, 2, 3, 4, 4, 5]

# Create an empty list to store unique values
res = []

# Iterate through each value in the list 'a'
for val in a:
  
    # Check if the value is not already in 'res'
    if val not in res:
      
        # If not present, append it to 'res'
        res.append(val)

print(res)

[1, 2, 3, 4, 5]

Note: This method preserves the original order but may be slower for large lists due to repeated membership checks ( ‘in’ operator).

Using List Comprehension

List comprehension is another way to remove duplicates while preserving order. This method provides a cleaner and one-liner approach.

a = [1, 2, 2, 3, 4, 4, 5]

# Create an empty list to store unique values
res = []

# Use list comprehension to append values to 'res'
# if they are not already present
[res.append(val) for val in a if val not in res]

print(res)

[1, 2, 3, 4, 5]

Note: This method is concise but it is less efficient for large lists similar to the loop approach.

Using Dictionary fromkeys()

Dictionaries maintain the order of items and by using dict.fromkeys() we can create a dictionary with unique elements and then convert it back to a list.

a = [1, 2, 2, 3, 4, 4, 5]

# Remove duplicates using dictionary fromkeys()
a = list(dict.fromkeys(a))

print(a)

[1, 2, 3, 4, 5]

Note: This method is generally faster than the loop method for larger lists.