Remove duplicates in a Django query

python

data-quality

database-queries

django-queries

byAnton Shumikhin·Mar 4, 2025

To zap duplicates, use Django's distinct() method on your QuerySet. Combine with values() when dealing with unique field combinations:

#bye_bye_duplicates
unique_combos = YourModel.objects.values('field1', 'field2').distinct()

For unique values of a lone field:

#lonewolf
unique_values = YourModel.objects.values('field1').distinct()

Note: If you're battling PostgreSQL, make sure ordering doesn't mess with your distinct():

#chaos_control
ordered_unique = YourModel.objects.order_by().values('field1', 'field2').distinct()

Hot tip: Using distinct() after values() fetches distinct field combinations or unique singles, but playful database-specific quirks might intervene.

Stifling duplicate entries

Grapple duplicates beyond simple queries. Use annotations and filters as your eraser:

Count down to destruction

Annotations with Count help spot duplicates:

#Who's Naughty?
from django.db.models import Count

duplicates = YourModel.objects.values('field').annotate(field_count=Count('field')).filter(field_count__gt=1)

Erasing duplicates

After identifying the baddies, kick them out:

#Extraction teame, assemble!
for entry in duplicates:
    YourModel.objects.filter(pk=entry['pk']).delete()

Warning: This operation zaps data—always keep a backup or fully understand the implications before proceeding with such a nuclear option.

Enforcing unique field constraints

Barricade your model against duplication:

#The Wall
class YourModel(models.Model):
    field = models.CharField(max_length=100, unique=True)
    # [...]

Craft unique value lists

Sleek single field value lists made possible with values_list() and distinct():

#Slimming diet for bloated lists
unique_emails = YourModel.objects.values_list('email', flat=True).distinct()

Pro-tip: Get an edge in data quality by working with the sleek, duplicate-free lists produced by this pattern.

Distinct difference in behaviors

Different database beasts have unique reactions to distinct(). Here's how you tame MySQL and PostgreSQL:

MySQL:

#Shoo duplicates! Shoo!
YourModel.objects.distinct()

PostgreSQL:

#Sly Fox! Distinct on specific fields only!
YourModel.objects.order_by('field').distinct('field')

Advanced exploration with `annotate()`

Cluster annotate() with values() for 'GROUP BY' operations:

#Get them in a bunch
from django.db.models import Count

unique_emails = (YourModel.objects
                .values('email')
                .annotate(email_count=Count('email'))
                .filter(email_count=1))

Long-term hygiene

Post clean-up, keep it spick and span:

Model validations: Build a pre-emptive strike team to check for duplicates before you hit save.
Database triggers and constraints: Your sentries for maintaining long-term stability.
Review imports and user inputs: Plug leakages that could let in more duplicates.

explain-codes / Python / Remove duplicates in a Django query

Linked

Convert Django Model object to dict with all of the fields intact



How to TRUNCATE TABLE using Django's ORM?



How do I clone a Django model instance object and save it to the database?



Delete multiple objects in django



How do I create a slug in Django?

