How to merge two arrays in JavaScript and de-duplicate items

How can we merge and de-duplicate arrays?

Here's a walkthrough of various methods you might use to merge arrays and remove duplicates. It includes techniques suitable for different JavaScript versions and data types.

1. Pure ES6: Spread and Set combo

With JavaScript ES6, merging using spread operator ... and deduplication by Set is the most direct method:

// Merging arrays like a Razor Scooter Gang
const merged = [...arr1, ...arr2];
// Removing duplicates - "There can be only one!"
const uniqueMerged = [...new Set(merged)];


2. Backwards Compatibility: ES5 Style

If you're interacting with ES5 environments, the Array.concat method merges arrays, while a custom function is used for deduplication:

// Merging arrays - Gotta Catch 'Em All!
var merged = arr1.concat(arr2);

// Deduplication - Repels duplicate Pokémon
function uniqueArray(array) {
  var unique = [];
  for (var i = 0; i < array.length; i++) {
    if (unique.indexOf(array[i]) === -1) {
      unique.push(array[i]);
    }
  }
  return unique;
}
// Applaud for our unique Pokémon trainers
var uniqueMerged = uniqueArray(merged);


3. Avoid Clone Wars: Prevent Side Effects

Before changing arrays, it's good practice to clone them to prevent unwanted alterations or side effects:

// Cloning - We're not in Kamino
const arr1Clone = [...arr1];
const arr2Clone = [...arr2];


4. Objects within arrays: Predicate Function

When objects within arrays present, a predicate function can help in comparisons during deduplication:

// Custom function to deal with those pesky objects
function mergeWithPredicate(arr1, arr2, predicate) {
  const merger = new Map(arr1.map(item => [predicate(item), item]));
  arr2.forEach(item => {
    if (!merger.has(predicate(item))) {
      merger.set(predicate(item), item);
    }
  });
  return Array.from(merger.values());
}


5. Trust the Tools: Lodash or Underscore.js

If you trust third-party libraries like Lodash or Underscore.js, the _.union function is nifty:

// Lodash for the rescue - Utility Belt equiped
var uniqueMerged = _.union(arr1, arr2);


Just don't forget to include Lodash in your CDN or module import.

Under the Hood: Use cases, Pitfalls, and Performance

Let's delve deeper and explore some additional scenarios, potential pitfalls and performance considerations.

1. Dealing with Objects in Arrays

If you're managing arrays of objects, make sure your predicate function optimally identifies unique items based on pertinent object properties:

const arr1 = [{ id: 1, name: 'Alice' }, { id: 2, name: 'Bob' }];
const arr2 = [{ id: 2, name: 'Bob' }, { id: 3, name: 'Charlie' }];

const uniqueObjects = mergeWithPredicate(arr1, arr2, item => item.id);


2. Efficiently managing Large Arrays

For large arrays, consider algorithm efficiency and seek speedier solutions. It will save you from lengthy processing times that resemble a snail race.

3. Multi-Browser Support: ES6 and Babel

When you're keen to use ES6 goodies but need to support older browsers, Babel can transpile your code to offer backward compatibility.

4. In-place Array Manipulation

When you wish to modify the original array and remove duplicates without spawning a new one, Array.prototype.splice in combination with Array.prototype.indexOf can be helpful:

// "Duplicates I sense in you."
for (var i = 0; i < merged.length; ++i) {
  for (var j = i + 1; j < merged.length; ++j) {
    if (merged[i] === merged[j]) {
      // "Exile you, we must."
      merged.splice(j--, 1);
    }
  }
}
