ICE: Removing duplicates from arrays


Here’s an ICE tree that removes all duplicate elements from an array. It uses Generate Sample Set, so there’s no repeat nodes. But it relies on the fact that you can feed in an array of indices into Remove from Array, and Remove from Array doesn’t care if that array of indices itself contains duplicate. So, if you plug the array [1,1,1,2,2,2,3,3,3] in the Index port, Remove from Array will nicely remove elements 1, 2, and 3 with no complaints.

GenerateSampleSet_Remove_Duplicates

Unlike some other methods, this works with scalars too:

GenerateSampleSet_Remove_Duplicates_Scalar

Using Generate Sample Set to avoid the Repeat node


Hat tip to Oleg who posted this tip on the mailing list some time ago.

The basic ideas is that instead of looping over an array with a Repeat with Counter, you use Generate Sample Set to get a data set, and you do everything in a per-generated element context.

GenerateSampleSEt_vs_Repeat

As an example, let’s revisit the problem of taking one array and creating a new array where each element is the sum of the elements before it.

Array_accumulation

The old way, with a Repeat with Counter node, looks like this:
RepeatWithCounter

Using Generate Sample Set, you can work with a data set and use that to access the array elements:
GenerateSampleSet

Generate Sample Set is set up to give an exact number of samples:
GenerateSampleSet-PPG