1

For example, input a 1-d tensor: 

l_in = [1,1,2,2,3,4,5,5,1,3,5]

I want to remove the consecutive duplicates, which means the output should be:

l_out = [1,2,3,4,5,1,3,5]

However, the tf.unique function only returns unique elements, indicating that the last three elements will also be eliminated. The output of tf.unique is:

[1,2,3,4,5], [0,0,1,1,2,3,4,4,0,2,4] = tf.unique(l_in)

where the second item is the corresponding IDs.

Is there any way to only remove consecutive duplicates while preserving non-duplicated and non-unique elements?

benjaminplanche
  • 14,689
  • 5
  • 57
  • 69
GEORGE GUO
  • 117
  • 1
  • 1
  • 5

2 Answers2

6

For 1D tensors, using array rotation/shifting:

import tensorflow as tf

l_in = tf.constant([1,1,2,2,3,4,5,5,1,3,5])
l_left_shift = tf.concat((l_in[1:], [0]), axis=0)
mask_left_shift = tf.not_equal(l_in - l_left_shift, 0)
mask = tf.concat(([True], mask_left_shift[:-1]), axis=0)
l_out = tf.boolean_mask(l_in, mask)

with tf.Session() as sess:
    print(sess.run(l_out))
# [1 2 3 4 5 1 3 5]

(i.e. the idea is to subtract each element with its right neighbor, then mask out the neighbor if the subtraction result is 0)

benjaminplanche
  • 14,689
  • 5
  • 57
  • 69
  • Thanks a lot! It's a neat and brilliant idea! – GEORGE GUO Jul 10 '18 at 11:13
  • Not gonna lie, but this was very clever solution. It's shame that even after these many years TensorFlow team is not able to provide a standalone function that do this. Cheers! – Snehal Aug 16 '20 at 19:21
-3

Don't know about tensorflow, but since it seems to be a simple list you can use groupby from itertools quite easily:

from itertools import groupby  
l_out = [x[0] for x in groupby(l_in)]
print(l_out) # prints [1, 2, 3, 4, 5, 1, 3, 5]

See also: Removing elements that have consecutive duplicates in Python

Bernhard
  • 1,253
  • 8
  • 18
  • 1
    Thanks for your information. I've noticed that answer before I opened this question, but I didn't find a corresponding solution in tensorflow. – GEORGE GUO Jul 10 '18 at 09:50