Skip Duplicates in Firebase Database

Question

I'm retrieving data from an RSS feed and adding in as structured date in Firebase's database. Every time the RSS feed is refreshed, a new UID is added but the issue is it adds the same values. In other words:

Is there a way to check if a value, e.g. update: "Fish showing on all.." already exists and skip instead adding a whole new UID?

I'm using feedparser (node.js) for the rss parsing and this is the code causing the duplicates on every refresh:

feedparser.on('readable', function() {
  // This is where the action is!

  var stream = this
    , meta = this.meta // **NOTE** the "meta" is always available in the context of the feedparser instance
    , item;

  while (item = stream.read()) {

      console.log(item);


var pubDate = item.pubDate;
    var date = new Date(pubDate);
    var description = item.description;

var parts = description.split(' - ', 2);

var time = parts[0];
var update  = parts[1];


  //  date.setTimezone  
  //  var time = date.getHours()+":"+date.getMinutes();


      var monthNames = ["January", "February", "March", "April", "May", "June",
    "July", "August", "September", "October", "November", "December"
                       ];

      var month = monthNames[date.getMonth()]+" "+date.getDate();

        var postData = {
        time: time,
        date: month,
        update: update,
        description: description,
        day: pubDate

  };


var newPostKey = firebase.database().ref().child('feed').push().key;
  var updates = {};
  updates['/feed/' + month + '/' + newPostKey] = postData;
  return firebase.database().ref().update(updates);

It seems like you want to guarantee that a certain value is unique: http://stackoverflow.com/questions/25294478/how-do-you-prevent-duplicate-user-properties-in-firebase — Frank van Puffelen, Jun 24 '16 at 04:27

score 0 · Accepted Answer · answered Jun 24 '16 at 17:32

The solution here, I think, is to use the parsed feed data to get some kind of unique identifier and use that as your Firebase Database key instead of using push. For example, many RSS feeds have a guid value that uniquely identifies a post, so you might be able to do something like:

firebase.database().ref('feed')
  .child(month).child(postData.guid)
  .update(updates);

That way, if you fetch the same item multiple times so long as the guid is the same it will always update in-place.

I can't believe I didn't think of this! Thank you, sir! – HalesEnchanted Jun 24 '16 at 21:53 — HalesEnchanted, Jun 24 '16 at 21:53

Skip Duplicates in Firebase Database

1 Answers1

Linked