How to archive an RSS feed?

Question

I need to take a few RSS feeds, and archive all the items that get added to them. I've never consumed or created RSS before, but I know xml, so the format seems pretty intuitive.

I know how to parse the feed: How can I get started making a C# RSS Reader?

I know I can't rely on the feed server to provide a complete history: Is it possible to get RSS archive

I know I'll have to have some custom logic around duplicates: how to check uniqueness (non duplication) of a post in an rss feed

My question is, how can I ensure I don't miss any items? My initial plan is to write a parser, where for each item in the feed: 1) Check to see if it's already in the archive database 2) If not, add it to the database If I schedule this to run once a day, can I be confident I won't be missing any items?

score 3 · Accepted Answer · answered May 01 '12 at 20:50

It depends on the feed, some sites publish articles very frequently and may have their RSS feed configured to show only the 10 most recent articles. Some sites are going to do the opposite.

Ideally your app should 'learn' the frequency this from the sites and tune itself to ping those sites based on the learnt frequency. (Ex: If you see new unique articles every time you ping, you'll need to ping more often, on the other hand if you see the same set of articles on multiple attempts, you may back off the next time).

score 1 · Answer 2 · answered Mar 30 '17 at 18:13

If you're open to relying on a service for this... I built my own RSS archival service (https://app.pub.center). You can access an RSS feed's data via our API.

Page 1 of The Atlantic https://pub.center/feed/02702624d8a4c825dde21af94e9169773454e0c3/articles?limit=10&page=1

Page 2 of The Atlantic https://pub.center/feed/02702624d8a4c825dde21af94e9169773454e0c3/articles?limit=10&page=2

The REST API is free. We have pricing plans for push notifications (email, SMS, your custom API endpoint)

score 0 · Answer 3 · edited Jun 20 '20 at 09:12

Use a series of decisions based on the feed and the storage limitations. For example:

    Connect to the Web site, and download the XML source of the feed. The Feed Download Engine downloads feeds and enclosures via HTTP or Secure Hypertext Transfer Protocol (HTTPS) protocols only.

    Transform the feed source into the Windows RSS Platform native format, which is based on RSS 2.0 with additional namespace extensions. (The native format is essentially a superset of all supported formats.) To do this, the Windows RSS Platform requires Microsoft XML (MSXML) 3.0 SP5 or later.

    Merge new feed items with existing feed items in the feed store.
    Purge older items from the feed store when the predetermined maximum number of items have been received.

    Optionally, schedule downloads of enclosures with Background Intelligent Transfer Service (BITS).

Use HTTP to its fullest to minimize wasted bandwidth:

To limit its impact on servers, the Feed Download Engine implements HTTP conditional GET combined with Delta encoding in HTTP (RFC3229) World Wide Web link. This implementation allows the server to transfer a minimal description of changes instead of transferring an entirely new instance of a resource cached on the client. The engine also supports compression using the HTTP gzip support of Microsoft Win32 Internet (WinInet).

A successful synchronization means that the feed was successfully downloaded, verified, transformed into the native format, and merged into the store. A server response of HTTP 304 Not Modified in response to a HTTP conditional GET (If-Modified-Since, If-None-Match, ETag, and so on) also constitutes success.

And define criteria for removal:

The following properties directly affect the number of items that remain after a synchronization operation.

    PubDate—used to determine the "age" of items. If PubDate is not set, LastDownloadTime is used. If the feed is a list, the order of items is predetermined and PubDate (if present) is ignored.

    MaxItemCount—a per-feed setting that limits the number of archived items. The feed's ItemCount will never exceed the maximum, even if there are more items that could be downloaded from the feed.

    ItemCountLimit—the upper limit of items for any one feed, normally defined as 2500. The value of MaxItemCount may not exceed this limit. Set MaxItemCount to ItemCountLimit to retain the highest possible number of items.

References

Understanding the Feed Download Engine

How to archive an RSS feed?

3 Answers3