Obtaining reddit data

Question

I am interested in obtaining data from different reddit subreddits. Does anyone know if there is a reddit/other api similar like twitter does to crawl all the pages?

Answer 1

Yes, reddit has an API that can be used for a variety of purposes such as data collection, automatic commenting bots, or even to assist in subreddit moderation.

There are a few places to discover information on reddit's API:

• github reddit wiki -- provides the overview and rules for using reddit's API (follow the rules)
• automatically generated API docs -- provides information on the requests needed to access most of the API endpoints
• /r/redditdev -- the reddit community dedicated to answering questions both about reddit's source code and about reddit's API

If there is a particular programming language you are already familiar with, you should check out the existing set of API wrappers for various languages. Despite my bias (I am the package maintainer) I am quite certain PRAW, for python, has support for the largest number of reddit API features.

Answer 2

Note that if you are only reading data, and not interested into posting back to reddit, you can get quite a bit of data from the json feeds associated with each subreddit. With this method, you don't need to worry about an API at all -- you simply request the relevant json file and parse it in your language of choice.

Here's an example URL that will return a json object containing the hot posts from the Justrolledintotheshop subreddit: https://www.reddit.com/r/Justrolledintotheshop/top.json

In place of top, you can use hot, new, or controversial. When using top, you can add ?t=day to the end of the url to specify the top post for the day. Other valid values are hour, day, week, month, year, or all.

Answer 3

To parse JSON data from reddit with ajax/javascript.

Reddit has CORS enabled for GET requests.

Here as example, parse the last videos from reddit in JSON format:

xhr = new XMLHttpRequest
xhr.open("GET","https://www.reddit.com/r/videos/.json",true)
xhr.send(null)
xhr.onreadystatechange = function() {
  if(this.status === 200) {
    console.log(JSON.parse(xhr.responseText))
  }
}

https://developer.mozilla.org/fr/docs/Web/API/XMLHttpRequest

To go deeper, check out this question:

Change youtube video ID without page reloading