Get Twitter Feed as JSON without authentication

Question

I wrote a small JavaScript a couple of years ago that grabbed a users (mine) most recent tweet and then parsed it out for display including links, date etc.

It used this json call to retrieve the tweets and it no longer works.

http://twitter.com/statuses/user_timeline/radfan.json

It now returns the error:

{"errors":[{"message":"Sorry, that page does not exist","code":34}]}

I have looked at using the api version (code below) but this requires authentication which I would rather avoid having to do as it is just to display my latest tweet on my website which is public anyway on my profile page:

http://api.twitter.com/1/statuses/radfan.json

I haven't kept up with Twitter's API changes as I no longer really work with it, is there a way round this problem or is it no longer possible?

Can't do this anymore. Here's how to do it now: http://stackoverflow.com/questions/17049821/setting-up-twitter-api-getting-the-last-few-tweets — Ben, Mar 19 '14 at 20:39
@SimonBarker it is possible to use a json call to retrieve the tweets. I have just post an answer about it — RafaSashi, May 18 '15 at 18:43

score 17 · Answer 1 · edited May 23 '17 at 12:02

Previously the Search API was the only Twitter API that didn't require some form of OAuth. Now it does require auth.

Twitter's Search API is acquired from a third party acquisition - they rarely support it and are seemingly unenthused that it even exists. On top of that, there are many limitations to the payload, including but not limited to a severely reduced set of key:value pairs in the JSON or XML file you get back.

When I heard this, I was shocked. I spent a LONG time figuring out how to use the least amount of code to do a simple GET request (like displaying a timeline).

I decided to go the OAuth route to be able to ensure a relevant payload. You need a server-side language to do this. JavaScript is visible to end users, and thus it's a bad idea to include the necessary keys and secrets in a .js file.

I didn't want to use a big library so the answer for me was PHP and help from @Rivers' answer here. The answer below it by @lackovic10 describes how to include queries in your authentication.

I hope this helps others save time thinking about how to go about using Twitter's API with the new OAuth requirement.

According to [this page](https://dev.twitter.com/docs/api/1.1/get/search/tweets), even search methods now require authentication. — Nathan Osman, Jul 10 '13 at 23:56
Yep, that's correct. OAuth is now required for all Twitter API's. It's my impression that OAuth is significantly easier to accomplish on Ruby than on PHP, if anyone is trying to decide how best to approach it. — Dustin, Aug 16 '13 at 04:17

RafaSashi · Answer 2 · 2020-09-16T06:43:11.837

You can access and scrape Twitter via advanced search without being logged in:

https://twitter.com/search-advanced

GET request

When performing a basic search request you get:

https://twitter.com/search?q=Babylon%205&src=typd

q (our query encoded)
src (assumed to be the source of the query, i.e. typed)

by default, Twitter returns top 25 results, but if you click on all you can get the realtime tweets:

https://twitter.com/search?f=realtime&q=Babylon%205&src=typd

JSON contents

More Tweets are loaded on the page via AJAX:

https://twitter.com/i/search/timeline?f=realtime&q=Babylon%205&src=typd&include_available_features=1&include_entities=1&last_note_ts=85&max_position=TWEET-553069642609344512-553159310448918528-BD1UO2FFu9QAAAAAAAAETAAAAAcAAAASAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

Use max_position to request the next tweets

The following json array returns all you need to scrape the contents:

https://twitter.com/i/search/timeline?f=realtime&q=Babylon%205&src=typd

has_more_items (bool)
items_html (html)
max_position (key)
refresh_cursor (key)

DOM elements

Here comes a list of DOM elements you can use to extract

The authors twitter handle

div.original-tweet[data-tweet-id]

The name of the author

div.original-tweet[data-name]

The user ID of the author

div.original-tweet[data-user-id]

Timestamp of the post

span._timestamp[data-time]

Timestamp of the post in ms

span._timestamp[data-time-ms]

Text of Tweet

p.tweet-text

Number of Retweets

span.ProfileTweet-action–retweet > span.ProfileTweet-actionCount[data-tweet-stat-count]

Number of Favo

span.ProfileTweet-action–favorite > span.ProfileTweet-actionCount[data-tweet-stat-count]

Resources

https://code.recuweb.com/2015/scraping-tweets-directly-from-twitter-without-authentication/

when I tried "https://twitter.com/i/search/timeline?f=realtime&q=Babylon%205&src=typd". It doesn't give scroll_cursor. How I can get the next page content. — Kishore, Jan 12 '16 at 07:32
@KishoreKumarSuthar Check the blog linked at the foot of the post. It now uses max_position instead of scroll_cursor. I've updated the answer accordingly. — Polyducks, Mar 26 '16 at 18:17
It should be noted that this solution isn't useful for anything JS based due to CORS enforcement. — Polyducks, Mar 26 '16 at 18:46
Indeed, it is a server side solution to be used with `file_get_contents()` for example — RafaSashi, Mar 26 '16 at 18:52

score 6 · Answer 3 · answered Aug 08 '13 at 13:04

6

If you're still looking for unauthenticated tweets in JSON, this should work: https://github.com/cosmocatalano/tweet-2-json

answered Aug 08 '13 at 13:04

cosmocatalano

77
1
4

score 0 · Accepted Answer · edited Mar 16 '16 at 09:51

0

As you can see in the documentation, using the REST API you'll need OAuth Tokens in order to do this. Luckily, we can use the Search (which doesn't use OAuth) and use the from:[USERNAME] operator

Example: http://search.twitter.com/search.json?q=from:marcofolio
Will give you a JSON object with tweets from that user, where

object.results[0]

~~will give you the last tweet.~~

edited Mar 16 '16 at 09:51

Roko C. Buljan

196,159
39
305
313

answered Nov 19 '12 at 13:03

MarcoK

6,090
2
28
40

8

All API methods now require authentication. – Nathan Osman Jul 10 '13 at 23:54
@PaulSchreiber I just post a working alternative to the deprecated : `http://search.twitter.com/search.json?q=` – RafaSashi May 18 '15 at 18:39
Looks like that since 1.1 requests to api must be identified. https://dev.twitter.com/rest/public/search: "Please note that now API v1.1 requires that the request must be authenticated" – Lothre1 Jun 17 '16 at 15:36
1

This worked really well for me, hope it helps someone! https://github.com/jasonmayes/Twitter-Post-Fetcher – Elon Zito Sep 20 '16 at 19:01
{"errors":[{"code":215,"message":"Bad Authentication data."}]} – djack109 Oct 12 '16 at 18:28

score 0 · Answer 5 · answered Feb 03 '15 at 18:11

Here is a quick hack (really a hack, should be used with caution as its not future proof) which uses http://anyorigin.com to scrape twitter site for the latest tweets.

http://codepen.io/JonOlick/pen/XJaXBd

It works by using anyorigin (you have to pay to use it) to grab the HTML. It then parses the HTML using jquery to extract out the relevant tweets.

Tweets on the mobile site use a div with the class .tweet-text, so this is pretty painless.

The relevant code looks like this:

$.getJSON('http://anyorigin.com/get?url=mobile.twitter.com/JonOlick&callback=?', function(data){

  // Remap ... utf8 encoding to ascii. 
  var bar = data.contents;
  bar = bar.replace(/â€¦/g, '...');

  var el = $( '<div></div>' );
  el.html(bar);

  // Change all links to point back at twitter
  $('.twitter-atreply', el).each(function(i){
    $(this).attr('href', "https://twitter.com" + $(this).attr('href'))
  });

  // For all tweets
  $('.tweet-text', el).each(function(i){
    // We only care about the first 4 tweets
    if(i < 4) {
      var foo = $(this).html();
      $('#test').html($('#test').html() + "<div class=ProfileTweet><div class=ProfileTweet-contents>" + foo + "</div></div><br>");
    }
  });

});

score 0 · Answer 6 · answered Aug 01 '19 at 15:22

You can use a Twitter API wrapper, such as TweetJS.com which offers a limited set of the Twitter API's functionality, but does not require authentication. It's called like this;

TweetJs.ListTweetsOnUserTimeline("PetrucciMusic",
function (data) {
    console.log(data);
});

score -1 · Answer 7 · answered Dec 05 '12 at 18:42

-1

You can use the twitter api v1 to take the tweets without using OAuth. For example: this link turns @jack's last 100 tweets.

The timeline documentation is here.

answered Dec 05 '12 at 18:42

Dorukcan Kişin

1,121
2
14
29

"The Twitter REST API v1 is no longer active. Please migrate to API v1.1. https://dev.twitter.com/docs/api/1.1/overview." – Markus Bucher May 09 '17 at 09:52

score -4 · Answer 8 · answered May 06 '13 at 17:20

The method "GET statuses/user_timeline" need a user Authentification like you can see on the official documentation :

You can use the search method "GET search" wich not require authentification.

You have a code for starting here : http://jsfiddle.net/73L4c/6/

function searchTwitter(query) {
    $.ajax({
        url: 'http://search.twitter.com/search.json?' + jQuery.param(query),
        dataType: 'jsonp',
        success: function(data) {
            var tweets = $('#tweets');
            tweets.html('');
            for (res in data['results']) {
                tweets.append('<div>' + data['results'][res]['from_user'] + ' wrote: <p>' + data['results'][res]['text'] + '</p></div><br />');
            }
        }
    });
}

$(document).ready(function() {
    $('#submit').click(function() {
        var params = {
            q: $('#query').val(),
            rpp: 5
        };
        // alert(jQuery.param(params));
        searchTwitter(params);
    });
})

stolen from http://stackoverflow.com/questions/8890257/example-json-http-request-for-twitter-api — Andrew, Jul 11 '13 at 17:31

Get Twitter Feed as JSON without authentication

8 Answers8

Linked