27

Here is an example of the code I'm using: 

library(jsonlite)
library(curl)

#url
url = "http://www.zillow.com/search/GetResults.htm?spt=homes&status=001000&lt=000000&ht=010000&pr=999999,10000001&mp=3779,37788&bd=0%2C&ba=0%2C&sf=,&lot=0%2C&yr=,1800&singlestory=0&hoa=0%2C&pho=0&pets=0&parking=0&laundry=0&income-restricted=0&pnd=0&red=0&zso=0&days=36m&ds=all&pmf=0&pf=0&sch=100111&zoom=6&rect=-91307373,29367814,-84759521,35554574&p=1&sort=globalrelevanceex&search=maplist&rid=4&rt=2&listright=true&isMapSearch=true&zoom=6"

#json
results_data_json = fromJSON(txt = url)

I used to be able to run similar code to this with no issue. Now I'm getting the following error:

Error in feed_push_parser(buf) : 
  lexical error: invalid char in json text.
                                   <html><head><title>Zillow: Real
                 (right here) ------^

Any ideas around this?

  • 11
    It's likely you hit an automated query threshold limit (if you were to examine the whole text of that HTML it might even say so). IOW: you're not getting JSON back you're getting an HTML page that prbly has the reason you're not getting JSON back. – hrbrmstr Dec 06 '16 at 16:27
  • @hrbrmstr that's what I was thinking - it looks like I have to answer a captcha when I go to the actual url. Is there a way around this? Or do I have to wait until the limit resets? –  Dec 06 '16 at 16:31
  • I'd poke around their site to see what rate limiting they may state they do. You'll have to work within said limit and — aye — wait until your current block is released. – hrbrmstr Dec 06 '16 at 16:32
  • Interesting, I could not replicate the error. It runs fine without any error. Using Microsoft R 3.3.0 on Rstudio Version 1.0.44 – user5249203 Dec 06 '16 at 16:33
  • [Zillow does have APIs](http://www.zillow.com/howto/api/APIOverview.htm) which will probably give you a more polite response. It may be in XML, though. – alistaire Dec 06 '16 at 16:58
  • I received this error when processing data via the CMS API. I discovered that the Socrata API service was down and this was returning the error. If you are using a Socrata interface, you can check if the API service is down by looking here: https://status.socrata.com/. – StatsStudent Apr 30 '21 at 04:04
  • I had a similar error for applying parse_json() to response objects I had saved after using httr. I needed to library(httr) for it to work. (Though this is not directly relevant to this question, it is relevant to the error, which came up when I searched.) – Richard DiSalvo Mar 13 '22 at 21:02
  • This also happens when passing a vector of file names to `fromJSON` instead of a single file name. Got stuck on this for 15 mins, just posting this in case s/one else is in my case – Antoine Oct 13 '22 at 16:05

9 Answers9

7

This happened to me reading in a JSON from a file. The code worked one day, and then the next day I got this error. I was eventually able to circumvent the error, although I do not understand why my solution works. I found a GitHub post that suggested adding the readLines() function. Eg.

r_object <- fromJSON(readLines("file.json"), warn = F)

The "warn" argument is set to FALSE to suppress the warning message triggered by the lack of a final EOL in many JSON files.

Shuli K
  • 3
  • 2
ADF
  • 522
  • 6
  • 14
3

This seems like it is a problem related to your current working directory in R.

You can view your current working directory by entering "getwd()" into your RStudio Console.

If this directory path is not pointing to the directory containing your json file, then R doesn't know how to find the file.

You can change your current working directory with "setwd()". An example would be "setwd(/Users/me/Documents/jsonFiles/)"

Alternatively, you can have your code also point to the full directory path for any given file. This will make your code robust to any working directory changes that might happen over time. However, this does mean that using this code on someone else's computer would require these paths to be edited. You can find the full path to any given file by navigating in the terminal to the file of interest and typing "pwd" for 'present working directory'. This works for Mac and Linux. On a windows machine run "cd" from the terminal.

Chelsea Sho
  • 31
  • 1
3

Two very common causes:

  1. What you're providing to fromJSON() is not actually JSON, but HTML
  2. The JSON you're trying to parse is not valid JSON

If the error message contains '<!DOCTYPE html>'

That means the data you're giving to fromJSON() is in fact HTML (not JSON).

  lexical error: invalid char in json text.
                                       <!DOCTYPE html> <html>   <head>
                     (right here) ------^

That can happen if you're trying to access an API and it's redirecting you to a login page, or a 'access forbidden' page, or similar, or if the server is down and instead of giving you JSON data, it gives you a message like "Server down for maintenance", fromJSON() doesn't know how to handle that.

But to really figure out what's going on, write the output as a text file and explore it:

library(rvest)
library(tidyverse)
endpoint_url <- "http://localhost:3000/currencies" # Replace with your JSON url
html_nodes(read_html(endpoint_url), "body") %>% as.character %>% writeLines("file.txt")
readLines("file.txt")

In my case, I could spot:

You need to sign in or sign up before continuing.

Which told me the URL I was giving to fromJSON() requires a password or API key.

Invalid JSON

The fromJSON() function can only parse JSON if it's valid (formatted correctly).

JSON is sets of key-value pairs.

Valid JSON:

fromJSON('{"Name":"Jane", "Age":20}')
# $Name
# [1] "Jane"

# $Age
# [1] 20

Invalid JSON

fromJSON("sdfsdf")
Error: lexical error: invalid char in json text.
                                       sdfsdf
                     (right here) ------^

Here are some tips for diagnosing what's wrong with your JSON:

  • Firstly, you can paste your JSON into a JSON validator and it will give you a 'yes'/'no' answer as to whether your JSON is valid.
  • To explore invalid JSON, open your JSON file with a text editor and search for the part of the JSON that the error message refers to (i.e. the text the -----^ is pointing to).
  • Do something about it. In my case I was happy to delete that part of the json data (it meant losing that data, but that was tolerable since it would make the rest of the JSON valid). Another option is to fix the invalid JSON, that could be trickier but possible.
stevec
  • 41,291
  • 27
  • 223
  • 311
1

I can't replicate error neither. 

class(results_data_json)
[1] "list"

My sessioninfo:

R version 3.3.2 (2016-10-31)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 7 x64 (build 7600)

locale:
[1] LC_COLLATE=Spanish_Colombia.1252  LC_CTYPE=Spanish_Colombia.1252    LC_MONETARY=Spanish_Colombia.1252
[4] LC_NUMERIC=C                      LC_TIME=Spanish_Colombia.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] curl_2.4     jsonlite_1.1

loaded via a namespace (and not attached):
[1] tools_3.3.2
gonzalez.ivan90
  • 1,322
  • 1
  • 12
  • 22
1

Posting this in case someone else encounters this problem...I had to set the open file as my working directory and it began to work fine again. Wildly simple. Just right click on the open json file.

Leah
  • 11
  • 1
0

On my mac, it was caused by iCloud. The json file had been stored in the cloud, and R couldn't find it. Downloading the file fixed it. This may or may not have been what happened in your case.

Apologies that I'm nearly five years too late.

Ezekiel
  • 1
0

The same thing happened to me as @ADF above.... Had a whole project I was working on a few months ago, and when re-visiting, fromJSON suddenly didn't work.

I found that even though I have the file named as "name.json" in my file browser, my R directory showed it as "name.json.txt". Double-check your file name and what it shows in the directory match up!

Basalty
  • 55
  • 8
0

If you doubt the JSON file is valid, test it in another way (there are online tools). For me, the problem was I got the file name wrong and for some reason is gives the same error (instead of missing file)

ornit
  • 125
  • 1
  • 1
  • 5
  • Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Feb 06 '23 at 02:12
-7

As: R 3.2.2: "rjson" and "RJSONIO" package installed, but error using "fromJSON"

Installing the package alone doesn't suffice. You need to load the library too... Use library(rjson) or library(RJSONIO) or ... depending on which package you prefer, before calling any functions in it.

Community
  • 1
  • 1
d8aninja
  • 3,233
  • 4
  • 36
  • 60