12

I have an application which records users visits. None of these visits are directly accessed, 100% of these visits are referred from another site.

I am passing $_SERVER['HTTP_REFERER'] through to the database. Approximately 35% of the logged entrees pass a referer, the rest are blank.

Is there a reason for this?

Andrew Marshall
  • 95,083
  • 20
  • 220
  • 214
reefine
  • 873
  • 2
  • 13
  • 25
  • `HTTP_REFERER` is an environment variable sent by the browser. Some browsers give the user the option to remove it when making a http request. This is not a PHP issue or an issue at all. – Khez Apr 13 '11 at 02:33
  • 1
    The `Referer:` is not a mandatory HTTP header. Its presence depends on browser settings (or proxies) and for privacy reasons is often disabled. – mario Apr 13 '11 at 02:35
  • I figured I'd get these responses. However, major market browsers like Chrome/IE/FireFox do indeed pass referrers. This makes up 99%+ of visitors. Also, most users don't disable passing referrers as they are not tech savvy enough to know how. – reefine Apr 13 '11 at 02:37
  • What is your reasoning behind knowing 100% of visits are referred from a specific site? – CrayonViolent Apr 13 '11 at 02:37
  • The nature of the application. Site.com passes a meta refresh to Site2.com which logs the visitor. – reefine Apr 13 '11 at 02:38
  • re: "figured I get these responses..." yes, by default it is passed. However, anonymous surfing is very popular. Adblocker addons and other anonymous surfing addons remove things like referrer. So even though browsers these days send it, it is common for people to remove it. – CrayonViolent Apr 13 '11 at 02:39
  • I see what you are saying and it is impossible for me to prove you wrong ;) I just would disagree that 65% of users are visiting from proxies/adblockers/manually disabled referrers in their browsers. Just doesn't seem right. – reefine Apr 13 '11 at 02:52

7 Answers7

18

There are a couple of number of reasons why HTTP_REFERER might be blank.

  1. You have to understand it's an environment variable given by the browser. Meaning users can remove it or even change it, if they so intend to.
  2. Users accessing the link from a bookmark, history or by typing the link manually do not have a referer.
  3. IE has also been known to remove the referer in situations revolving around javascript. Such as window.open, window.location and even setting target="_blank" in anchors or meta refresh.
  4. Clicking an embedded link in a chat application, PDF/Word/Excel document, will also not set a referer.
  5. Using AJAX, file_get_contents, fopen and other similar functions in other languages will probably not set a referer request.
  6. cURL, fsockopen, applications that have browser-like components might not set a referer.

There are probably more situations when this could happen, I'll update if I can think of anything that seems reasonable.

Andrew Marshall
  • 95,083
  • 20
  • 220
  • 214
Khez
  • 10,172
  • 2
  • 31
  • 51
  • 1
    in such situation, any alternative method that can keep this info on the site that referred to? i mean the info of the user is aboard from site A – 1myb Oct 18 '13 at 03:23
9

If a user visits your site directly, there is no referrer. It's also possible they have set it up so their browser never sends the referrer.

According to this answer, browsers do not necessarily send a referrer when doing a meta refresh.

Community
  • 1
  • 1
Andrew Marshall
  • 95,083
  • 20
  • 220
  • 214
  • I mentioned above how most users are probably not tech savvy enough to change their browser settings to disable referrers being passed. I know I would be making an assumption saying less than 1% of users do this. However, I think it is a good assumption and a conservative estimate. – reefine Apr 13 '11 at 02:39
  • well you asked how it was possible, not what is likely. If you're just going to go with your assumptions, why bother asking at all? – CrayonViolent Apr 13 '11 at 02:41
  • @user you failed to find an issue with why it isn't possible that they're visiting your site directly. I'd say _much_ more than "less than 1% of users" fall into this category. – Andrew Marshall Apr 13 '11 at 02:43
  • Like I said in a comment above, 100% of the visits are coming from Site.com which is passing as a meta refresh to Site2.com. The only possibility for direct visits might be search engine crawlers, scrapers, etc which is insignificant. – reefine Apr 13 '11 at 02:45
  • @user Are you absolutely sure about that? What about bookmarks? Crawlers, scrappers, etc. are still traffic. – Andrew Marshall Apr 13 '11 at 02:47
  • Yes, I am absolutely sure. When pulling these numbers, I'm only pulling unique IPs. I've also taken the liberty to do a random sample of 100 logged IPs that have no referrer being passed and none of them were search engines. – reefine Apr 13 '11 at 02:49
  • Very interesting! Thanks for finding that link :) I am going to try a 301 redirect instead and test against all the major browsers with default settings. Will update with my results. – reefine Apr 13 '11 at 02:57
1

When linking from one document to another in Internet Explorer 4.0 and later, the Referer header will not be sent when the link is from an HTTPS page to a non-HTTPS page. The Referer header also will not be sent when the link is from a non-HTTP(S) protocol, such as file://, to another page. for more info go to this link

Shiva
  • 11,485
  • 2
  • 67
  • 84
1

Browsers sometimes will include the referer in the request. But it is not mandatory to do so (the referer is 100% voluntary). Indeed there are various privacy and security issues surrounding the referer (for example, if an HTTPS site refers you to an HTTP site, the browser should not include the referring site as the referer). So don't rely on it.

yfeldblum
  • 65,165
  • 12
  • 129
  • 169
0

It is common when you are stuck finding why it is missing: - Sometime your referer is https and you are on http, it will be lost.

Otherwise: - User accessing by inputing url directly. - A user has bookmarked and come from bookmarks. - Sometime user keep the url default for browser (similar like bookmark) - Proxy surfying may remove referer. - accessing website as bots (search engine)

justnajm
  • 4,422
  • 6
  • 36
  • 56
0
  1. Direct access to your page (typing URL in address bar or from bookmarks, history, etc)
  2. Browser settings (disabled referrer or empty)
  3. if someone requests page content with file_get_contents() function...
Wh1T3h4Ck5
  • 8,399
  • 9
  • 59
  • 79
0

It also depends on the Transport layer, I encountered an issue where my Consumer Application A was running on the HTTP layer while the Application from where I was sending the request was running on the HTTPS layer.

Achal Parikh
  • 71
  • 2