Redirect crawlers to internal microservice in NGINX

Question

I'm running a client-side rendered React app built using create-react-app which I need to get OpenGraph meta tags working on. I've written some PHP (based on this https://rck.ms/angular-handlebars-open-graph-facebook-share/) which is designed to serve just OpenGraph meta tags for specific pages based on the contents of JSON files. What I need to do is pass requests from crawler user agents to this PHP page from inside NGINX.

server {
    server_name example.com www.example.com;

    root /var/www/example;
    index index.html;

    listen 80;

    location @crawler {
        fastcgi_pass unix:/run/php/php7.0-fpm.sock;
        fastcgi_index crawler.php;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        include fastcgi_params;
    }

    location / {
        if ($http_user_agent ~* "linkedinbot|googlebot|yahoo|bingbot|baiduspider|yandex|yeti|yodaobot|gigabot|ia_archiver|facebookexternalhit|twitterbot|developers\.google\.com") {
            proxy_pass @crawler;

        }
        try_files $uri /index.html;
    }
}

This is causing NGINX to fail with the following error:

May 10 00:01:59 ip-172-31-14-46 nginx[10400]: nginx: [emerg] invalid URL prefix in /etc/nginx/sites-enabled/example.com:23
May 10 00:01:59 ip-172-31-14-46 systemd[1]: nginx.service: Control process exited, code=exited status=1
May 10 00:01:59 ip-172-31-14-46 systemd[1]: Reload failed for A high performance web server and a reverse proxy server.

For reference - here's the content of the PHP file:

<?php
// 1. get the content Id (here: an Integer) and sanitize it properly
$uri = $_SERVER[REQUEST_URI];
$hash = hash('md5', $uri);

// 2. get the content from a flat file (or API, or Database, or ...)
$contents = file_get_contents("./meta/". $hash . ".json");
$data = array();
if ($contents) {
    $data = json_decode($contents);
}
$data = array_merge(json_decode(file_get_contents("./meta/default.json")), $data);

// 3. return the page
return makePage($data); 

function makePage($data) {
    // 1. get the page
    $pageUrl = "https://example.com" . $uri;
    // 2. generate the HTML with open graph tags
    $html  = '<!doctype html>'.PHP_EOL;
    $html .= '<html>'.PHP_EOL;
    $html .= '<head>'.PHP_EOL;
    $html .= '<title>'.$data->title.'</title>'.PHP_EOL;
    $html .= '<meta property="og:title" content="'.$data->title.'"/>'.PHP_EOL;
    $html .= '<meta property="og:description" content="'.$data->description.'"/>'.PHP_EOL;
    $html .= '<meta property="og:image" content="'.$data->poster.'"/>'.PHP_EOL;
    $html .= '<meta http-equiv="refresh" content="0;url='.$pageUrl.'">'.PHP_EOL;
    $html .= '</head>'.PHP_EOL;
    $html .= '<body></body>'.PHP_EOL;
    $html .= '</html>';
    // 3. return the page
    echo $html;
}

When you say Nginx is failing, do you have more info? An error, perhaps? — Charlie Schliesser, May 09 '18 at 23:59
Added the error to the main question - thanks for pointing that out! — Michael Smith, May 10 '18 at 00:03

score 1 · Answer 1 · answered May 10 '18 at 00:06

1

From the error, it looks like you are missing the URL prefix on the address passed to proxy_pass, perhaps it should be: fastcgi_pass http://unix:/run/php/php7.0-fpm.sock;

See this Q&A for the same issue: Nginx invalid URL prefix

answered May 10 '18 at 00:06

Charlie Schliesser

7,851
4
46
76

Thanks for the pointer - definitely something I need to look at in the crawler location block but unfortunately the error message refers to line 23 which is this one: proxy_pass @crawler; – Michael Smith May 10 '18 at 00:11
Isn't that because you're passing the @crawler configuration to proxy_pass in line 23, which includes a fastcgi_pass that doesn't contain the URL prefix? – Charlie Schliesser May 10 '18 at 00:12
Let me go check that out - hadn't thought of that! (In my defence it's just after 1am). – Michael Smith May 10 '18 at 00:14
Just tried this and it's failing with a different error message on the fastcgi_pass line. I'm pretty sure the fastcgi_pass is correct - as the linked StackOverflow question you referenced was talking about proxy_pass rather than fastcgi_pass - I think they have different requirements for links. – Michael Smith May 10 '18 at 00:16
May 10 00:37:04 ip-172-31-14-46 nginx[10652]: nginx: [emerg] invalid host in upstream "http://unix:/run/php/php7.0-fpm.sock" in /etc/nginx/sites-enabled/example.com:XX – Michael Smith May 10 '18 at 00:38
Start to search for these errors – https://serverfault.com/questions/640174/nginx-http-prefixes-in-upstream-server, https://stackoverflow.com/questions/35914245/unable-to-map-upstream-with-folder-in-nginx-server should point you in the right direction. You may only need to slightly change your config file – Charlie Schliesser May 10 '18 at 00:43

Redirect crawlers to internal microservice in NGINX

1 Answers1