26

My nginx site root points to a symlink. If I alter the symlink (aka deploy a new version of the website) the old version of the php script keeps showing up. That smells like cache, or a bug.

First it looked like Nginx was caching the symlinked dir, but reloading/restarting/killing and starting nginx didn't fix it, so I restarted php5-fpm - this fix my issue.

But I dont want to restart nginx and/or php5-fpm after a deploy - I want to know why there is such a cache (or bug), and why it didn't work properly.

Usefull information:

  • OS: Ubuntu 13.10 (GNU/Linux 3.8.0-19-generic x86_64)
  • Nginx: via ppa:nginx/stable
  • PHP: via ppa:ondrej/php5 (php5-fpm)

Nginx site config:

root /home/rob/sandbox/deploy/public/;
index index.php index.html index.htm;
location / {
    try_files $uri $uri/ /index.php?$args;
}
location ~ \.php$ {
    try_files $uri =404;
    fastcgi_split_path_info ^(.+\.php)(/.+)$;
    include fastcgi_params;
    fastcgi_index index.php;
    fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
    fastcgi_pass php;
}

Nginx server config (partly, rest is default):

http {
    sendfile off;
    upstream php {
        server unix:/var/run/php5-fpm.sock;
    }
}

Tree for /home/rob/sandbox:

├── deploy -> web2
├── web1
│   └── public
│       └── index.php (echo ONE)
└── web2
    └── public
        └── index.php (echo TWO)
  • request: http://localhost/index.php
  • expected response: TWO
  • actual response: ONE

Part of the output from realpath_cache_get()

[/home/rob/sandbox/deploy/public/index.php] => Array (
    [key] => 1.4538996210143E+19
    [is_dir] => 
    [realpath] => /home/rob/sandbox/web2/public/index.php
    [expires] => 1383730041
)

So this means deploy/public/index.php is properly linked to web2/public/index.php, right? Well, even with the correct paths in the realpath_cache list, the respone still is ONE.

After rm deploy and ln -s web2 deploy Nginx was restarted, no effect. Restarting php5-fpm after this gives the expected response of 'TWO'.

It's good to know that beside the index.php files, I did some test with .css and .js files. After removing and recreating the symlink from/to web1 and web2, nginx will respond with the correct contents of the files.

What did I miss, what am I not seeing?

Rob Gordijn
  • 6,381
  • 1
  • 22
  • 29
  • 2
    You said php was the key, then the most important configuration files are the php-fpm ones (pool settings, php ini, apc settings?). And you could also try with a php-fpm reload, it's a graceful operation and after moving directories it seems legit. – regilero Nov 06 '13 at 17:03
  • Thanks, you pointed me in the right redirection. (see my own answer) And after all, the graceful reload of php-fpm isn't that bad I guess. – Rob Gordijn Nov 06 '13 at 20:05
  • I think two of my issues are related to this issue:https://stackoverflow.com/questions/48812327/single-laravel-route-not-found-404-on-nginx-production-server and https://stackoverflow.com/questions/48667130/laravel-5-4-stuck-in-maintenance-mode ... symlink deploy an old state is remembered... so is symlink the best way to go? Would it be better to just delete the directory, then make it again with new code? The answers below are pretty scary! – Iannazzi Feb 15 '18 at 19:19

3 Answers3

33

Configure your nginx using $realpath_root. It should help.

fastcgi_param SCRIPT_FILENAME $realpath_root$fastcgi_script_name;
fastcgi_param DOCUMENT_ROOT $realpath_root;

Kudos go to Vitaly Chirkov (https://stackoverflow.com/a/23904770/219272).

Community
  • 1
  • 1
sobstel
  • 1,432
  • 16
  • 11
  • Hi there, thanks to Google, Stackoverflow, and you, for saving my life today. Same issue, this trick worked perfectly. Should be the accepted answer. Further info: https://hackernoon.com/truly-atomic-deployments-with-nginx-and-php-fpm-aed8a8ac1cd9 – Ben Apr 16 '18 at 09:56
  • I've lost like four hours because of this easter egg. – reim Jul 23 '21 at 12:54
7

Once I altered the realpath_cache_ttl to '2' (That should fix it) the incorrect content was still showing.

After some digging in the loaded mods for php-fpm, I discovered that opcache was up and running. Disabling that will clear the cached realpath's when the ttl is over.

I don't wanna lower the realpath cache ttl to much,so I will settle in with a reload of php-fpm, since it is graceful. I hope this thread and my answers will help others ;)

Rob Gordijn
  • 6,381
  • 1
  • 22
  • 29
  • I'm having the same issue. What exactly do you do to get it to work? Do you just reload (not restart, right?) php-fpm now? – danronmoon Nov 25 '13 at 01:32
  • lowering the 'realpath_cache_ttl' value does work. (if you don't have any other caching enabled, like opcache). But lowering the value means that php must work harder. So I went for a restart, since my intallation of php-fpm didn't reload properly. I'm not sure yet if the restart is graceful, or that I could reinstall to reload it. – Rob Gordijn Nov 25 '13 at 06:38
  • @RobGordijn have you found out if the restart is graceful or not? Does it affect users currently exploring the website? – rzb Jan 05 '17 at 15:49
  • A reload does not affect current transactions. A restart isn't graceful at al ;) – Rob Gordijn Jan 07 '17 at 11:36
0

I had exactly same problem and none of the tricks did help. Beside all tricks listed on this page I ensured opcache is disabled then nginx cache was also disabled. I set 

sendfile off;

It was described somewhere on stackoverflow.

I started to strace the php and nginx and it turned out that some libraries are read new location but also some were read from OLD location that symlink does not point to anymore. On top of that updating the php script inside OLD directory was always showing properly - so that did not look like cache to me. To make it more confusing command line worked fine and followed the symlink to NEW location. I was pulling hair from my head over this!

It turned out that responsible for all this was the composer cache - it was caching some libraries. I know not everyone uses it, but if you do and have similar problem it is worth checking. I have vendor at the same level as the deployment scripts and I assume composer cache was causing a lot of confusion which location should be used. After cleaning it up with 

composer clear-cache

System started to behave as expected.

I hope it will help someone.

J. Raczkiewicz
  • 289
  • 3
  • 7