Nginx and PHP-FPM suddenly permission denied

Seemingly, without cause, one of our web applications began exhibiting strange behavior. It had run without needing much intervention for years. But suddenly files were not fully downloading, they were only getting a small partial download when the file download stalled.

So I went into debug mode. First checking the obvious, how was the load on the machine, was it failing on downloads because of a memory or cpu issue? I didn't think so, the load at that time was below average. I restarted the webserver just in case. Problem still present.

Next I took a look at nginx-error.log and there was an interesting string showing up in there [some details removed]:

[datetime] [crit] 63972#0: *149793 open() "/var/tmp/nginx/fastcgi_temp/0/59/0000005590" failed
 (13: Permission denied) while reading upstream, client: [client ip], server: [domain], request: 
 "GET [url]. HTTP/1.1", upstream: "fastcgi://unix:/tmp/php-fpm.sock:", host: "www.rfpdb.com", referrer: "[referrer]"

Permission denied, always the devil on a unix box, nine out of ten times some permission thing is the cause of things-not-working-right. This is all good to know, the files (for some reason) no longer have the correct permissions to be read.

] ls -l files/
# ...
-rwxrwx---  1 www  developers    25901 Jun 30  2010 10252
# ...

All files are still owned by www, which is to be expected since the webserver is running as www. The webserver should be able to read these. The webserver is still running as www right?

]$ ps -aux | grep nginx
root     5581  0.0  0.4 15528  5100  ??  Is    8:28PM   0:00.03 nginx: master process /usr/local/sbin/nginx
nobody   5661  0.0  0.5 15528  7100  ??  S     8:40PM   8:03.49 nginx: worker process (nginx)
tleen   18810  0.0  0.1  9124  1168   0  S+    6:13PM   0:00.01 grep nginx

My Nginx worker was running a nobody not www, which explains why that file cannot be read. That's not right. How did that happen?

Time to head over to the Nginx config and see what is happening. One thing I always do is Git All The Things!, including this Nginx configuration. I had just done an update to the Nginx configuration while updating a SSL certificate. I didn't really see why that would result in a permissions change for the webserver, but it wouldn't take long just to check.

[/usr/local/etc/nginx]# git diff 18e5:nginx.conf 3e74:nginx.conf
diff --git a/18e5:nginx.conf b/3e74:nginx.conf
index 1d60814..62515c7 100755
--- a/18e5:nginx.conf
+++ b/3e74:nginx.conf
@@ -1,4 +1,4 @@
-#user  nobody;
+user  nobody;
 worker_processes  1;

# more SSL stuff

Looks like while I was updating the config file I somehow unintentionally removed the hash from the line setting the nginx user to "nobody", so when I restarted with the new SSL setup Nginx started running as the new user.

One small character, big consequences.