I recently had a problem where a website runs normally all month, but after an “update notification Email” was sent, the website ground to a halt.
The first useful command I found was `free` which displays the memory status.
total used free shared buffers cached Mem: 8176976 1245760 6931216 24060 190016 696392 -/+ buffers/cache: 359352 7817624 Swap: 4194300 0 4194300
This is the current status, which shows a 0 in the swap space usage. This is good! At the time of my problem, I was showing 200000 swap in use.
Digging deeper, my /etc/apache2/mods-enabled/mpm_prefork.conf file had the following
StartServers 5 MinSpareServers 5 MaxSpareServers 10 MaxRequestWorkers 256 MaxConnectionsPerChild 0
Pretty standard I thought, but then I found the following script on http://www.stephenwalker.com/2013/09/finding-apache-average-memory-usage-on-debian-wheezy/
sudo ps -ef | grep apache2 | grep -v ^$USER | awk '{ print $2 '} | xargs sudo pmap -d | grep ^mapped: | awk '{ print $4 }' | cut -dK -f1 | awk '{SUM += $1} END { print SUM/NR }'
I ran this to discover my apache process’s were averaging about 170Mb. My 4Gb server could only handle 4096/170 = 24 simultaneous requests in memory. This is why it was disk swapping and grinding to a halt.
I changed MaxRequestWorkers to 20, and this stopped the disk swapping, and seemingly fixed the website speed.
However, a few days later I had the website grinding to a halt again. Further investigation, using
sudo netstat -plan | grep 443
showed I had about 300 users waiting for requests. It was clear that my 20 MaxRequestsWorkers was not enough for my peaks. My CPU load was running at 0.4 so processing power was not the bottle neck. Searching for a solution, I began to realise that 170Mb was rather excessive as others were saying their processes run at 10-20Mb.
To solve the bottleneck issue, I would have to reduce the memory of the apache process in order to allow more simultaneous connections. The standard way is to remove unwanted modules. I removed the mysql php extensions and removed the apache module mod_access_compat. This save 2Mb, not curing my problem but allowing me a minimal configuration once solved.
mod_access_compat
This module is added to apache 2.4 for backwards compatibility in the apache configurations files. It is a simple task though to update your configurations thus removing the need for the module. The following is the main change I had to perform.
Order allow,deny Allow from all #change to Require all granted
The final solution
I looked at the processor usage on another linux box, and found that was running at 10Mb. This finally led me to the answer. I ran phpinfo() on both machines, and compared them side by side.
I discovered that the machine with the problem had the php command get_browser() enabled. This involves downloading a file called browscap.ini which contains information on all the browsers. Although only 17Mb, each process loads the information into memory taking up 150Mb. Dont ask me why!
Removing this option took my process down to 8Mb !!!
I can now handle 256 (a system default) without the need for disk swapping.