Why did my apache2 crash? Althoought getting much more space and memory on my server

Question

My website stopped loading on the browser, and worked back again just after I restarted the apache2.

I use Amazon AWS ububuntu server moved from t2.micro to t2.large (vCPU: 2, CPU Credits / hour: 36, Mem (GiB) 8) 3 months ago.

This is the log of the apache2:

[Tue Dec 15 09:04:41.976924 2015] [autoindex:error] [pid 31049] [client 66.66.66.666:53331] AH01276: Cannot serve directory /var/www/html/wp-content/themes/twentyfifteen-child/images/: No matching DirectoryIndex (index.html,index.cgi,index.pl,index.php,index.xhtml,index.htm) found, and server-generated directory index forbidden by Options directive, referer: http://www.example.com/blog/
[Tue Dec 15 18:28:59.565037 2015] [autoindex:error] [pid 31987] [client 50.115.68.210:13697] AH01276: Cannot serve directory /var/www/html/wp-content/themes/twentyfifteen-child/images/: No matching DirectoryIndex (index.html,index.cgi,index.pl,index.php,index.xhtml,index.htm) found, and server-generated directory index forbidden by Options directive, referer: http://www.example.com/wp-content/themes/twentyfifteen-child/style.css?ver=4.3.1
[Wed Dec 16 02:34:52.643069 2015] [mpm_prefork:error] [pid 27704] AH00161: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting
[Wed Dec 16 10:40:04.422974 2015] [core:notice] [pid 27704] AH00051: child pid 28345 exit signal Segmentation fault (11), possible coredump in /etc/apache2
[Wed Dec 16 10:40:04.423039 2015] [core:notice] [pid 27704] AH00051: child pid 28512 exit signal Segmentation fault (11), possible coredump in /etc/apache2
[Wed Dec 16 10:40:04.423089 2015] [core:notice] [pid 27704] AH00051: child pid 28347 exit signal Segmentation fault (11), possible coredump in /etc/apache2
[Wed Dec 16 10:40:04.423174 2015] [core:notice] [pid 27704] AH00051: child pid 27850 exit signal Segmentation fault (11), possible coredump in /etc/apache2
[Wed Dec 16 10:40:04.423220 2015] [core:notice] [pid 27704] AH00051: child pid 13545 exit signal Segmentation fault (11), possible coredump in /etc/apache2
[Wed Dec 16 10:40:04.423259 2015] [core:notice] [pid 27704] AH00051: child pid 31810 exit signal Segmentation fault (11), possible coredump in /etc/apache2
[Wed Dec 16 10:40:04.423297 2015] [core:notice] [pid 27704] AH00051: child pid 28520 exit signal Segmentation fault (11), possible coredump in /etc/apache2

The problem really started around 10:40 (you can see on the log)

And this is the MaxRequestWorkers.

<IfModule mpm_prefork_module>
        StartServers              5
        MinSpareServers           5
        MaxSpareServers          10
        MaxRequestWorkers        256
        MaxConnectionsPerChild   1000
</IfModule>

Can anyone help me debug the problem on my server?

what might be the problem?

EDIT:

GBD Output:

(gdb) gdb apache2 -core /tmp/apache-coredumps/core
[1]+  Stopped                 gdb /usr/sbin/apache2
root@ip-66-66-66-66:/# db apache2 -core /tmp/apache-coredumps/core
db: command not found
root@ip-66-66-66-66:/# gdb apache2 -core /tmp/apache-coredumps/core
GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.2) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from apache2...Reading symbols from /usr/lib/debug//usr/sbin/apache2...done.
done.
[New LWP 6194]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

warning: the debug information found in "/usr/lib/debug//usr/lib/php5/20121212/mysql.so" does not match "/usr/lib/php5/20121212/mysql.so" (CRC mismatch).


warning: the debug information found in "/usr/lib/debug/usr/lib/php5/20121212/mysql.so" does not match "/usr/lib/php5/20121212/mysql.so" (CRC mismatch).


warning: the debug information found in "/usr/lib/debug//usr/lib/php5/20121212/mysqli.so" does not match "/usr/lib/php5/20121212/mysqli.so" (CRC mismatch).


warning: the debug information found in "/usr/lib/debug/usr/lib/php5/20121212/mysqli.so" does not match "/usr/lib/php5/20121212/mysqli.so" (CRC mismatch).


warning: the debug information found in "/usr/lib/debug//usr/lib/php5/20121212/pdo_mysql.so" does not match "/usr/lib/php5/20121212/pdo_mysql.so" (CRC mismatch).


warning: the debug information found in "/usr/lib/debug/usr/lib/php5/20121212/pdo_mysql.so" does not match "/usr/lib/php5/20121212/pdo_mysql.so" (CRC mismatch).

Core was generated by `/usr/sbin/apache2 -k start'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  _zend_mm_free_int (heap=0x7f4b910ecfb0, p=0x7f4b910f0e70) at /build/php5-pO28mL/php5-5.5.9+dfsg/Zend/zend_alloc.c:2104
2104    /build/php5-pO28mL/php5-5.5.9+dfsg/Zend/zend_alloc.c: No such file or directory.

score 1 · Answer 1 · answered Dec 16 '15 at 11:33

Your server might just had a simple overload. I can't tell, even for a t2.micro, because it depends on too much parameters.

There are options to prevent crashes like these. One of the simplest is to install Monit. This is a service that can monitor Apache, and then restart it if it sees that there are too many connections. You can use Monit for other processes as well.

score 0 · Answer 2 · edited May 23 '17 at 12:41

[core:notice] [pid 27704] AH00051: child pid 28345 exit signal Segmentation fault (11), possible coredump in /etc/apache2

It looks like your application might have a bug. You can attach gdb to one of the processes an wait for it to crash again then get a backtrace and work from there.

You might also want to look in your other logs to see is there is any correlated relevant information which can help track the issue down.

Why did my apache2 crash? Althoought getting much more space and memory on my server

2 Answers2