what is the best universal watchdog script?

Question

OK, so I'm familiar with several methods of implementing a "watchdog" script. The problem is, that none of these check for a "hung" or unresponsive process. They all just check if the process is still present.

Perhaps I'm showing my lack of programming knowledge, but I'm under the impression, a process can sometimes continue to appear running to this system but in fact be crashed/hung.

Is there a way to detect this condition and trigger (pkill blah && blah) the process in question?

Some examples of what I'm NOT looking for:

• http://blog.eracc.com/2010/05/08/linux-monitor-a-service-with-a-watchdog-script/
• http://community.webfaction.com/questions/6157/watchdog-script-to-keep-process-running-with-cron
• linux + create simple watch dog process in shell script
• Monitoring a one-off service

• cron jobs like:

  #!/bin/sh
  if [ ` pgrep processname -c` -le "0" ]; then processname
  

Answer 1

Look into the Monit or M/Mmonit utility.

Its uses are covered well here on Server Fault, as well as in the documentation examples.

You can check process by PID or presence easily; but additional parameters like CPU utilization or RAM consumption can also be triggers for a variety of actions.