DokuWiki

Note well, as10 and as11 are no longer used for the assistment project's production.

Asssitments production uses the machine as12, primarily, and as2 as a small NFS server of image files. We are attempting to divest of as2 due to its age.

–mvoorhis 25 Jan 2014.

Message-Id: <20090904181823.44508AF1DE@eressea>
Subject: procedure for restarting MONGREL on as10/as11
Date: Fri,  4 Sep 2009 14:18:23 -0400 (EDT)

Assistment not working? Check MONGREL processes on as10/as11.

Check individual machines by going to:

• http://as10.cs.wpi.edu:3000/
• http://as11.cs.wpi.edu:3000/

NOTE WELL: as of early October 2011, this checking technique appears to fail all the time. I have asked the assistment folks what the technique is, to check for positive asisstment-mongrel function. At this point, if http://www.assistments.org/ works, I guess all is well?? –mvoorhis 8 October 2011

If the machine does not respond here, then its mongrel is dead and needs to be restarted.

• login to as10/11
• become MONGREL user (command: sudo su - mongrel)
• cd /var/www/assistment/current

./stop.sh

[check, are the mongrel ruby jobs still there?] if they are still there:

kill -9 the mongrel-ruby jobs
cd ../shared/pids
rm * [i.e., nuke all pid files]
cd ../../current
./start.sh

Mongrel should be running again; verify by connecting to the machine's port 3000 with a web browser.

BOTH as10 and as11 need to be good for a connection to http://assistments.org/ to function correctly (???).

Script to restart mongrel shouldn't be concerned about the reason to do the restart; shouldn't be concerned about being clean or pilite with the currently running mongrel processes. Kill them, nuke the PIDS and restart.

(This script should ideally make checks to check WHY the server died?)

cd /var/www/assistment/current
kill -9 `ps auxw | grep '^mongrel.*mongrel.rails.start' | grep -v grep | awk '{print $2}'|xargs`
rm -f /var/www/assistment/shared/pids/*
sh ./start.sh

After this, perhaps a check to see if mongrel started:

#!/bin/sh
# check to see if we've got some mongrels running.
# if there are more htan zero we are happy;
# if there are zero we are not happy.
num=`ps auxw | grep '^mongrel.*mongrel.rails.start'|wc -l`
if [ $num -gt 0 ]
then
 echo "$num mongrels are running; good."
else
 echo "zero mongrels are running; bad."
fi

/var on as9 filles up pretty regularly, but apparently this doesn't matter?

/tmp on as11 appears to be filling up more commonly now (October 2011). I sent an email to the assistments people asking what the fix for this would be, since killing off mongrels and memcache and restarting them did NOT appear to cause http://as11.cs.wpi.edu:3000/ to become available again.