Qsub and MPI example

This is a complete example of how to use qsub and MPI on a rocks cluster.

Note that this uses the default MPI on the system, which for current versions of rocks is OpenMPI.

Rather than copying and pasting these files, you might want to right click on the following links and use Save As instead:

• Makefile
• simple.c
• simple.sge

Or, download the files directly on the cluster with the following command:

wget -r --no-host-directories http://newton.i2lab.ucf.edu/simpletest/

Save this as Makefile: (PLEASE NOTE: make sure the lines in the Makefile are indented with a single TAB, not spaces!)

run: simple
        qsub simple.sge
        sleep 7
        qstat

simple: simple.c
        mpicc -o simple simple.c

clean:
        rm -f simple.sge.*

clobber: clean
        rm -f simple

Save this as simple.c:

/*The Parallel Hello World Program*/
#include <stdio.h>
#include <mpi.h>

main(int argc, char **argv)
{
   int node,i;

      printf("starting.\n");
      MPI_Init(&argc,&argv);
      MPI_Comm_rank(MPI_COMM_WORLD, &node);
      printf("started %d.\n",node);
      system("hostname");
      /*sleep(100);*/
      MPI_Finalize();
      printf("exiting.\n");
}

Save this as simple.sge:

#$ -cwd -pe mpich 4

echo === machinefile
# Include the machine file in your logs for debugging...
cat $TMPDIR/machines
echo ===
mpirun  ./simple

The make command should compile the job, run it, and wait for the output. Change the number after mpich to the number of cpus you want. Lines starting with #$ are equivalent to qsub command line options.

Note that it may take longer than 7 seconds for the job to start. (Note that OpenMPI picks up the hostfile from sge automatically, and specifying it on the command line confuses it.)