TORQUE notes

A complex queue setup

This basic installation works fine for one queue, but normally TORQUE users use it on a cluster with many nodes. The jobs must be scheduled so no user get's more priority than others. Another different and more complex setup, is the one that follows (the output of qmgr -c 'p s':

create queue routing
set queue routing queue_type = Route
set queue routing route_destinations = P16
set queue routing route_destinations += P8
set queue routing route_destinations += P4
set queue routing route_destinations += P1-2
set queue routing route_destinations += serial
set queue routing enabled = True
set queue routing started = True

create queue serial
set queue serial queue_type = Execution
set queue serial resources_max.mem = 20gb
set queue serial resources_max.ncpus = 1
set queue serial resources_max.nodes = 1
set queue serial resources_max.procs = 1
set queue serial resources_max.walltime = 720:00:00
set queue serial resources_min.procs = 1
set queue serial max_user_run = 4
set queue serial enabled = True
set queue serial started = True

create queue P1-2
set queue P1-2 queue_type = Execution
set queue P1-2 resources_max.mem = 20gb
set queue P1-2 resources_max.procs = 2
set queue P1-2 resources_max.walltime = 24:00:00
set queue P1-2 resources_min.procs = 1
set queue P1-2 enabled = True
set queue P1-2 started = True

create queue P4
set queue P4 queue_type = Execution
set queue P4 resources_max.mem = 12gb
set queue P4 resources_max.procs = 4
set queue P4 resources_max.walltime = 24:00:00
set queue P4 enabled = True
set queue P4 started = True

create queue P8
set queue P8 queue_type = Execution
set queue P8 resources_max.mem = 24gb
set queue P8 resources_max.procs = 8
set queue P8 resources_max.walltime = 24:00:00
set queue P8 max_user_run = 8
set queue P8 enabled = True
set queue P8 started = True

create queue P16
set queue P16 queue_type = Execution
set queue P16 resources_max.mem = 24gb
set queue P16 resources_max.procs = 16
set queue P16 resources_max.walltime = 24:00:00
set queue P16 max_user_run = 1
set queue P16 enabled = True
set queue P16 started = True

set server scheduling = True
(... rest of server configuration)

We now have a more complex setup, with different queues that have different attributes (number of processors, walltime, memory available etc.). More information about the attributes in the TORQUE Administration Guide. We also have a "routing" queue to route jobs to the right queues. TORQUE alone (with its very simple scheduler, pbs_sched cannot do that for us.

In that case, we need to use another scheduler. We are going to use Maui. Take a look at our page about Maui for maui installation and setup with TORQUE.

Error qmgr obj= svr=default: Bad ACL entry in host list MSG=First bad host

For some reason that I don't know why (Google didn't help) I got this error when running ./torque.setup root command as recommended by the TORQUE Administration Guide. So I ran pbs_server -t create and configured the queues manually.

pbsnodes showing down host

If the output of pbsnodes is:

# pbsnodes
localhost
     state = down
     np = 134
     ntype = cluster
     mom_service_port = 15002
     mom_manager_port = 15003

Check the content of the $TORQUE_HOME/server_priv/nodes and $TORQUE_HOME/mom_priv/config files, as well the hostname of the host you are running TORQUE server and clients.

Only one job run per time, even if there is resources free

There can be different reasons for this problem, like a misconfigured scheduler or queue. In our case, we were trying to configure TORQUE in a supercomputer environment, with lots of CPUs and memory.

We know that in a cluster environment, pbs_server is executed in a "master node" and pbs_mom on the others. A supercomputer environment is a single computer that both runs pbs_server and pbs_mom. There is no problem for that. When we execute pbsnodes command we will see one single node with lots of processors:

# pbsnodes
localhost
     state = down
     np = 134
     ntype = cluster
     mom_service_port = 15002
     mom_manager_port = 15003

After running the job, it just put the whole computer in a job-exclusive state, preventing other jobs to run:

# pbsnodes
bachianas
     state = job-exclusive
     np = 134
     ntype = cluster
     jobs = 0/36.bachianas
     status = rectime=1391694740,varattr=,jobs=36.bachianas,state=free,netload=17322218165,gres=,loadave=6.02,ncpus=136,physmem=269558192kb,availmem=271858224kb,totmem=280048592kb,idletime=420,nusers=1,nsessions=4,sessions=50921 55356 55540 192338,uname=Linux bachianas 2.6.16.60-0.42.10-default #1 SMP Tue Apr 27 05:11:27 UTC 2010 ia64,opsys=linux
     mom_service_port = 15002
     mom_manager_port = 15003

We should not treat this machine as a cluster. TORQUE sees the whole machine as a single node and locks it for one job, doesn't matter if the job uses 1 or 134 CPUs.

In this very specific case, the machine supported the NUMA architecture, so we can compile TORQUE with NUMA support to divide the CPUs in logical units. Check the TORQUE on NUMA systems section of the TORQUE Administration Guide for more information.

After configuring the

qrun: Unknown node-attribute

This error can have different causes. One is that your scheduler is probably not running or cannot communicate with pbs_server. Log of my maui setup showed something like:

02/14 12:00:13 MRMClusterQuery()
02/14 12:00:13 WARNING:  no resources detected
02/14 12:00:13 MRMWorkloadQuery()
02/14 12:00:13 WARNING:  no workload detected

In my case it was a problem with Maui. This was a problem with the RMCFG[HOSTNAME] setting. See our page about Maui for more details.

job failing into the wrong queue

Job failing in the wrong queue can have several reasons. The most common reason, IMO, is wrong queue configuration, not a problem with the job itself. But another common reason is wrong PBS directives. The following directive:

#PBS nodes=2:ppn=4

Is wrong. The right one would be:

#PBS -l nodes=2:ppn=4

See -l argument? That is the right way to use it.

job in 'R' state, but Time Use is always 00:00:00

After queueing a job, I executed qrun(8) on it, because Maui's scheduler was stopped, for testing purposes. After running qrun, it changed to the R state in qstat(1B), but the column Time Use didn't change. It was 00:00:00.

07/02/2014 15:53:53;0001;PBS_Server;Svr;PBS_Server;LOG_ERROR::is_request, bad attempt to connect from 200.XXX.XXX.XXX:1023 (address not trusted - check entry in server_priv/nodes)

Solution 1: In one case, the origin of this problem was that in $TORQUE_HOME/server_name it was correctly the hostname of the machine, but in /etc/hosts the hostname was associated with a external IP (200.XXX...) that nodes could not access. The solution was to change the IP entry in /etc/hosts to the internal IP address other nodes can access.

Solution 2: In another case, pbs_mom(8) was just having problems to execute in some nodes. Why? The log directory (usually /var) was full. I needed to delete some stuff.

Job being completed just after submit, with no further information

This error can be several reasons. It is better to check the logs in $TORQUE_HOME/server_logs.

There is an error like:

PBS_Server;Req;req_reject;Reject reply code=15001(Unknown Job Id), aux=0, ty pe=LocateJob, from user@node

where user is the user login name and node is the name of the node.

Server is OK, scheduler is OK too. It is probably some problem in the node. After logging in it, I realized that users had problems with the NFS partitions, since we made changes to the NFS server and the local firewall. umount and mount them again was not enough, so we had to reboot all nodes. So, it worked out.

Job never enters state R (Run)

This is a very common problem that can have many different reasons.

# checkjob 3038

(...)

job is deferred.  Reason:  NoResources  (exceeds available partition procs)
Holds:    Batch  Defer  (hold reason:  NoResources)
PE:  16.00  StartPriority:  31
cannot select job 3038 for partition DEFAULT (job hold active)

bachianas-1
     state = free
     np = 4
     ntype = cluster
     status = rectime=1413550150,varattr=,jobs=,state=free,netload=? 0,gres=,message=ERROR: torque spool filesystem full,loadave=0.00,ncpus=4,physmem=8077312kb,availmem=7872128kb,totmem=8077312kb,idletime=11,nusers=0,nsessions=0,uname=Linux bachianas 2.6.16.60-0.42.10-default #1 SMP Tue Apr 27 05:11:27
 UTC 2010 ia64,opsys=linux
     mom_service_port = 15002
     mom_manager_port = 15003

job is deferred.  Reason:  RMFailure  (cannot start job - RM failure, rc: 15085, msg: 'Time out MSG=connection to mom timed out')

11/05/2014 18:30:47;0008;PBS_Server.23876;Job;3080.bachianas.ufabc.edu.br;unable to run job, send to MOM '3364214663' failed

qrun: Time out MSG=connection to mom timed out 3083.bachianas.ufabc.edu.br