2017-09-19 13:58:08,193 INFO input command: pbs_benchpress -f test/tests/functional/pbs_enforce_on_exclhost.py -o ptl_output.txt 2017-09-19 13:58:08,193 INFO param: None 2017-09-19 13:58:08,193 INFO ptl version: 17.1.0 2017-09-19 13:58:08,193 INFO platform: Linux pbspro-ptl 3.16.0-4-amd64 #1 SMP Debian 3.16.39-1 (2016-12-30) x86_64 2017-09-19 13:58:08,193 INFO python version: 2.7.9 2017-09-19 13:58:08,194 INFO user: root 2017-09-19 13:58:08,194 INFO -------------------------------------------------------------------------------- 2017-09-19 13:58:08,194 INFO Cleaning up temporary files 2017-09-19 13:58:08,201 INFO ====================================================================== 2017-09-19 13:58:08,201 INFO suite name: Test_enforce_on_exclhost 2017-09-19 13:58:08,201 INFO suite docstring: Check whether or not jobs with exceeded limits (burst, sum, mem) will survive the limit enforcement if the enforce_on_exclhost is false 2017-09-19 13:58:08,203 INFO ====================================================================== 2017-09-19 13:58:08,203 INFO ============================================ 2017-09-19 13:58:08,203 INFO Entered Test_enforce_on_exclhost setUpClass 2017-09-19 13:58:08,203 INFO ============================================ 2017-09-19 13:58:08,204 INFOCLI2 pbspro-ptl: id pbsuser 2017-09-19 13:58:08,206 INFOCLI2 pbspro-ptl: id pbsuser1 2017-09-19 13:58:08,208 INFOCLI2 pbspro-ptl: id pbsuser2 2017-09-19 13:58:08,210 INFOCLI2 pbspro-ptl: id pbsuser3 2017-09-19 13:58:08,219 INFO server pbspro-ptl: server operating mode set to cli 2017-09-19 13:58:08,219 INFOCLI pbspro-ptl: /usr/bin/qstat -Bf pbspro-ptl 2017-09-19 13:58:08,222 INFO server pbspro-ptl: version 17.1.0 2017-09-19 13:58:08,222 INFO expect action: created new action kicksched 2017-09-19 13:58:08,222 INFO expect action: added action kicksched to server pbspro-ptl 2017-09-19 13:58:08,236 INFOCLI2 pbspro-ptl: sudo -H /usr/sbin/pbsfs 2017-09-19 13:58:08,243 INFOCLI2 pbspro-ptl: sudo -H /bin/cat /var/spool/pbs/sched_priv/resource_group 2017-09-19 13:58:08,247 INFOCLI2 pbspro-ptl: sudo -H /bin/cat /var/spool/pbs/sched_priv/holidays 2017-09-19 13:58:08,252 INFOCLI pbspro-ptl: /usr/bin/qmgr -c list sched default 2017-09-19 13:58:08,265 INFOCLI2 pbspro-ptl: sudo -H /usr/sbin/pbsfs 2017-09-19 13:58:08,272 INFOCLI2 pbspro-ptl: sudo -H /bin/cat /var/spool/pbs/sched_priv/resource_group 2017-09-19 13:58:08,277 INFOCLI2 pbspro-ptl: sudo -H /bin/cat /var/spool/pbs/sched_priv/holidays 2017-09-19 13:58:08,285 INFO ============================================= 2017-09-19 13:58:08,285 INFO Completed Test_enforce_on_exclhost setUpClass 2017-09-19 13:58:08,285 INFO ============================================= 2017-09-19 13:58:08,285 INFO test name: test_job_not_killed_cpuaverage (tests.functional.pbs_enforce_on_exclhost.Test_enforce_on_exclhost)... 2017-09-19 13:58:08,285 INFO test start time: Tue Sep 19 13:58:08 2017 2017-09-19 13:58:08,285 INFO test docstring: Set cpuaverage enforcement and run a job with exclhost Check wheter or not the job will survive 2017-09-19 13:58:08,285 INFO ======================================= 2017-09-19 13:58:08,285 INFO Entered Test_enforce_on_exclhost setUp 2017-09-19 13:58:08,285 INFO ======================================= 2017-09-19 13:58:08,285 INFOCLI pbspro-ptl: /usr/bin/qstat -Bf pbspro-ptl 2017-09-19 13:58:08,289 INFO status on pbspro-ptl: server 2017-09-19 13:58:08,289 INFOCLI pbspro-ptl: /usr/bin/qstat -Bf pbspro-ptl 2017-09-19 13:58:08,294 INFO manager on pbspro-ptl: set server {'managers': (3, 'root@*')} 2017-09-19 13:58:08,294 INFOCLI pbspro-ptl: sudo -H /usr/bin/qmgr -c set server managers-=root@* 2017-09-19 13:58:08,304 INFO manager on pbspro-ptl: set server {'managers': (2, 'root@*')} 2017-09-19 13:58:08,305 INFOCLI pbspro-ptl: sudo -H /usr/bin/qmgr -c set server managers+=root@* 2017-09-19 13:58:08,334 INFO server pbspro-ptl: reverting configuration to defaults 2017-09-19 13:58:08,334 INFOCLI pbspro-ptl: /usr/bin/qstat -Bf pbspro-ptl 2017-09-19 13:58:08,338 INFO select on pbspro-ptl: __ALL__ 2017-09-19 13:58:08,338 INFOCLI pbspro-ptl: /usr/bin/qselect 2017-09-19 13:58:08,341 INFO delete job on pbspro-ptl: 4967.pbspro-ptl 2017-09-19 13:58:08,342 INFOCLI pbspro-ptl: /usr/bin/qdel -W force 4967.pbspro-ptl 2017-09-19 13:58:08,357 INFOCLI pbspro-ptl: /usr/bin/qstat -f 4967.pbspro-ptl 2017-09-19 13:58:08,374 INFOCLI pbspro-ptl: /usr/bin/qstat -f @pbspro-ptl 2017-09-19 13:58:08,378 INFO expect on server pbspro-ptl: job_state set 0 job ... OK 2017-09-19 13:58:08,378 INFOCLI pbspro-ptl: /usr/bin/pbs_rstat -f 2017-09-19 13:58:08,382 INFOCLI pbspro-ptl: sudo -H /usr/bin/qmgr -c list hook 2017-09-19 13:58:08,389 INFOCLI pbspro-ptl: /usr/bin/qstat -Qf @pbspro-ptl 2017-09-19 13:58:08,394 INFO status on pbspro-ptl: node 2017-09-19 13:58:08,394 INFOCLI pbspro-ptl: /usr/bin/pbsnodes -s pbspro-ptl -v -a 2017-09-19 13:58:08,402 INFO manager on pbspro-ptl: delete queue workq 2017-09-19 13:58:08,402 INFOCLI pbspro-ptl: /usr/bin/qmgr -c delete queue workq 2017-09-19 13:58:08,408 INFO server pbspro-ptl: expect offset set to 0.5 2017-09-19 13:58:08,909 INFOCLI pbspro-ptl: /usr/bin/qstat -Qf workq@pbspro-ptl 2017-09-19 13:58:08,914 INFO expect on server pbspro-ptl: unset queue workq ... OK 2017-09-19 13:58:08,914 INFO manager on pbspro-ptl: create queue workq {'started': 'True', 'queue_type': 'Execution', 'enabled': 'True'} 2017-09-19 13:58:08,914 INFOCLI pbspro-ptl: /usr/bin/qmgr -c create queue workq started=True,queue_type=Execution,enabled=True 2017-09-19 13:58:08,921 INFO server pbspro-ptl: expect offset set to 0.5 2017-09-19 13:58:09,423 INFOCLI pbspro-ptl: /usr/bin/qstat -Qf workq@pbspro-ptl 2017-09-19 13:58:09,428 INFO expect on server pbspro-ptl: started set True || queue_type set Execution || enabled set True queue workq ... OK 2017-09-19 13:58:09,428 INFO manager on pbspro-ptl: set server {'default_queue': 'workq'} 2017-09-19 13:58:09,428 INFOCLI pbspro-ptl: /usr/bin/qmgr -c set server default_queue=workq 2017-09-19 13:58:09,435 INFO status on pbspro-ptl: resource 2017-09-19 13:58:09,435 INFO manager on pbspro-ptl: list resource 2017-09-19 13:58:09,435 INFOCLI pbspro-ptl: /usr/bin/qmgr -c list resource 2017-09-19 13:58:09,441 INFOCLI status on pbspro-ptl: server license_count 2017-09-19 13:58:09,441 INFOCLI pbspro-ptl: /usr/bin/qstat -Bf pbspro-ptl 2017-09-19 13:58:09,445 INFO server: pbspro-ptl licensed 2017-09-19 13:58:09,450 INFOCLI2 pbspro-ptl: sudo -H /bin/cat /var/spool/pbs/server_priv/comm.lock 2017-09-19 13:58:09,463 INFOCLI2 pbspro-ptl: sudo -H /bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-09-19 13:58:09,468 INFO scheduler pbspro-ptl: reverting configuration to defaults 2017-09-19 13:58:09,468 INFO manager on pbspro-ptl: delete sched @default 2017-09-19 13:58:09,468 INFOCLI pbspro-ptl: /usr/bin/qmgr -c delete sched @default 2017-09-19 13:58:09,472 INFO manager on pbspro-ptl: list sched 2017-09-19 13:58:09,472 INFOCLI pbspro-ptl: /usr/bin/qmgr -c list sched 2017-09-19 13:58:09,476 INFO manager on pbspro-ptl: unset sched ['sched_cycle_length'] 2017-09-19 13:58:09,476 INFOCLI pbspro-ptl: /usr/bin/qmgr -c unset sched sched_cycle_length 2017-09-19 13:58:09,481 INFOCLI2 pbspro-ptl: sudo -H /bin/cat /var/spool/pbs/sched_priv/dedicated_time 2017-09-19 13:58:09,486 INFOCLI2 pbspro-ptl: cmp /usr/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-09-19 13:58:09,489 INFO scheduler pbspro-ptl: reverting holidays file to default 2017-09-19 13:58:09,489 INFOCLI2 pbspro-ptl: cmp /usr/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-09-19 13:58:09,492 INFOCLI2 pbspro-ptl: cmp /usr/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-09-19 13:58:09,494 INFO scheduler pbspro-ptl: sent signal -HUP 2017-09-19 13:58:09,494 INFOCLI2 pbspro-ptl: sudo -H kill -HUP 1752 2017-09-19 13:58:09,501 INFOCLI2 pbspro-ptl: sudo -H /usr/sbin/pbsfs -e 2017-09-19 13:58:09,517 INFOCLI2 pbspro-ptl: sudo -H /bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-09-19 13:58:09,530 INFOCLI2 pbspro-ptl: sudo -H /bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-09-19 13:58:09,535 INFOCLI2 pbspro-ptl: sudo -H /usr/sbin/pbs_mom --version 2017-09-19 13:58:09,544 INFO mom pbspro-ptl: reverting configuration to defaults 2017-09-19 13:58:09,544 INFOCLI2 pbspro-ptl: which rm 2017-09-19 13:58:09,546 INFOCLI2 pbspro-ptl: sudo -H /bin/rm -f /var/spool/pbs/mom_priv/epilogue 2017-09-19 13:58:09,550 INFOCLI2 pbspro-ptl: sudo -H /bin/rm -f /var/spool/pbs/mom_priv/prologue 2017-09-19 13:58:09,557 INFOCLI pbspro-ptl: sudo -H /usr/sbin/pbs_mom -s list 2017-09-19 13:58:09,571 INFOCLI2 pbspro-ptl: which cp 2017-09-19 13:58:09,573 INFOCLI2 pbspro-ptl: sudo -H /bin/cp -p /tmp/PtlPbsDQ4g5X /var/spool/pbs/mom_priv/config 2017-09-19 13:58:09,578 INFOCLI2 pbspro-ptl: which chmod 2017-09-19 13:58:09,580 INFOCLI2 pbspro-ptl: sudo -H /bin/chmod 0644 /var/spool/pbs/mom_priv/config 2017-09-19 13:58:09,585 INFOCLI2 pbspro-ptl: which chown 2017-09-19 13:58:09,587 INFOCLI2 pbspro-ptl: sudo -H /bin/chown root /var/spool/pbs/mom_priv/config 2017-09-19 13:58:09,595 INFOCLI2 pbspro-ptl: which chgrp 2017-09-19 13:58:09,596 INFOCLI2 pbspro-ptl: sudo -H /bin/chgrp root /var/spool/pbs/mom_priv/config 2017-09-19 13:58:09,602 INFO mom pbspro-ptl: sent signal -HUP 2017-09-19 13:58:09,602 INFOCLI2 pbspro-ptl: sudo -H kill -HUP 3780 2017-09-19 13:58:09,614 INFOCLI2 pbspro-ptl: sudo -H /bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-09-19 13:58:09,619 INFOCLI pbspro-ptl: /usr/bin/pbsnodes -s pbspro-ptl -v -a 2017-09-19 13:58:09,625 INFO status on pbspro-ptl: node pbspro-ptl 2017-09-19 13:58:09,625 INFOCLI pbspro-ptl: /usr/bin/pbsnodes -s pbspro-ptl -v pbspro-ptl 2017-09-19 13:58:09,631 INFOCLI pbspro-ptl: /usr/bin/pbsnodes -s pbspro-ptl -v pbspro-ptl 2017-09-19 13:58:09,634 INFO expect on server pbspro-ptl: state = free node pbspro-ptl ... OK 2017-09-19 13:58:09,635 INFO ======================================== 2017-09-19 13:58:09,635 INFO Completed Test_enforce_on_exclhost setUp 2017-09-19 13:58:09,635 INFO ======================================== 2017-09-19 13:58:09,635 INFO mom pbspro-ptl: config {'$enforce_on_exclhost': 'false'} 2017-09-19 13:58:09,636 INFOCLI2 pbspro-ptl: sudo -H /bin/cp -p /tmp/PtlPbsR05nfY /var/spool/pbs/mom_priv/config 2017-09-19 13:58:09,641 INFOCLI2 pbspro-ptl: sudo -H /bin/chmod 0644 /var/spool/pbs/mom_priv/config 2017-09-19 13:58:09,647 INFOCLI2 pbspro-ptl: sudo -H /bin/chown root /var/spool/pbs/mom_priv/config 2017-09-19 13:58:09,651 INFOCLI2 pbspro-ptl: sudo -H /bin/chgrp root /var/spool/pbs/mom_priv/config 2017-09-19 13:58:09,659 INFO mom pbspro-ptl: sent signal -HUP 2017-09-19 13:58:09,659 INFOCLI2 pbspro-ptl: sudo -H kill -HUP 3780 2017-09-19 13:58:09,672 INFOCLI2 pbspro-ptl: sudo -H /bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-09-19 13:58:09,679 INFO Set {'PBS_START_MOM': 1, 'PBS_START_SERVER': 0, 'PBS_START_SCHED': 0, 'PBS_START_COMM': 0} in /tmp/PtlPbsaBUVoI 2017-09-19 13:58:09,681 INFOCLI2 pbspro-ptl: sudo -H /bin/chown root /tmp/PtlPbsqgxiqL 2017-09-19 13:58:09,686 INFOCLI2 pbspro-ptl: sudo -H /bin/chgrp root /tmp/PtlPbsqgxiqL 2017-09-19 13:58:09,699 INFOCLI2 pbspro-ptl: sudo -H /bin/rm /tmp/PtlPbsqgxiqL 2017-09-19 13:58:09,703 INFOCLI2 pbspro-ptl: sudo -H ls -l /usr/libexec/pbs_init.d 2017-09-19 13:58:09,710 INFO running init script to stop pbs mom on pbspro-ptl using /etc/pbs.conf init_cmd=['sudo', 'PBS_CONF_FILE=/tmp/PtlPbsaBUVoI', '/usr/libexec/pbs_init.d', 'stop'] 2017-09-19 13:58:09,710 INFOCLI2 pbspro-ptl: /tmp/PtlPbssMsyiJ Contents of /tmp/PtlPbssMsyiJ: ---------------------------------------- #!/bin/bash sudo PBS_CONF_FILE=/tmp/PtlPbsaBUVoI /usr/libexec/pbs_init.d stop ---------------------------------------- 2017-09-19 13:58:10,739 INFOCLI2 pbspro-ptl: /bin/rm /tmp/PtlPbssMsyiJ 2017-09-19 13:58:10,742 INFO Set {'PBS_START_MOM': 1, 'PBS_START_SERVER': 0, 'PBS_START_SCHED': 0, 'PBS_START_COMM': 0} in /tmp/PtlPbsELOqen 2017-09-19 13:58:10,744 INFOCLI2 pbspro-ptl: sudo -H /bin/chown root /tmp/PtlPbsDxMITb 2017-09-19 13:58:10,748 INFOCLI2 pbspro-ptl: sudo -H /bin/chgrp root /tmp/PtlPbsDxMITb 2017-09-19 13:58:10,760 INFOCLI2 pbspro-ptl: sudo -H /bin/rm /tmp/PtlPbsDxMITb 2017-09-19 13:58:10,765 INFOCLI2 pbspro-ptl: sudo -H ls -l /usr/libexec/pbs_init.d 2017-09-19 13:58:10,772 INFO running init script to start pbs mom on pbspro-ptl using /etc/pbs.conf init_cmd=['sudo', 'PBS_CONF_FILE=/tmp/PtlPbsELOqen', '/usr/libexec/pbs_init.d', 'start'] 2017-09-19 13:58:10,773 INFOCLI2 pbspro-ptl: /tmp/PtlPbsK23sWA Contents of /tmp/PtlPbsK23sWA: ---------------------------------------- #!/bin/bash sudo PBS_CONF_FILE=/tmp/PtlPbsELOqen /usr/libexec/pbs_init.d start ---------------------------------------- 2017-09-19 13:58:10,824 INFOCLI2 pbspro-ptl: /bin/rm /tmp/PtlPbsK23sWA 2017-09-19 13:58:10,842 INFOCLI2 pbspro-ptl: sudo -H /bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-09-19 13:58:10,848 INFO mom pbspro-ptl: config {'$enforce average_cpufactor': '0.1', '$enforce average_percent_over': '1', '$enforce average_trialperiod': '1', '$enforce': 'cpuaverage'} 2017-09-19 13:58:10,849 INFOCLI2 pbspro-ptl: sudo -H /bin/cp -p /tmp/PtlPbsR5bi5l /var/spool/pbs/mom_priv/config 2017-09-19 13:58:10,854 INFOCLI2 pbspro-ptl: sudo -H /bin/chmod 0644 /var/spool/pbs/mom_priv/config 2017-09-19 13:58:10,859 INFOCLI2 pbspro-ptl: sudo -H /bin/chown root /var/spool/pbs/mom_priv/config 2017-09-19 13:58:10,864 INFOCLI2 pbspro-ptl: sudo -H /bin/chgrp root /var/spool/pbs/mom_priv/config 2017-09-19 13:58:10,870 INFO mom pbspro-ptl: sent signal -HUP 2017-09-19 13:58:10,870 INFOCLI2 pbspro-ptl: sudo -H kill -HUP 4196 2017-09-19 13:58:10,884 INFOCLI2 pbspro-ptl: sudo -H /bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-09-19 13:58:10,890 INFO Set {'PBS_START_MOM': 1, 'PBS_START_SERVER': 0, 'PBS_START_SCHED': 0, 'PBS_START_COMM': 0} in /tmp/PtlPbs_cM69V 2017-09-19 13:58:10,892 INFOCLI2 pbspro-ptl: sudo -H /bin/chown root /tmp/PtlPbsJkIwqS 2017-09-19 13:58:10,897 INFOCLI2 pbspro-ptl: sudo -H /bin/chgrp root /tmp/PtlPbsJkIwqS 2017-09-19 13:58:10,909 INFOCLI2 pbspro-ptl: sudo -H /bin/rm /tmp/PtlPbsJkIwqS 2017-09-19 13:58:10,915 INFOCLI2 pbspro-ptl: sudo -H ls -l /usr/libexec/pbs_init.d 2017-09-19 13:58:10,920 INFO running init script to stop pbs mom on pbspro-ptl using /etc/pbs.conf init_cmd=['sudo', 'PBS_CONF_FILE=/tmp/PtlPbs_cM69V', '/usr/libexec/pbs_init.d', 'stop'] 2017-09-19 13:58:10,921 INFOCLI2 pbspro-ptl: /tmp/PtlPbsFtN9Hl Contents of /tmp/PtlPbsFtN9Hl: ---------------------------------------- #!/bin/bash sudo PBS_CONF_FILE=/tmp/PtlPbs_cM69V /usr/libexec/pbs_init.d stop ---------------------------------------- 2017-09-19 13:58:11,956 INFOCLI2 pbspro-ptl: /bin/rm /tmp/PtlPbsFtN9Hl 2017-09-19 13:58:11,957 INFO Set {'PBS_START_MOM': 1, 'PBS_START_SERVER': 0, 'PBS_START_SCHED': 0, 'PBS_START_COMM': 0} in /tmp/PtlPbsBvoIKD 2017-09-19 13:58:11,962 INFOCLI2 pbspro-ptl: sudo -H /bin/chown root /tmp/PtlPbs6HgfKh 2017-09-19 13:58:11,968 INFOCLI2 pbspro-ptl: sudo -H /bin/chgrp root /tmp/PtlPbs6HgfKh 2017-09-19 13:58:11,981 INFOCLI2 pbspro-ptl: sudo -H /bin/rm /tmp/PtlPbs6HgfKh 2017-09-19 13:58:11,986 INFOCLI2 pbspro-ptl: sudo -H ls -l /usr/libexec/pbs_init.d 2017-09-19 13:58:11,993 INFO running init script to start pbs mom on pbspro-ptl using /etc/pbs.conf init_cmd=['sudo', 'PBS_CONF_FILE=/tmp/PtlPbsBvoIKD', '/usr/libexec/pbs_init.d', 'start'] 2017-09-19 13:58:11,994 INFOCLI2 pbspro-ptl: /tmp/PtlPbsVnVZGN Contents of /tmp/PtlPbsVnVZGN: ---------------------------------------- #!/bin/bash sudo PBS_CONF_FILE=/tmp/PtlPbsBvoIKD /usr/libexec/pbs_init.d start ---------------------------------------- 2017-09-19 13:58:12,045 INFOCLI2 pbspro-ptl: /bin/rm /tmp/PtlPbsVnVZGN 2017-09-19 13:58:12,060 INFOCLI2 pbspro-ptl: sudo -H /bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-09-19 13:58:12,067 INFO job: executable set to /bin/sleep with arguments: 100 2017-09-19 13:58:12,067 INFOCLI2 pbspro-ptl: sudo -H /bin/chmod 0755 /tmp/PtlPbsJobScriptzWD8JX 2017-09-19 13:58:12,072 INFOCLI pbspro-ptl: sudo -H -u pbsuser /usr/bin/qsub -l place=exclhost -l select=1:ncpus=1 /tmp/PtlPbsJobScriptzWD8JX 2017-09-19 13:58:12,084 INFO submit to pbspro-ptl as pbsuser: job 4968.pbspro-ptl OrderedDict([('Resource_List.place', 'exclhost'), ('Resource_List.select', '1:ncpus=1')]) 2017-09-19 13:58:12,084 INFOCLI job script /tmp/PtlPbsJobScriptzWD8JX --- #!/bin/bash yes > /dev/null --- 2017-09-19 13:58:12,084 INFOCLI pbspro-ptl: /usr/bin/qstat -f 4968.pbspro-ptl 2017-09-19 13:58:12,090 INFO expect on server pbspro-ptl: job_state = R && substate = 42 job 4968.pbspro-ptl got: job_state = Q 2017-09-19 13:58:12,591 INFOCLI pbspro-ptl: /usr/bin/qmgr -c set server scheduling=True 2017-09-19 13:58:12,606 INFOCLI pbspro-ptl: /usr/bin/qstat -f 4968.pbspro-ptl 2017-09-19 13:58:12,623 INFO expect on server pbspro-ptl: job_state = R && substate = 42 job 4968.pbspro-ptl attempt: 2 ... OK 2017-09-19 13:58:12,624 INFO Giving a chance to pbs to kill the job... 2017-09-19 13:58:27,641 INFOCLI pbspro-ptl: /usr/bin/qstat -f 4968.pbspro-ptl 2017-09-19 13:58:27,665 INFO expect on server pbspro-ptl: job_state = R && substate = 42 job 4968.pbspro-ptl ... OK 2017-09-19 13:58:27,665 INFO ========================================== 2017-09-19 13:58:27,665 INFO Entered Test_enforce_on_exclhost tearDown 2017-09-19 13:58:27,665 INFO ========================================== 2017-09-19 13:58:27,665 INFO =========================================== 2017-09-19 13:58:27,665 INFO Completed Test_enforce_on_exclhost tearDown 2017-09-19 13:58:27,665 INFO =========================================== 2017-09-19 13:58:27,665 INFO ok 2017-09-19 13:58:27,666 INFO test name: test_job_not_killed_cpuburst (tests.functional.pbs_enforce_on_exclhost.Test_enforce_on_exclhost)... 2017-09-19 13:58:27,666 INFO test start time: Tue Sep 19 13:58:27 2017 2017-09-19 13:58:27,666 INFO test docstring: Set cpuburst enforcement and run a job with exclhost Check wheter or not the job will survive 2017-09-19 13:58:27,666 INFO ======================================= 2017-09-19 13:58:27,666 INFO Entered Test_enforce_on_exclhost setUp 2017-09-19 13:58:27,666 INFO ======================================= 2017-09-19 13:58:27,666 INFOCLI pbspro-ptl: /usr/bin/qstat -Bf pbspro-ptl 2017-09-19 13:58:27,682 INFO status on pbspro-ptl: server 2017-09-19 13:58:27,683 INFOCLI pbspro-ptl: /usr/bin/qstat -Bf pbspro-ptl 2017-09-19 13:58:27,692 INFO manager on pbspro-ptl: set server {'managers': (3, 'root@*')} 2017-09-19 13:58:27,693 INFOCLI pbspro-ptl: sudo -H /usr/bin/qmgr -c set server managers-=root@* 2017-09-19 13:58:27,733 INFO manager on pbspro-ptl: set server {'managers': (2, 'root@*')} 2017-09-19 13:58:27,733 INFOCLI pbspro-ptl: sudo -H /usr/bin/qmgr -c set server managers+=root@* 2017-09-19 13:58:27,764 INFO server pbspro-ptl: reverting configuration to defaults 2017-09-19 13:58:27,764 INFOCLI pbspro-ptl: /usr/bin/qstat -Bf pbspro-ptl 2017-09-19 13:58:27,782 INFO select on pbspro-ptl: __ALL__ 2017-09-19 13:58:27,782 INFOCLI pbspro-ptl: /usr/bin/qselect 2017-09-19 13:58:27,792 INFO delete job on pbspro-ptl: 4968.pbspro-ptl 2017-09-19 13:58:27,792 INFOCLI pbspro-ptl: /usr/bin/qdel -W force 4968.pbspro-ptl 2017-09-19 13:58:27,810 INFOCLI pbspro-ptl: /usr/bin/qstat -f 4968.pbspro-ptl 2017-09-19 13:58:27,826 INFOCLI pbspro-ptl: /usr/bin/qstat -f @pbspro-ptl 2017-09-19 13:58:27,831 INFO expect on server pbspro-ptl: job_state set 0 job ... OK 2017-09-19 13:58:27,831 INFOCLI pbspro-ptl: /usr/bin/pbs_rstat -f 2017-09-19 13:58:27,835 INFOCLI pbspro-ptl: sudo -H /usr/bin/qmgr -c list hook 2017-09-19 13:58:27,843 INFOCLI pbspro-ptl: /usr/bin/qstat -Qf @pbspro-ptl 2017-09-19 13:58:27,847 INFO status on pbspro-ptl: node 2017-09-19 13:58:27,847 INFOCLI pbspro-ptl: /usr/bin/pbsnodes -s pbspro-ptl -v -a 2017-09-19 13:58:27,851 INFO manager on pbspro-ptl: delete queue workq 2017-09-19 13:58:27,851 INFOCLI pbspro-ptl: /usr/bin/qmgr -c delete queue workq 2017-09-19 13:58:27,858 INFO server pbspro-ptl: expect offset set to 0.5 2017-09-19 13:58:28,359 INFOCLI pbspro-ptl: /usr/bin/qstat -Qf workq@pbspro-ptl 2017-09-19 13:58:28,363 INFO expect on server pbspro-ptl: unset queue workq ... OK 2017-09-19 13:58:28,363 INFO manager on pbspro-ptl: create queue workq {'started': 'True', 'queue_type': 'Execution', 'enabled': 'True'} 2017-09-19 13:58:28,363 INFOCLI pbspro-ptl: /usr/bin/qmgr -c create queue workq started=True,queue_type=Execution,enabled=True 2017-09-19 13:58:28,374 INFO server pbspro-ptl: expect offset set to 0.5 2017-09-19 13:58:28,875 INFOCLI pbspro-ptl: /usr/bin/qstat -Qf workq@pbspro-ptl 2017-09-19 13:58:28,879 INFO expect on server pbspro-ptl: started set True || queue_type set Execution || enabled set True queue workq ... OK 2017-09-19 13:58:28,879 INFO manager on pbspro-ptl: set server {'default_queue': 'workq'} 2017-09-19 13:58:28,879 INFOCLI pbspro-ptl: /usr/bin/qmgr -c set server default_queue=workq 2017-09-19 13:58:28,885 INFO status on pbspro-ptl: resource 2017-09-19 13:58:28,885 INFO manager on pbspro-ptl: list resource 2017-09-19 13:58:28,885 INFOCLI pbspro-ptl: /usr/bin/qmgr -c list resource 2017-09-19 13:58:28,893 INFOCLI status on pbspro-ptl: server license_count 2017-09-19 13:58:28,893 INFOCLI pbspro-ptl: /usr/bin/qstat -Bf pbspro-ptl 2017-09-19 13:58:28,897 INFO server: pbspro-ptl licensed 2017-09-19 13:58:28,906 INFOCLI2 pbspro-ptl: sudo -H /bin/cat /var/spool/pbs/server_priv/comm.lock 2017-09-19 13:58:28,918 INFOCLI2 pbspro-ptl: sudo -H /bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-09-19 13:58:28,924 INFO scheduler pbspro-ptl: reverting configuration to defaults 2017-09-19 13:58:28,925 INFO manager on pbspro-ptl: delete sched @default 2017-09-19 13:58:28,925 INFOCLI pbspro-ptl: /usr/bin/qmgr -c delete sched @default 2017-09-19 13:58:28,929 INFO manager on pbspro-ptl: list sched 2017-09-19 13:58:28,929 INFOCLI pbspro-ptl: /usr/bin/qmgr -c list sched 2017-09-19 13:58:28,933 INFO manager on pbspro-ptl: unset sched ['sched_cycle_length'] 2017-09-19 13:58:28,933 INFOCLI pbspro-ptl: /usr/bin/qmgr -c unset sched sched_cycle_length 2017-09-19 13:58:28,940 INFOCLI2 pbspro-ptl: sudo -H /bin/cat /var/spool/pbs/sched_priv/dedicated_time 2017-09-19 13:58:28,946 INFOCLI2 pbspro-ptl: cmp /usr/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-09-19 13:58:28,948 INFO scheduler pbspro-ptl: reverting holidays file to default 2017-09-19 13:58:28,948 INFOCLI2 pbspro-ptl: cmp /usr/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-09-19 13:58:28,950 INFOCLI2 pbspro-ptl: cmp /usr/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-09-19 13:58:28,952 INFO scheduler pbspro-ptl: sent signal -HUP 2017-09-19 13:58:28,952 INFOCLI2 pbspro-ptl: sudo -H kill -HUP 1752 2017-09-19 13:58:28,959 INFOCLI2 pbspro-ptl: sudo -H /usr/sbin/pbsfs -e 2017-09-19 13:58:28,981 INFOCLI2 pbspro-ptl: sudo -H /bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-09-19 13:58:28,993 INFOCLI2 pbspro-ptl: sudo -H /bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-09-19 13:58:29,000 INFO mom pbspro-ptl: reverting configuration to defaults 2017-09-19 13:58:29,000 INFOCLI2 pbspro-ptl: sudo -H /bin/rm -f /var/spool/pbs/mom_priv/epilogue 2017-09-19 13:58:29,007 INFOCLI2 pbspro-ptl: sudo -H /bin/rm -f /var/spool/pbs/mom_priv/prologue 2017-09-19 13:58:29,013 INFOCLI pbspro-ptl: sudo -H /usr/sbin/pbs_mom -s list 2017-09-19 13:58:29,028 INFOCLI2 pbspro-ptl: sudo -H /bin/cp -p /tmp/PtlPbstRdciy /var/spool/pbs/mom_priv/config 2017-09-19 13:58:29,035 INFOCLI2 pbspro-ptl: sudo -H /bin/chmod 0644 /var/spool/pbs/mom_priv/config 2017-09-19 13:58:29,043 INFOCLI2 pbspro-ptl: sudo -H /bin/chown root /var/spool/pbs/mom_priv/config 2017-09-19 13:58:29,048 INFOCLI2 pbspro-ptl: sudo -H /bin/chgrp root /var/spool/pbs/mom_priv/config 2017-09-19 13:58:29,054 INFO mom pbspro-ptl: sent signal -HUP 2017-09-19 13:58:29,054 INFOCLI2 pbspro-ptl: sudo -H kill -HUP 4365 2017-09-19 13:58:29,068 INFOCLI2 pbspro-ptl: sudo -H /bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-09-19 13:58:29,075 INFOCLI pbspro-ptl: /usr/bin/pbsnodes -s pbspro-ptl -v -a 2017-09-19 13:58:29,081 INFO status on pbspro-ptl: node pbspro-ptl 2017-09-19 13:58:29,081 INFOCLI pbspro-ptl: /usr/bin/pbsnodes -s pbspro-ptl -v pbspro-ptl 2017-09-19 13:58:29,085 INFOCLI pbspro-ptl: /usr/bin/pbsnodes -s pbspro-ptl -v pbspro-ptl 2017-09-19 13:58:29,090 INFO expect on server pbspro-ptl: state = free node pbspro-ptl ... OK 2017-09-19 13:58:29,090 INFO ======================================== 2017-09-19 13:58:29,090 INFO Completed Test_enforce_on_exclhost setUp 2017-09-19 13:58:29,090 INFO ======================================== 2017-09-19 13:58:29,097 INFOCLI2 pbspro-ptl: sudo -H /bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-09-19 13:58:29,104 INFO Set {'PBS_START_MOM': 1, 'PBS_START_SERVER': 0, 'PBS_START_SCHED': 0, 'PBS_START_COMM': 0} in /tmp/PtlPbsFtvDE1 2017-09-19 13:58:29,106 INFOCLI2 pbspro-ptl: sudo -H /bin/chown root /tmp/PtlPbs_VxX0p 2017-09-19 13:58:29,113 INFOCLI2 pbspro-ptl: sudo -H /bin/chgrp root /tmp/PtlPbs_VxX0p 2017-09-19 13:58:29,126 INFOCLI2 pbspro-ptl: sudo -H /bin/rm /tmp/PtlPbs_VxX0p 2017-09-19 13:58:29,132 INFOCLI2 pbspro-ptl: sudo -H ls -l /usr/libexec/pbs_init.d 2017-09-19 13:58:29,139 INFO running init script to stop pbs mom on pbspro-ptl using /etc/pbs.conf init_cmd=['sudo', 'PBS_CONF_FILE=/tmp/PtlPbsFtvDE1', '/usr/libexec/pbs_init.d', 'stop'] 2017-09-19 13:58:29,139 INFOCLI2 pbspro-ptl: /tmp/PtlPbs8hogll Contents of /tmp/PtlPbs8hogll: ---------------------------------------- #!/bin/bash sudo PBS_CONF_FILE=/tmp/PtlPbsFtvDE1 /usr/libexec/pbs_init.d stop ---------------------------------------- 2017-09-19 13:58:30,171 INFOCLI2 pbspro-ptl: /bin/rm /tmp/PtlPbs8hogll 2017-09-19 13:58:30,174 INFO Set {'PBS_START_MOM': 1, 'PBS_START_SERVER': 0, 'PBS_START_SCHED': 0, 'PBS_START_COMM': 0} in /tmp/PtlPbsMS6trd 2017-09-19 13:58:30,176 INFOCLI2 pbspro-ptl: sudo -H /bin/chown root /tmp/PtlPbsRWsLEW 2017-09-19 13:58:30,181 INFOCLI2 pbspro-ptl: sudo -H /bin/chgrp root /tmp/PtlPbsRWsLEW 2017-09-19 13:58:30,193 INFOCLI2 pbspro-ptl: sudo -H /bin/rm /tmp/PtlPbsRWsLEW 2017-09-19 13:58:30,198 INFOCLI2 pbspro-ptl: sudo -H ls -l /usr/libexec/pbs_init.d 2017-09-19 13:58:30,206 INFO running init script to start pbs mom on pbspro-ptl using /etc/pbs.conf init_cmd=['sudo', 'PBS_CONF_FILE=/tmp/PtlPbsMS6trd', '/usr/libexec/pbs_init.d', 'start'] 2017-09-19 13:58:30,206 INFOCLI2 pbspro-ptl: /tmp/PtlPbsZzIX4U Contents of /tmp/PtlPbsZzIX4U: ---------------------------------------- #!/bin/bash sudo PBS_CONF_FILE=/tmp/PtlPbsMS6trd /usr/libexec/pbs_init.d start ---------------------------------------- 2017-09-19 13:58:30,264 INFOCLI2 pbspro-ptl: /bin/rm /tmp/PtlPbsZzIX4U 2017-09-19 13:58:30,279 INFOCLI2 pbspro-ptl: sudo -H /bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-09-19 13:58:30,285 INFO mom pbspro-ptl: config {'$enforce delta_percent_over': '1', '$enforce delta_cpufactor': '0.1', '$enforce': 'cpuburst'} 2017-09-19 13:58:30,286 INFOCLI2 pbspro-ptl: sudo -H /bin/cp -p /tmp/PtlPbsGCJmNP /var/spool/pbs/mom_priv/config 2017-09-19 13:58:30,294 INFOCLI2 pbspro-ptl: sudo -H /bin/chmod 0644 /var/spool/pbs/mom_priv/config 2017-09-19 13:58:30,300 INFOCLI2 pbspro-ptl: sudo -H /bin/chown root /var/spool/pbs/mom_priv/config 2017-09-19 13:58:30,307 INFOCLI2 pbspro-ptl: sudo -H /bin/chgrp root /var/spool/pbs/mom_priv/config 2017-09-19 13:58:30,313 INFO mom pbspro-ptl: sent signal -HUP 2017-09-19 13:58:30,314 INFOCLI2 pbspro-ptl: sudo -H kill -HUP 4683 2017-09-19 13:58:30,327 INFOCLI2 pbspro-ptl: sudo -H /bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-09-19 13:58:30,334 INFO Set {'PBS_START_MOM': 1, 'PBS_START_SERVER': 0, 'PBS_START_SCHED': 0, 'PBS_START_COMM': 0} in /tmp/PtlPbsKiIjIR 2017-09-19 13:58:30,336 INFOCLI2 pbspro-ptl: sudo -H /bin/chown root /tmp/PtlPbsYuWzXr 2017-09-19 13:58:30,344 INFOCLI2 pbspro-ptl: sudo -H /bin/chgrp root /tmp/PtlPbsYuWzXr 2017-09-19 13:58:30,357 INFOCLI2 pbspro-ptl: sudo -H /bin/rm /tmp/PtlPbsYuWzXr 2017-09-19 13:58:30,364 INFOCLI2 pbspro-ptl: sudo -H ls -l /usr/libexec/pbs_init.d 2017-09-19 13:58:30,370 INFO running init script to stop pbs mom on pbspro-ptl using /etc/pbs.conf init_cmd=['sudo', 'PBS_CONF_FILE=/tmp/PtlPbsKiIjIR', '/usr/libexec/pbs_init.d', 'stop'] 2017-09-19 13:58:30,371 INFOCLI2 pbspro-ptl: /tmp/PtlPbs5fC2oG Contents of /tmp/PtlPbs5fC2oG: ---------------------------------------- #!/bin/bash sudo PBS_CONF_FILE=/tmp/PtlPbsKiIjIR /usr/libexec/pbs_init.d stop ---------------------------------------- 2017-09-19 13:58:31,402 INFOCLI2 pbspro-ptl: /bin/rm /tmp/PtlPbs5fC2oG 2017-09-19 13:58:31,404 INFO Set {'PBS_START_MOM': 1, 'PBS_START_SERVER': 0, 'PBS_START_SCHED': 0, 'PBS_START_COMM': 0} in /tmp/PtlPbsYAUAlc 2017-09-19 13:58:31,406 INFOCLI2 pbspro-ptl: sudo -H /bin/chown root /tmp/PtlPbszbkjQ7 2017-09-19 13:58:31,412 INFOCLI2 pbspro-ptl: sudo -H /bin/chgrp root /tmp/PtlPbszbkjQ7 2017-09-19 13:58:31,423 INFOCLI2 pbspro-ptl: sudo -H /bin/rm /tmp/PtlPbszbkjQ7 2017-09-19 13:58:31,431 INFOCLI2 pbspro-ptl: sudo -H ls -l /usr/libexec/pbs_init.d 2017-09-19 13:58:31,438 INFO running init script to start pbs mom on pbspro-ptl using /etc/pbs.conf init_cmd=['sudo', 'PBS_CONF_FILE=/tmp/PtlPbsYAUAlc', '/usr/libexec/pbs_init.d', 'start'] 2017-09-19 13:58:31,439 INFOCLI2 pbspro-ptl: /tmp/PtlPbsVDpm1d Contents of /tmp/PtlPbsVDpm1d: ---------------------------------------- #!/bin/bash sudo PBS_CONF_FILE=/tmp/PtlPbsYAUAlc /usr/libexec/pbs_init.d start ---------------------------------------- 2017-09-19 13:58:31,496 INFOCLI2 pbspro-ptl: /bin/rm /tmp/PtlPbsVDpm1d 2017-09-19 13:58:31,509 INFOCLI2 pbspro-ptl: sudo -H /bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-09-19 13:58:31,518 INFO job: executable set to /bin/sleep with arguments: 100 2017-09-19 13:58:31,518 INFOCLI2 pbspro-ptl: sudo -H /bin/chmod 0755 /tmp/PtlPbsJobScript7DP_y3 2017-09-19 13:58:31,525 INFOCLI pbspro-ptl: sudo -H -u pbsuser /usr/bin/qsub -l place=exclhost -l select=1:ncpus=1 /tmp/PtlPbsJobScript7DP_y3 2017-09-19 13:58:31,539 INFO submit to pbspro-ptl as pbsuser: job 4969.pbspro-ptl OrderedDict([('Resource_List.place', 'exclhost'), ('Resource_List.select', '1:ncpus=1')]) 2017-09-19 13:58:31,540 INFOCLI job script /tmp/PtlPbsJobScript7DP_y3 --- #!/bin/bash yes > /dev/null --- 2017-09-19 13:58:31,540 INFOCLI pbspro-ptl: /usr/bin/qstat -f 4969.pbspro-ptl 2017-09-19 13:58:31,545 INFO expect on server pbspro-ptl: job_state = R && substate = 42 job 4969.pbspro-ptl got: job_state = Q 2017-09-19 13:58:32,047 INFOCLI pbspro-ptl: /usr/bin/qmgr -c set server scheduling=True 2017-09-19 13:58:32,066 INFOCLI pbspro-ptl: /usr/bin/qstat -f 4969.pbspro-ptl 2017-09-19 13:58:32,081 INFO expect on server pbspro-ptl: job_state = R && substate = 42 job 4969.pbspro-ptl attempt: 2 ... OK 2017-09-19 13:58:32,081 INFO Giving a chance to pbs to kill the job... 2017-09-19 13:58:47,101 INFOCLI pbspro-ptl: /usr/bin/qstat -f 4969.pbspro-ptl 2017-09-19 13:58:47,115 INFO expect on server pbspro-ptl: job_state = R && substate = 42 job 4969.pbspro-ptl ... OK 2017-09-19 13:58:47,115 INFO ========================================== 2017-09-19 13:58:47,115 INFO Entered Test_enforce_on_exclhost tearDown 2017-09-19 13:58:47,115 INFO ========================================== 2017-09-19 13:58:47,115 INFO =========================================== 2017-09-19 13:58:47,115 INFO Completed Test_enforce_on_exclhost tearDown 2017-09-19 13:58:47,115 INFO =========================================== 2017-09-19 13:58:47,115 INFO ok 2017-09-19 13:58:47,115 INFO test name: test_job_not_killed_mem (tests.functional.pbs_enforce_on_exclhost.Test_enforce_on_exclhost)... 2017-09-19 13:58:47,116 INFO test start time: Tue Sep 19 13:58:47 2017 2017-09-19 13:58:47,116 INFO test docstring: Set mem enforcement and run a job with exclhost Check wheter or not the job will survive 2017-09-19 13:58:47,116 INFO ======================================= 2017-09-19 13:58:47,116 INFO Entered Test_enforce_on_exclhost setUp 2017-09-19 13:58:47,116 INFO ======================================= 2017-09-19 13:58:47,116 INFOCLI pbspro-ptl: /usr/bin/qstat -Bf pbspro-ptl 2017-09-19 13:58:47,129 INFO status on pbspro-ptl: server 2017-09-19 13:58:47,129 INFOCLI pbspro-ptl: /usr/bin/qstat -Bf pbspro-ptl 2017-09-19 13:58:47,143 INFO manager on pbspro-ptl: set server {'managers': (3, 'root@*')} 2017-09-19 13:58:47,143 INFOCLI pbspro-ptl: sudo -H /usr/bin/qmgr -c set server managers-=root@* 2017-09-19 13:58:47,182 INFO manager on pbspro-ptl: set server {'managers': (2, 'root@*')} 2017-09-19 13:58:47,183 INFOCLI pbspro-ptl: sudo -H /usr/bin/qmgr -c set server managers+=root@* 2017-09-19 13:58:47,206 INFO server pbspro-ptl: reverting configuration to defaults 2017-09-19 13:58:47,206 INFOCLI pbspro-ptl: /usr/bin/qstat -Bf pbspro-ptl 2017-09-19 13:58:47,223 INFO select on pbspro-ptl: __ALL__ 2017-09-19 13:58:47,223 INFOCLI pbspro-ptl: /usr/bin/qselect 2017-09-19 13:58:47,242 INFO delete job on pbspro-ptl: 4969.pbspro-ptl 2017-09-19 13:58:47,243 INFOCLI pbspro-ptl: /usr/bin/qdel -W force 4969.pbspro-ptl 2017-09-19 13:58:47,255 INFOCLI pbspro-ptl: /usr/bin/qstat -f 4969.pbspro-ptl 2017-09-19 13:58:47,273 INFOCLI pbspro-ptl: /usr/bin/qstat -f @pbspro-ptl 2017-09-19 13:58:47,277 INFO expect on server pbspro-ptl: job_state set 0 job ... OK 2017-09-19 13:58:47,278 INFOCLI pbspro-ptl: /usr/bin/pbs_rstat -f 2017-09-19 13:58:47,281 INFOCLI pbspro-ptl: sudo -H /usr/bin/qmgr -c list hook 2017-09-19 13:58:47,291 INFOCLI pbspro-ptl: /usr/bin/qstat -Qf @pbspro-ptl 2017-09-19 13:58:47,295 INFO status on pbspro-ptl: node 2017-09-19 13:58:47,296 INFOCLI pbspro-ptl: /usr/bin/pbsnodes -s pbspro-ptl -v -a 2017-09-19 13:58:47,300 INFO manager on pbspro-ptl: delete queue workq 2017-09-19 13:58:47,301 INFOCLI pbspro-ptl: /usr/bin/qmgr -c delete queue workq 2017-09-19 13:58:47,308 INFO server pbspro-ptl: expect offset set to 0.5 2017-09-19 13:58:47,809 INFOCLI pbspro-ptl: /usr/bin/qstat -Qf workq@pbspro-ptl 2017-09-19 13:58:47,812 INFO expect on server pbspro-ptl: unset queue workq ... OK 2017-09-19 13:58:47,813 INFO manager on pbspro-ptl: create queue workq {'started': 'True', 'queue_type': 'Execution', 'enabled': 'True'} 2017-09-19 13:58:47,813 INFOCLI pbspro-ptl: /usr/bin/qmgr -c create queue workq started=True,queue_type=Execution,enabled=True 2017-09-19 13:58:47,819 INFO server pbspro-ptl: expect offset set to 0.5 2017-09-19 13:58:48,320 INFOCLI pbspro-ptl: /usr/bin/qstat -Qf workq@pbspro-ptl 2017-09-19 13:58:48,324 INFO expect on server pbspro-ptl: started set True || queue_type set Execution || enabled set True queue workq ... OK 2017-09-19 13:58:48,324 INFO manager on pbspro-ptl: set server {'default_queue': 'workq'} 2017-09-19 13:58:48,324 INFOCLI pbspro-ptl: /usr/bin/qmgr -c set server default_queue=workq 2017-09-19 13:58:48,329 INFO status on pbspro-ptl: resource 2017-09-19 13:58:48,329 INFO manager on pbspro-ptl: list resource 2017-09-19 13:58:48,329 INFOCLI pbspro-ptl: /usr/bin/qmgr -c list resource 2017-09-19 13:58:48,335 INFOCLI status on pbspro-ptl: server license_count 2017-09-19 13:58:48,335 INFOCLI pbspro-ptl: /usr/bin/qstat -Bf pbspro-ptl 2017-09-19 13:58:48,339 INFO server: pbspro-ptl licensed 2017-09-19 13:58:48,345 INFOCLI2 pbspro-ptl: sudo -H /bin/cat /var/spool/pbs/server_priv/comm.lock 2017-09-19 13:58:48,358 INFOCLI2 pbspro-ptl: sudo -H /bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-09-19 13:58:48,363 INFO scheduler pbspro-ptl: reverting configuration to defaults 2017-09-19 13:58:48,363 INFO manager on pbspro-ptl: delete sched @default 2017-09-19 13:58:48,363 INFOCLI pbspro-ptl: /usr/bin/qmgr -c delete sched @default 2017-09-19 13:58:48,367 INFO manager on pbspro-ptl: list sched 2017-09-19 13:58:48,367 INFOCLI pbspro-ptl: /usr/bin/qmgr -c list sched 2017-09-19 13:58:48,372 INFO manager on pbspro-ptl: unset sched ['sched_cycle_length'] 2017-09-19 13:58:48,373 INFOCLI pbspro-ptl: /usr/bin/qmgr -c unset sched sched_cycle_length 2017-09-19 13:58:48,378 INFOCLI2 pbspro-ptl: sudo -H /bin/cat /var/spool/pbs/sched_priv/dedicated_time 2017-09-19 13:58:48,384 INFOCLI2 pbspro-ptl: cmp /usr/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-09-19 13:58:48,387 INFO scheduler pbspro-ptl: reverting holidays file to default 2017-09-19 13:58:48,387 INFOCLI2 pbspro-ptl: cmp /usr/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-09-19 13:58:48,390 INFOCLI2 pbspro-ptl: cmp /usr/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-09-19 13:58:48,391 INFO scheduler pbspro-ptl: sent signal -HUP 2017-09-19 13:58:48,391 INFOCLI2 pbspro-ptl: sudo -H kill -HUP 1752 2017-09-19 13:58:48,396 INFOCLI2 pbspro-ptl: sudo -H /usr/sbin/pbsfs -e 2017-09-19 13:58:48,416 INFOCLI2 pbspro-ptl: sudo -H /bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-09-19 13:58:48,428 INFOCLI2 pbspro-ptl: sudo -H /bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-09-19 13:58:48,433 INFO mom pbspro-ptl: reverting configuration to defaults 2017-09-19 13:58:48,433 INFOCLI2 pbspro-ptl: sudo -H /bin/rm -f /var/spool/pbs/mom_priv/epilogue 2017-09-19 13:58:48,439 INFOCLI2 pbspro-ptl: sudo -H /bin/rm -f /var/spool/pbs/mom_priv/prologue 2017-09-19 13:58:48,445 INFOCLI pbspro-ptl: sudo -H /usr/sbin/pbs_mom -s list 2017-09-19 13:58:48,459 INFOCLI2 pbspro-ptl: sudo -H /bin/cp -p /tmp/PtlPbsKrS8Lq /var/spool/pbs/mom_priv/config 2017-09-19 13:58:48,466 INFOCLI2 pbspro-ptl: sudo -H /bin/chmod 0644 /var/spool/pbs/mom_priv/config 2017-09-19 13:58:48,472 INFOCLI2 pbspro-ptl: sudo -H /bin/chown root /var/spool/pbs/mom_priv/config 2017-09-19 13:58:48,477 INFOCLI2 pbspro-ptl: sudo -H /bin/chgrp root /var/spool/pbs/mom_priv/config 2017-09-19 13:58:48,484 INFO mom pbspro-ptl: sent signal -HUP 2017-09-19 13:58:48,484 INFOCLI2 pbspro-ptl: sudo -H kill -HUP 4852 2017-09-19 13:58:48,496 INFOCLI2 pbspro-ptl: sudo -H /bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-09-19 13:58:48,503 INFOCLI pbspro-ptl: /usr/bin/pbsnodes -s pbspro-ptl -v -a 2017-09-19 13:58:48,508 INFO status on pbspro-ptl: node pbspro-ptl 2017-09-19 13:58:48,508 INFOCLI pbspro-ptl: /usr/bin/pbsnodes -s pbspro-ptl -v pbspro-ptl 2017-09-19 13:58:48,512 INFOCLI pbspro-ptl: /usr/bin/pbsnodes -s pbspro-ptl -v pbspro-ptl 2017-09-19 13:58:48,516 INFO expect on server pbspro-ptl: state = free node pbspro-ptl ... OK 2017-09-19 13:58:48,517 INFO ======================================== 2017-09-19 13:58:48,517 INFO Completed Test_enforce_on_exclhost setUp 2017-09-19 13:58:48,517 INFO ======================================== 2017-09-19 13:58:48,523 INFOCLI2 pbspro-ptl: sudo -H /bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-09-19 13:58:48,531 INFO Set {'PBS_START_MOM': 1, 'PBS_START_SERVER': 0, 'PBS_START_SCHED': 0, 'PBS_START_COMM': 0} in /tmp/PtlPbsvqiaZk 2017-09-19 13:58:48,533 INFOCLI2 pbspro-ptl: sudo -H /bin/chown root /tmp/PtlPbsNx6867 2017-09-19 13:58:48,540 INFOCLI2 pbspro-ptl: sudo -H /bin/chgrp root /tmp/PtlPbsNx6867 2017-09-19 13:58:48,551 INFOCLI2 pbspro-ptl: sudo -H /bin/rm /tmp/PtlPbsNx6867 2017-09-19 13:58:48,558 INFOCLI2 pbspro-ptl: sudo -H ls -l /usr/libexec/pbs_init.d 2017-09-19 13:58:48,563 INFO running init script to stop pbs mom on pbspro-ptl using /etc/pbs.conf init_cmd=['sudo', 'PBS_CONF_FILE=/tmp/PtlPbsvqiaZk', '/usr/libexec/pbs_init.d', 'stop'] 2017-09-19 13:58:48,563 INFOCLI2 pbspro-ptl: /tmp/PtlPbsVJzCSo Contents of /tmp/PtlPbsVJzCSo: ---------------------------------------- #!/bin/bash sudo PBS_CONF_FILE=/tmp/PtlPbsvqiaZk /usr/libexec/pbs_init.d stop ---------------------------------------- 2017-09-19 13:58:49,595 INFOCLI2 pbspro-ptl: /bin/rm /tmp/PtlPbsVJzCSo 2017-09-19 13:58:49,598 INFO Set {'PBS_START_MOM': 1, 'PBS_START_SERVER': 0, 'PBS_START_SCHED': 0, 'PBS_START_COMM': 0} in /tmp/PtlPbsVoKDsS 2017-09-19 13:58:49,600 INFOCLI2 pbspro-ptl: sudo -H /bin/chown root /tmp/PtlPbsVo0B3q 2017-09-19 13:58:49,606 INFOCLI2 pbspro-ptl: sudo -H /bin/chgrp root /tmp/PtlPbsVo0B3q 2017-09-19 13:58:49,619 INFOCLI2 pbspro-ptl: sudo -H /bin/rm /tmp/PtlPbsVo0B3q 2017-09-19 13:58:49,625 INFOCLI2 pbspro-ptl: sudo -H ls -l /usr/libexec/pbs_init.d 2017-09-19 13:58:49,634 INFO running init script to start pbs mom on pbspro-ptl using /etc/pbs.conf init_cmd=['sudo', 'PBS_CONF_FILE=/tmp/PtlPbsVoKDsS', '/usr/libexec/pbs_init.d', 'start'] 2017-09-19 13:58:49,634 INFOCLI2 pbspro-ptl: /tmp/PtlPbs2cpsdn Contents of /tmp/PtlPbs2cpsdn: ---------------------------------------- #!/bin/bash sudo PBS_CONF_FILE=/tmp/PtlPbsVoKDsS /usr/libexec/pbs_init.d start ---------------------------------------- 2017-09-19 13:58:49,687 INFOCLI2 pbspro-ptl: /bin/rm /tmp/PtlPbs2cpsdn 2017-09-19 13:58:49,701 INFOCLI2 pbspro-ptl: sudo -H /bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-09-19 13:58:49,709 INFO mom pbspro-ptl: config {'$enforce': 'mem'} 2017-09-19 13:58:49,710 INFOCLI2 pbspro-ptl: sudo -H /bin/cp -p /tmp/PtlPbsGgj7c6 /var/spool/pbs/mom_priv/config 2017-09-19 13:58:49,716 INFOCLI2 pbspro-ptl: sudo -H /bin/chmod 0644 /var/spool/pbs/mom_priv/config 2017-09-19 13:58:49,722 INFOCLI2 pbspro-ptl: sudo -H /bin/chown root /var/spool/pbs/mom_priv/config 2017-09-19 13:58:49,728 INFOCLI2 pbspro-ptl: sudo -H /bin/chgrp root /var/spool/pbs/mom_priv/config 2017-09-19 13:58:49,733 INFO mom pbspro-ptl: sent signal -HUP 2017-09-19 13:58:49,733 INFOCLI2 pbspro-ptl: sudo -H kill -HUP 5169 2017-09-19 13:58:49,746 INFOCLI2 pbspro-ptl: sudo -H /bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-09-19 13:58:49,755 INFO Set {'PBS_START_MOM': 1, 'PBS_START_SERVER': 0, 'PBS_START_SCHED': 0, 'PBS_START_COMM': 0} in /tmp/PtlPbsLpAx1L 2017-09-19 13:58:49,757 INFOCLI2 pbspro-ptl: sudo -H /bin/chown root /tmp/PtlPbstRlG8b 2017-09-19 13:58:49,763 INFOCLI2 pbspro-ptl: sudo -H /bin/chgrp root /tmp/PtlPbstRlG8b 2017-09-19 13:58:49,775 INFOCLI2 pbspro-ptl: sudo -H /bin/rm /tmp/PtlPbstRlG8b 2017-09-19 13:58:49,780 INFOCLI2 pbspro-ptl: sudo -H ls -l /usr/libexec/pbs_init.d 2017-09-19 13:58:49,786 INFO running init script to stop pbs mom on pbspro-ptl using /etc/pbs.conf init_cmd=['sudo', 'PBS_CONF_FILE=/tmp/PtlPbsLpAx1L', '/usr/libexec/pbs_init.d', 'stop'] 2017-09-19 13:58:49,786 INFOCLI2 pbspro-ptl: /tmp/PtlPbsG_s1bi Contents of /tmp/PtlPbsG_s1bi: ---------------------------------------- #!/bin/bash sudo PBS_CONF_FILE=/tmp/PtlPbsLpAx1L /usr/libexec/pbs_init.d stop ---------------------------------------- 2017-09-19 13:58:50,816 INFOCLI2 pbspro-ptl: /bin/rm /tmp/PtlPbsG_s1bi 2017-09-19 13:58:50,819 INFO Set {'PBS_START_MOM': 1, 'PBS_START_SERVER': 0, 'PBS_START_SCHED': 0, 'PBS_START_COMM': 0} in /tmp/PtlPbsumvEGV 2017-09-19 13:58:50,821 INFOCLI2 pbspro-ptl: sudo -H /bin/chown root /tmp/PtlPbsYkNJIe 2017-09-19 13:58:50,826 INFOCLI2 pbspro-ptl: sudo -H /bin/chgrp root /tmp/PtlPbsYkNJIe 2017-09-19 13:58:50,838 INFOCLI2 pbspro-ptl: sudo -H /bin/rm /tmp/PtlPbsYkNJIe 2017-09-19 13:58:50,844 INFOCLI2 pbspro-ptl: sudo -H ls -l /usr/libexec/pbs_init.d 2017-09-19 13:58:50,850 INFO running init script to start pbs mom on pbspro-ptl using /etc/pbs.conf init_cmd=['sudo', 'PBS_CONF_FILE=/tmp/PtlPbsumvEGV', '/usr/libexec/pbs_init.d', 'start'] 2017-09-19 13:58:50,850 INFOCLI2 pbspro-ptl: /tmp/PtlPbswlZM2H Contents of /tmp/PtlPbswlZM2H: ---------------------------------------- #!/bin/bash sudo PBS_CONF_FILE=/tmp/PtlPbsumvEGV /usr/libexec/pbs_init.d start ---------------------------------------- 2017-09-19 13:58:50,902 INFOCLI2 pbspro-ptl: /bin/rm /tmp/PtlPbswlZM2H 2017-09-19 13:58:50,915 INFOCLI2 pbspro-ptl: sudo -H /bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-09-19 13:58:50,921 INFO job: executable set to /bin/sleep with arguments: 100 2017-09-19 13:58:50,921 INFOCLI2 pbspro-ptl: sudo -H /bin/chmod 0755 /tmp/PtlPbsJobScriptNrojU7 2017-09-19 13:58:50,927 INFOCLI pbspro-ptl: sudo -H -u pbsuser /usr/bin/qsub -l place=exclhost -l select=1:ncpus=1:mem=1kb /tmp/PtlPbsJobScriptNrojU7 2017-09-19 13:58:50,936 INFO submit to pbspro-ptl as pbsuser: job 4970.pbspro-ptl OrderedDict([('Resource_List.place', 'exclhost'), ('Resource_List.select', '1:ncpus=1:mem=1kb')]) 2017-09-19 13:58:50,936 INFOCLI job script /tmp/PtlPbsJobScriptNrojU7 --- #!/bin/bash sleep 1000 --- 2017-09-19 13:58:50,937 INFOCLI pbspro-ptl: /usr/bin/qstat -f 4970.pbspro-ptl 2017-09-19 13:58:50,942 INFO expect on server pbspro-ptl: job_state = R && substate = 42 job 4970.pbspro-ptl got: job_state = Q 2017-09-19 13:58:51,443 INFOCLI pbspro-ptl: /usr/bin/qmgr -c set server scheduling=True 2017-09-19 13:58:51,545 INFOCLI pbspro-ptl: /usr/bin/qstat -f 4970.pbspro-ptl 2017-09-19 13:58:51,553 INFO expect on server pbspro-ptl: job_state = R && substate = 42 job 4970.pbspro-ptl attempt: 2 ... OK 2017-09-19 13:58:51,553 INFO Giving a chance to pbs to kill the job... 2017-09-19 13:59:06,569 INFOCLI pbspro-ptl: /usr/bin/qstat -f 4970.pbspro-ptl 2017-09-19 13:59:06,576 INFO expect on server pbspro-ptl: job_state = R && substate = 42 job 4970.pbspro-ptl ... OK 2017-09-19 13:59:06,576 INFO ========================================== 2017-09-19 13:59:06,577 INFO Entered Test_enforce_on_exclhost tearDown 2017-09-19 13:59:06,577 INFO ========================================== 2017-09-19 13:59:06,577 INFO =========================================== 2017-09-19 13:59:06,578 INFO Completed Test_enforce_on_exclhost tearDown 2017-09-19 13:59:06,578 INFO =========================================== 2017-09-19 13:59:06,578 INFO ok 2017-09-19 13:59:06,579 INFO ================================================================================ run: 3, succeeded: 3, failed: 0, errors: 0, skipped: 0, timedout: 0 Tests run in 0:00:58.378360 2017-09-19 13:59:06,579 INFO Cleaning up temporary files