2017-10-24 23:22:31,932 INFO input command: pbs_benchpress -f tests/functional/pbs_cray_suspend_resume.py -o pbs_pp864_test1.txt 2017-10-24 23:22:31,933 INFO param: None 2017-10-24 23:22:31,933 INFO ptl version: 17.1.0 2017-10-24 23:22:31,933 INFO platform: Linux linux-kt3k 3.12.28-4-default #1 SMP Thu Sep 25 17:02:34 UTC 2014 (9879bd4) x86_64 x86_64 2017-10-24 23:22:31,933 INFO python version: 2.7.12 2017-10-24 23:22:31,933 INFO user: root 2017-10-24 23:22:31,933 INFO -------------------------------------------------------------------------------- 2017-10-24 23:22:31,933 INFO Cleaning up temporary files 2017-10-24 23:22:31,944 INFO ====================================================================== 2017-10-24 23:22:31,944 INFO suite name: TestSuspendResumeOnCray 2017-10-24 23:22:31,945 INFO suite docstring: Test special cases where suspend/resume functionality differs on cray as compared to other platforms. This test suite expects the platform to be 'cray' and assumes that suspend/resume feature is enabled on it. 2017-10-24 23:22:31,945 INFO ====================================================================== 2017-10-24 23:22:31,945 INFO =========================================== 2017-10-24 23:22:31,945 INFO Entered TestSuspendResumeOnCray setUpClass 2017-10-24 23:22:31,945 INFO =========================================== 2017-10-24 23:22:31,946 INFOCLI2 linux-kt3k: id pbsuser 2017-10-24 23:22:31,949 INFOCLI2 linux-kt3k: id pbsuser1 2017-10-24 23:22:31,951 INFOCLI2 linux-kt3k: id pbsuser2 2017-10-24 23:22:31,954 INFOCLI2 linux-kt3k: id pbsuser3 2017-10-24 23:22:31,964 INFO server linux-kt3k: server operating mode set to cli 2017-10-24 23:22:31,964 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-24 23:22:31,974 INFO server linux-kt3k: version 18.2.0.20171019011017 2017-10-24 23:22:31,974 INFO expect action: created new action kicksched 2017-10-24 23:22:31,975 INFO expect action: added action kicksched to server linux-kt3k 2017-10-24 23:22:32,008 INFOCLI2 linux-kt3k: sudo -H /opt/pbs/sbin/pbsfs 2017-10-24 23:22:32,036 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2017-10-24 23:22:32,054 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2017-10-24 23:22:32,074 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list sched default 2017-10-24 23:22:32,109 INFOCLI2 linux-kt3k: sudo -H /opt/pbs/sbin/pbsfs 2017-10-24 23:22:32,131 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2017-10-24 23:22:32,149 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2017-10-24 23:22:32,173 INFO ============================================ 2017-10-24 23:22:32,173 INFO Completed TestSuspendResumeOnCray setUpClass 2017-10-24 23:22:32,173 INFO ============================================ 2017-10-24 23:22:32,173 INFO test name: test_admin_suspend_wrong_state (tests.functional.pbs_cray_suspend_resume.TestSuspendResumeOnCray)... 2017-10-24 23:22:32,173 INFO test start time: Tue Oct 24 23:22:32 2017 2017-10-24 23:22:32,173 INFO test docstring: Check that wrong 'resume' signal is correctly rejected. 2017-10-24 23:22:32,173 INFO ====================================== 2017-10-24 23:22:32,174 INFO Entered TestSuspendResumeOnCray setUp 2017-10-24 23:22:32,174 INFO ====================================== 2017-10-24 23:22:32,174 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-24 23:22:32,182 INFO status on linux-kt3k: server 2017-10-24 23:22:32,182 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-24 23:22:32,190 INFO manager on linux-kt3k: set server {'managers': (3, 'root@*')} 2017-10-24 23:22:32,190 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c set server managers-=root@* 2017-10-24 23:22:32,220 INFO manager on linux-kt3k: set server {'managers': (2, 'root@*')} 2017-10-24 23:22:32,221 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c set server managers+=root@* 2017-10-24 23:22:32,247 INFO server linux-kt3k: reverting configuration to defaults 2017-10-24 23:22:32,248 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-24 23:22:32,256 INFO select on linux-kt3k: __ALL__ 2017-10-24 23:22:32,257 INFOCLI linux-kt3k: /opt/pbs/bin/qselect 2017-10-24 23:22:32,266 INFO delete job on linux-kt3k: 126.linux-kt3k 2017-10-24 23:22:32,266 INFOCLI linux-kt3k: /opt/pbs/bin/qdel -W force 126.linux-kt3k 2017-10-24 23:22:32,291 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 126.linux-kt3k 2017-10-24 23:22:32,302 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f @linux-kt3k 2017-10-24 23:22:32,313 INFO expect on server linux-kt3k: job_state set 0 job ... OK 2017-10-24 23:22:32,313 INFOCLI linux-kt3k: /opt/pbs/bin/pbs_rstat -f 2017-10-24 23:22:32,326 INFO manager on linux-kt3k: unset server ['pbs_license_min', 'max_run_res_soft.ncpus', 'restrict_res_to_release_on_suspend', 'pbs_license_max'] 2017-10-24 23:22:32,326 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c unset server pbs_license_min,max_run_res_soft.ncpus,restrict_res_to_release_on_suspend,pbs_license_max 2017-10-24 23:22:32,348 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-10-24 23:22:32,377 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Qf @linux-kt3k 2017-10-24 23:22:32,396 INFO status on linux-kt3k: node 2017-10-24 23:22:32,396 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v -a 2017-10-24 23:22:32,408 INFO manager on linux-kt3k: delete queue workq,expressq 2017-10-24 23:22:32,408 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c delete queue workq,expressq 2017-10-24 23:22:32,423 INFO server linux-kt3k: expect offset set to 0.5 2017-10-24 23:22:32,924 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Qf workq@linux-kt3k 2017-10-24 23:22:32,933 INFO expect on server linux-kt3k: unset queue workq ... OK 2017-10-24 23:22:32,934 INFO server linux-kt3k: expect offset set to 0.5 2017-10-24 23:22:33,435 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Qf expressq@linux-kt3k 2017-10-24 23:22:33,444 INFO expect on server linux-kt3k: unset queue expressq ... OK 2017-10-24 23:22:33,444 INFO manager on linux-kt3k: create queue workq {'started': 'True', 'queue_type': 'Execution', 'enabled': 'True'} 2017-10-24 23:22:33,444 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c create queue workq started=True,queue_type=Execution,enabled=True 2017-10-24 23:22:33,456 INFO server linux-kt3k: expect offset set to 0.5 2017-10-24 23:22:33,958 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Qf workq@linux-kt3k 2017-10-24 23:22:33,969 INFO expect on server linux-kt3k: started set True || queue_type set Execution || enabled set True queue workq ... OK 2017-10-24 23:22:33,969 INFOCLI2 linux-kt3k: sudo -H cmp /opt/pbs/lib/python/altair/pbs_hooks/PBS_translate_mpp.HK /var/spool/pbs/server_priv/hooks/PBS_translate_mpp.HK 2017-10-24 23:22:33,990 INFOCLI2 linux-kt3k: which cp 2017-10-24 23:22:33,992 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /opt/pbs/lib/python/altair/pbs_hooks/PBS_translate_mpp.HK /var/spool/pbs/server_priv/hooks/PBS_translate_mpp.HK 2017-10-24 23:22:34,018 INFOCLI2 linux-kt3k: which chmod 2017-10-24 23:22:34,020 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/server_priv/hooks/PBS_translate_mpp.HK 2017-10-24 23:22:34,039 INFO server linux-kt3k: sent signal -HUP 2017-10-24 23:22:34,039 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/server_priv/server.lock 2017-10-24 23:22:34,062 INFOCLI2 linux-kt3k: sudo -H kill -HUP 74316 2017-10-24 23:22:34,084 INFO manager on linux-kt3k: set pbshook PBS_translate_mpp {'enabled': 'true'} 2017-10-24 23:22:34,085 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c set pbshook PBS_translate_mpp enabled=true 2017-10-24 23:22:34,113 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-10-24 23:22:34,146 INFO manager on linux-kt3k: set server {'eligible_time_enable': 'False', 'default_queue': 'workq', 'restrict_res_to_release_on_suspend': 'ncpus'} 2017-10-24 23:22:34,146 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server eligible_time_enable=False,default_queue=workq,restrict_res_to_release_on_suspend=ncpus 2017-10-24 23:22:34,161 INFO status on linux-kt3k: resource 2017-10-24 23:22:34,161 INFO manager on linux-kt3k: list resource 2017-10-24 23:22:34,161 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource 2017-10-24 23:22:34,171 INFO manager on linux-kt3k: delete resource PBScraylabel_G34_IL,PBScrayorder,PBScrayhost,PBScraynid 2017-10-24 23:22:34,171 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c delete resource PBScraylabel_G34_IL,PBScrayorder,PBScrayhost,PBScraynid 2017-10-24 23:22:34,254 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource PBScraylabel_G34_IL 2017-10-24 23:22:34,263 INFO expect on server linux-kt3k: unset resource PBScraylabel_G34_IL ... OK 2017-10-24 23:22:34,263 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource PBScrayorder 2017-10-24 23:22:34,271 INFO expect on server linux-kt3k: unset resource PBScrayorder ... OK 2017-10-24 23:22:34,272 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource PBScrayhost 2017-10-24 23:22:34,282 INFO expect on server linux-kt3k: unset resource PBScrayhost ... OK 2017-10-24 23:22:34,282 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource PBScraynid 2017-10-24 23:22:34,294 INFO expect on server linux-kt3k: unset resource PBScraynid ... OK 2017-10-24 23:22:34,294 INFOCLI status on linux-kt3k: server license_count 2017-10-24 23:22:34,294 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-24 23:22:34,303 INFO server: linux-kt3k licensed 2017-10-24 23:22:34,313 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/server_priv/comm.lock 2017-10-24 23:22:34,341 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-10-24 23:22:34,362 INFO scheduler linux-kt3k: reverting configuration to defaults 2017-10-24 23:22:34,363 INFO manager on linux-kt3k: delete sched @default 2017-10-24 23:22:34,363 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c delete sched @default 2017-10-24 23:22:34,374 INFO manager on linux-kt3k: list sched 2017-10-24 23:22:34,374 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list sched 2017-10-24 23:22:34,383 INFO manager on linux-kt3k: unset sched ['sched_cycle_length'] 2017-10-24 23:22:34,383 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c unset sched sched_cycle_length 2017-10-24 23:22:34,393 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/dedicated_time 2017-10-24 23:22:34,411 INFOCLI2 linux-kt3k: cmp /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-10-24 23:22:34,414 INFO scheduler linux-kt3k: reverting holidays file to default 2017-10-24 23:22:34,414 INFOCLI2 linux-kt3k: cmp /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-10-24 23:22:34,416 INFOCLI2 linux-kt3k: cmp /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-10-24 23:22:34,418 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-10-24 23:22:34,437 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-10-24 23:22:34,456 INFO scheduler linux-kt3k: sent signal -HUP 2017-10-24 23:22:34,456 INFOCLI2 linux-kt3k: sudo -H kill -HUP 96312 2017-10-24 23:22:34,483 INFOCLI2 linux-kt3k: sudo -H /opt/pbs/sbin/pbsfs -e 2017-10-24 23:22:34,527 INFO scheduler linux-kt3k: config {'resources': '"ncpus, mem, arch, host, vnode, aoe, eoe, vntype"'} 2017-10-24 23:22:34,528 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /var/spool/pbs/sched_priv/sched_config /var/spool/pbs/sched_priv/sched_config.bak 2017-10-24 23:22:34,546 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /tmp/PtlPbs64Afla /var/spool/pbs/sched_priv/sched_config 2017-10-24 23:22:34,567 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-10-24 23:22:34,590 INFOCLI2 linux-kt3k: which chown 2017-10-24 23:22:34,593 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chown root /var/spool/pbs/sched_priv/sched_config 2017-10-24 23:22:34,621 INFOCLI2 linux-kt3k: which chgrp 2017-10-24 23:22:34,623 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chgrp root /var/spool/pbs/sched_priv/sched_config 2017-10-24 23:22:34,648 INFO scheduler linux-kt3k: sent signal -HUP 2017-10-24 23:22:34,648 INFOCLI2 linux-kt3k: sudo -H kill -HUP 96312 2017-10-24 23:22:34,678 INFO scheduler linux-kt3k log match: searching for "Error reading line" - No match 2017-10-24 23:22:35,688 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-10-24 23:22:35,715 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-10-24 23:22:35,735 INFOCLI2 linux-kt3k: sudo -H /opt/pbs/sbin/pbs_mom --version 2017-10-24 23:22:35,779 INFO mom linux-kt3k: reverting configuration to defaults 2017-10-24 23:22:35,780 INFOCLI2 linux-kt3k: which rm 2017-10-24 23:22:35,783 INFOCLI2 linux-kt3k: sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2017-10-24 23:22:35,803 INFOCLI2 linux-kt3k: sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2017-10-24 23:22:35,828 INFOCLI linux-kt3k: sudo -H /opt/pbs/sbin/pbs_mom -s list 2017-10-24 23:22:35,870 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /tmp/PtlPbsZgLIvs /var/spool/pbs/mom_priv/config 2017-10-24 23:22:35,898 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/mom_priv/config 2017-10-24 23:22:35,920 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chown root /var/spool/pbs/mom_priv/config 2017-10-24 23:22:35,945 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chgrp root /var/spool/pbs/mom_priv/config 2017-10-24 23:22:35,972 INFO mom linux-kt3k: sent signal -HUP 2017-10-24 23:22:35,972 INFOCLI2 linux-kt3k: sudo -H kill -HUP 96290 2017-10-24 23:22:36,019 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-10-24 23:22:36,051 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v -a 2017-10-24 23:22:36,063 INFO manager on linux-kt3k: delete node 2017-10-24 23:22:36,064 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c delete node @default 2017-10-24 23:22:36,078 INFO manager on linux-kt3k: create node linux-kt3k 2017-10-24 23:22:36,078 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c create node linux-kt3k 2017-10-24 23:22:36,104 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k 2017-10-24 23:22:36,124 INFO expect on server linux-kt3k: state = free node linux-kt3k got: state = state-unknown,down 2017-10-24 23:22:37,126 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k 2017-10-24 23:22:37,138 INFO expect on server linux-kt3k: state = free node linux-kt3k attempt: 2 got: state = state-unknown,down 2017-10-24 23:22:38,144 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k 2017-10-24 23:22:38,180 INFO expect on server linux-kt3k: state = free node linux-kt3k attempt: 3 ... OK 2017-10-24 23:22:38,181 INFO ======================================= 2017-10-24 23:22:38,181 INFO Completed TestSuspendResumeOnCray setUp 2017-10-24 23:22:38,181 INFO ======================================= 2017-10-24 23:22:38,181 INFO job: executable set to /bin/sleep with arguments: 100 2017-10-24 23:22:38,181 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0755 /tmp/PtlPbsJobScriptqfk5XL 2017-10-24 23:22:38,203 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chown pbsuser /tmp/PtlPbsJobScriptqfk5XL 2017-10-24 23:22:38,222 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chgrp tstgrp00 /tmp/PtlPbsJobScriptqfk5XL 2017-10-24 23:22:38,240 INFOCLI linux-kt3k: sudo -H -u pbsuser /opt/pbs/bin/qsub -l vntype=cray_compute /tmp/PtlPbsJobScriptqfk5XL 2017-10-24 23:22:38,277 INFO submit to linux-kt3k as pbsuser: job 127.linux-kt3k OrderedDict([('Resource_List.vntype', 'cray_compute')]) 2017-10-24 23:22:38,277 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 127.linux-kt3k 2017-10-24 23:22:38,286 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 127.linux-kt3k got: job_state = Q 2017-10-24 23:22:38,788 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:22:38,799 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 127.linux-kt3k 2017-10-24 23:22:38,808 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 127.linux-kt3k attempt: 2 ... OK 2017-10-24 23:22:38,808 INFO signal on linux-kt3k as root: 127.linux-kt3k with signal = suspend 2017-10-24 23:22:38,808 INFOCLI linux-kt3k: /opt/pbs/bin/qsig -s suspend 127.linux-kt3k 2017-10-24 23:22:48,097 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 127.linux-kt3k 2017-10-24 23:22:48,109 INFO expect on server linux-kt3k: job_state = S job 127.linux-kt3k ... OK 2017-10-24 23:22:48,110 INFO signal on linux-kt3k as root: 127.linux-kt3k with signal = admin-resume 2017-10-24 23:22:48,110 INFOCLI linux-kt3k: /opt/pbs/bin/qsig -s admin-resume 127.linux-kt3k 2017-10-24 23:22:48,118 ERROR err: ['qsig: Job can not be resumed with the requested resume signal 127.linux-kt3k'] 2017-10-24 23:22:48,118 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 127.linux-kt3k 2017-10-24 23:22:48,130 INFO expect on server linux-kt3k: job_state = S job 127.linux-kt3k ... OK 2017-10-24 23:22:48,131 INFO job: executable set to /bin/sleep with arguments: 100 2017-10-24 23:22:48,132 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0755 /tmp/PtlPbsJobScriptS6kKca 2017-10-24 23:22:48,163 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chown pbsuser /tmp/PtlPbsJobScriptS6kKca 2017-10-24 23:22:48,184 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chgrp tstgrp00 /tmp/PtlPbsJobScriptS6kKca 2017-10-24 23:22:48,208 INFOCLI linux-kt3k: sudo -H -u pbsuser /opt/pbs/bin/qsub -l vntype=cray_compute /tmp/PtlPbsJobScriptS6kKca 2017-10-24 23:22:48,238 INFO submit to linux-kt3k as pbsuser: job 128.linux-kt3k OrderedDict([('Resource_List.vntype', 'cray_compute')]) 2017-10-24 23:22:48,238 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 128.linux-kt3k 2017-10-24 23:22:48,247 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 128.linux-kt3k got: job_state = Q 2017-10-24 23:22:48,748 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:22:48,762 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 128.linux-kt3k 2017-10-24 23:22:48,773 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 128.linux-kt3k attempt: 2 ... OK 2017-10-24 23:22:48,774 INFO signal on linux-kt3k as root: 128.linux-kt3k with signal = admin-suspend 2017-10-24 23:22:48,774 INFOCLI linux-kt3k: /opt/pbs/bin/qsig -s admin-suspend 128.linux-kt3k 2017-10-24 23:22:58,048 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 128.linux-kt3k 2017-10-24 23:22:58,057 INFO expect on server linux-kt3k: job_state = S || substate = 43 job 128.linux-kt3k ... OK 2017-10-24 23:22:58,058 INFO signal on linux-kt3k as root: 128.linux-kt3k with signal = resume 2017-10-24 23:22:58,058 INFOCLI linux-kt3k: /opt/pbs/bin/qsig -s resume 128.linux-kt3k 2017-10-24 23:22:58,066 ERROR err: ['qsig: Job can not be resumed with the requested resume signal 128.linux-kt3k'] 2017-10-24 23:22:58,067 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 128.linux-kt3k 2017-10-24 23:22:58,076 INFO expect on server linux-kt3k: job_state = S || substate = 43 job 128.linux-kt3k ... OK 2017-10-24 23:22:58,076 INFO ========================================= 2017-10-24 23:22:58,077 INFO Entered TestSuspendResumeOnCray tearDown 2017-10-24 23:22:58,077 INFO ========================================= 2017-10-24 23:22:58,077 INFO ========================================== 2017-10-24 23:22:58,077 INFO Completed TestSuspendResumeOnCray tearDown 2017-10-24 23:22:58,077 INFO ========================================== 2017-10-24 23:22:58,078 INFO ok 2017-10-24 23:22:58,078 INFO test name: test_basic_admin_suspend_restart (tests.functional.pbs_cray_suspend_resume.TestSuspendResumeOnCray)... 2017-10-24 23:22:58,078 INFO test start time: Tue Oct 24 23:22:58 2017 2017-10-24 23:22:58,078 INFO test docstring: Test basic admin-suspend funcionality for jobs and array jobs with restart on Cray. The restart will test if the node recovers properly in maintenance. After turning off scheduling and a server restart, a subjob is always requeued and node shows up as free. 2017-10-24 23:22:58,078 INFO ====================================== 2017-10-24 23:22:58,078 INFO Entered TestSuspendResumeOnCray setUp 2017-10-24 23:22:58,078 INFO ====================================== 2017-10-24 23:22:58,079 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-24 23:22:58,090 INFO status on linux-kt3k: server 2017-10-24 23:22:58,090 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-24 23:22:58,099 INFO manager on linux-kt3k: set server {'managers': (3, 'root@*')} 2017-10-24 23:22:58,099 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c set server managers-=root@* 2017-10-24 23:22:58,133 INFO manager on linux-kt3k: set server {'managers': (2, 'root@*')} 2017-10-24 23:22:58,133 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c set server managers+=root@* 2017-10-24 23:22:58,166 INFO server linux-kt3k: reverting configuration to defaults 2017-10-24 23:22:58,167 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-24 23:22:58,177 INFO select on linux-kt3k: __ALL__ 2017-10-24 23:22:58,177 INFOCLI linux-kt3k: /opt/pbs/bin/qselect 2017-10-24 23:22:58,185 INFO delete job on linux-kt3k: 127.linux-kt3k, 128.linux-kt3k 2017-10-24 23:22:58,185 INFOCLI linux-kt3k: /opt/pbs/bin/qdel -W force 127.linux-kt3k 128.linux-kt3k 2017-10-24 23:22:58,214 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 127.linux-kt3k 2017-10-24 23:22:58,239 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 128.linux-kt3k 2017-10-24 23:22:58,275 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f @linux-kt3k 2017-10-24 23:22:58,293 INFO expect on server linux-kt3k: job_state set 0 job ... OK 2017-10-24 23:22:58,293 INFOCLI linux-kt3k: /opt/pbs/bin/pbs_rstat -f 2017-10-24 23:22:58,314 INFO manager on linux-kt3k: unset server ['restrict_res_to_release_on_suspend'] 2017-10-24 23:22:58,314 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c unset server restrict_res_to_release_on_suspend 2017-10-24 23:22:58,333 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-10-24 23:22:58,386 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Qf @linux-kt3k 2017-10-24 23:22:58,400 INFO status on linux-kt3k: node 2017-10-24 23:22:58,400 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v -a 2017-10-24 23:22:58,413 INFO manager on linux-kt3k: delete queue workq 2017-10-24 23:22:58,413 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c delete queue workq 2017-10-24 23:22:58,424 INFO server linux-kt3k: expect offset set to 0.5 2017-10-24 23:22:58,926 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Qf workq@linux-kt3k 2017-10-24 23:22:58,935 INFO expect on server linux-kt3k: unset queue workq ... OK 2017-10-24 23:22:58,935 INFO manager on linux-kt3k: create queue workq {'started': 'True', 'queue_type': 'Execution', 'enabled': 'True'} 2017-10-24 23:22:58,935 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c create queue workq started=True,queue_type=Execution,enabled=True 2017-10-24 23:22:58,945 INFO server linux-kt3k: expect offset set to 0.5 2017-10-24 23:22:59,446 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Qf workq@linux-kt3k 2017-10-24 23:22:59,456 INFO expect on server linux-kt3k: started set True || queue_type set Execution || enabled set True queue workq ... OK 2017-10-24 23:22:59,456 INFOCLI2 linux-kt3k: sudo -H cmp /opt/pbs/lib/python/altair/pbs_hooks/PBS_translate_mpp.HK /var/spool/pbs/server_priv/hooks/PBS_translate_mpp.HK 2017-10-24 23:22:59,476 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /opt/pbs/lib/python/altair/pbs_hooks/PBS_translate_mpp.HK /var/spool/pbs/server_priv/hooks/PBS_translate_mpp.HK 2017-10-24 23:22:59,501 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/server_priv/hooks/PBS_translate_mpp.HK 2017-10-24 23:22:59,522 INFO server linux-kt3k: sent signal -HUP 2017-10-24 23:22:59,522 INFOCLI2 linux-kt3k: sudo -H kill -HUP 74316 2017-10-24 23:22:59,542 INFO manager on linux-kt3k: set pbshook PBS_translate_mpp {'enabled': 'true'} 2017-10-24 23:22:59,542 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c set pbshook PBS_translate_mpp enabled=true 2017-10-24 23:22:59,571 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-10-24 23:22:59,602 INFO manager on linux-kt3k: set server {'default_queue': 'workq', 'restrict_res_to_release_on_suspend': 'ncpus'} 2017-10-24 23:22:59,602 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server default_queue=workq,restrict_res_to_release_on_suspend=ncpus 2017-10-24 23:22:59,620 INFO status on linux-kt3k: resource 2017-10-24 23:22:59,621 INFO manager on linux-kt3k: list resource 2017-10-24 23:22:59,621 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource 2017-10-24 23:22:59,633 INFO manager on linux-kt3k: delete resource PBScraylabel_G34_IL,PBScrayorder,PBScrayhost,PBScraynid 2017-10-24 23:22:59,633 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c delete resource PBScraylabel_G34_IL,PBScrayorder,PBScrayhost,PBScraynid 2017-10-24 23:22:59,711 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource PBScraylabel_G34_IL 2017-10-24 23:22:59,723 INFO expect on server linux-kt3k: unset resource PBScraylabel_G34_IL ... OK 2017-10-24 23:22:59,723 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource PBScrayorder 2017-10-24 23:22:59,731 INFO expect on server linux-kt3k: unset resource PBScrayorder ... OK 2017-10-24 23:22:59,731 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource PBScrayhost 2017-10-24 23:22:59,740 INFO expect on server linux-kt3k: unset resource PBScrayhost ... OK 2017-10-24 23:22:59,740 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource PBScraynid 2017-10-24 23:22:59,752 INFO expect on server linux-kt3k: unset resource PBScraynid ... OK 2017-10-24 23:22:59,753 INFOCLI status on linux-kt3k: server license_count 2017-10-24 23:22:59,753 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-24 23:22:59,763 INFO server: linux-kt3k licensed 2017-10-24 23:22:59,775 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/server_priv/comm.lock 2017-10-24 23:22:59,804 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-10-24 23:22:59,822 INFO scheduler linux-kt3k: reverting configuration to defaults 2017-10-24 23:22:59,822 INFO manager on linux-kt3k: delete sched @default 2017-10-24 23:22:59,822 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c delete sched @default 2017-10-24 23:22:59,836 INFO manager on linux-kt3k: list sched 2017-10-24 23:22:59,836 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list sched 2017-10-24 23:22:59,846 INFO manager on linux-kt3k: unset sched ['sched_cycle_length'] 2017-10-24 23:22:59,846 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c unset sched sched_cycle_length 2017-10-24 23:22:59,859 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/dedicated_time 2017-10-24 23:22:59,881 INFOCLI2 linux-kt3k: cmp /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-10-24 23:22:59,884 INFO scheduler linux-kt3k: reverting holidays file to default 2017-10-24 23:22:59,884 INFOCLI2 linux-kt3k: cmp /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-10-24 23:22:59,886 INFOCLI2 linux-kt3k: cmp /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-10-24 23:22:59,889 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-10-24 23:22:59,912 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-10-24 23:22:59,933 INFO scheduler linux-kt3k: sent signal -HUP 2017-10-24 23:22:59,933 INFOCLI2 linux-kt3k: sudo -H kill -HUP 96312 2017-10-24 23:22:59,952 INFOCLI2 linux-kt3k: sudo -H /opt/pbs/sbin/pbsfs -e 2017-10-24 23:22:59,998 INFO scheduler linux-kt3k: config {'resources': '"ncpus, mem, arch, host, vnode, aoe, eoe, vntype"'} 2017-10-24 23:22:59,999 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /var/spool/pbs/sched_priv/sched_config /var/spool/pbs/sched_priv/sched_config.bak 2017-10-24 23:23:00,025 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /tmp/PtlPbsfObYCs /var/spool/pbs/sched_priv/sched_config 2017-10-24 23:23:00,049 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-10-24 23:23:00,076 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chown root /var/spool/pbs/sched_priv/sched_config 2017-10-24 23:23:00,100 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chgrp root /var/spool/pbs/sched_priv/sched_config 2017-10-24 23:23:00,123 INFO scheduler linux-kt3k: sent signal -HUP 2017-10-24 23:23:00,123 INFOCLI2 linux-kt3k: sudo -H kill -HUP 96312 2017-10-24 23:23:00,145 INFO scheduler linux-kt3k log match: searching for "Error reading line" - No match 2017-10-24 23:23:01,156 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-10-24 23:23:01,182 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-10-24 23:23:01,208 INFO mom linux-kt3k: reverting configuration to defaults 2017-10-24 23:23:01,208 INFOCLI2 linux-kt3k: sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2017-10-24 23:23:01,233 INFOCLI2 linux-kt3k: sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2017-10-24 23:23:01,254 INFOCLI linux-kt3k: sudo -H /opt/pbs/sbin/pbs_mom -s list 2017-10-24 23:23:01,300 INFO mom linux-kt3k: sent signal -HUP 2017-10-24 23:23:01,300 INFOCLI2 linux-kt3k: sudo -H kill -HUP 96290 2017-10-24 23:23:01,332 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-10-24 23:23:01,364 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v -a 2017-10-24 23:23:01,374 INFO manager on linux-kt3k: delete node 2017-10-24 23:23:01,374 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c delete node @default 2017-10-24 23:23:01,388 INFO manager on linux-kt3k: create node linux-kt3k 2017-10-24 23:23:01,388 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c create node linux-kt3k 2017-10-24 23:23:01,407 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k 2017-10-24 23:23:01,422 INFO expect on server linux-kt3k: state = free node linux-kt3k got: state = state-unknown,down 2017-10-24 23:23:02,424 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k 2017-10-24 23:23:02,436 INFO expect on server linux-kt3k: state = free node linux-kt3k attempt: 2 got: state = state-unknown,down 2017-10-24 23:23:03,440 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k 2017-10-24 23:23:03,476 INFO expect on server linux-kt3k: state = free node linux-kt3k attempt: 3 ... OK 2017-10-24 23:23:03,476 INFO ======================================= 2017-10-24 23:23:03,476 INFO Completed TestSuspendResumeOnCray setUp 2017-10-24 23:23:03,476 INFO ======================================= 2017-10-24 23:23:03,476 INFO job: executable set to /bin/sleep with arguments: 100 2017-10-24 23:23:03,477 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0755 /tmp/PtlPbsJobScriptbNHOoF 2017-10-24 23:23:03,497 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chown pbsuser /tmp/PtlPbsJobScriptbNHOoF 2017-10-24 23:23:03,516 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chgrp tstgrp00 /tmp/PtlPbsJobScriptbNHOoF 2017-10-24 23:23:03,534 INFOCLI linux-kt3k: sudo -H -u pbsuser /opt/pbs/bin/qsub -l vntype=cray_compute /tmp/PtlPbsJobScriptbNHOoF 2017-10-24 23:23:03,558 INFO submit to linux-kt3k as pbsuser: job 129.linux-kt3k OrderedDict([('Resource_List.vntype', 'cray_compute')]) 2017-10-24 23:23:03,559 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 129.linux-kt3k 2017-10-24 23:23:03,568 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 129.linux-kt3k got: job_state = Q 2017-10-24 23:23:04,070 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:23:04,083 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 129.linux-kt3k 2017-10-24 23:23:04,095 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 129.linux-kt3k attempt: 2 ... OK 2017-10-24 23:23:04,095 INFO status on linux-kt3k: job 129.linux-kt3k exec_vnode 2017-10-24 23:23:04,095 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 129.linux-kt3k 2017-10-24 23:23:04,104 INFO signal on linux-kt3k as root: 129.linux-kt3k with signal = admin-suspend 2017-10-24 23:23:04,104 INFOCLI linux-kt3k: /opt/pbs/bin/qsig -s admin-suspend 129.linux-kt3k 2017-10-24 23:23:14,047 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 129.linux-kt3k 2017-10-24 23:23:14,056 INFO expect on server linux-kt3k: job_state = S job 129.linux-kt3k ... OK 2017-10-24 23:23:14,056 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-24 23:23:14,065 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 ... OK 2017-10-24 23:23:14,066 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v -a 2017-10-24 23:23:14,075 INFO expect on server linux-kt3k: maintenance_jobs = 129.linux-kt3k node ... OK 2017-10-24 23:23:14,075 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-24 23:23:14,086 INFO Set {'PBS_START_MOM': 0, 'PBS_START_SERVER': 1, 'PBS_START_SCHED': 0, 'PBS_START_COMM': 0} in /tmp/PtlPbs1XwgTR 2017-10-24 23:23:14,088 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chown root /tmp/PtlPbsR0zlkW 2017-10-24 23:23:14,110 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chgrp root /tmp/PtlPbsR0zlkW 2017-10-24 23:23:14,150 INFOCLI2 linux-kt3k: sudo -H /usr/bin/rm /tmp/PtlPbsR0zlkW 2017-10-24 23:23:14,168 INFOCLI2 linux-kt3k: sudo -H ls -l /opt/pbs/libexec/pbs_init.d 2017-10-24 23:23:14,190 INFO running init script to stop pbs server on linux-kt3k using /etc/pbs.conf init_cmd=['sudo', 'PBS_CONF_FILE=/tmp/PtlPbs1XwgTR', '/opt/pbs/libexec/pbs_init.d', 'stop'] 2017-10-24 23:23:14,190 INFOCLI2 linux-kt3k: /tmp/PtlPbsgDptPm Contents of /tmp/PtlPbsgDptPm: ---------------------------------------- #!/bin/bash sudo PBS_CONF_FILE=/tmp/PtlPbs1XwgTR /opt/pbs/libexec/pbs_init.d stop ---------------------------------------- 2017-10-24 23:23:18,762 INFOCLI2 linux-kt3k: /usr/bin/rm /tmp/PtlPbsgDptPm 2017-10-24 23:23:18,766 INFO Set {'PBS_START_MOM': 0, 'PBS_START_SERVER': 1, 'PBS_START_SCHED': 0, 'PBS_START_COMM': 0} in /tmp/PtlPbsc4ItY6 2017-10-24 23:23:18,768 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chown root /tmp/PtlPbsAFtMnA 2017-10-24 23:23:18,787 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chgrp root /tmp/PtlPbsAFtMnA 2017-10-24 23:23:18,833 INFOCLI2 linux-kt3k: sudo -H /usr/bin/rm /tmp/PtlPbsAFtMnA 2017-10-24 23:23:18,853 INFOCLI2 linux-kt3k: sudo -H ls -l /opt/pbs/libexec/pbs_init.d 2017-10-24 23:23:18,875 INFO running init script to start pbs server on linux-kt3k using /etc/pbs.conf init_cmd=['sudo', 'PBS_CONF_FILE=/tmp/PtlPbsc4ItY6', '/opt/pbs/libexec/pbs_init.d', 'start'] 2017-10-24 23:23:18,875 INFOCLI2 linux-kt3k: /tmp/PtlPbsTyk16Y Contents of /tmp/PtlPbsTyk16Y: ---------------------------------------- #!/bin/bash sudo PBS_CONF_FILE=/tmp/PtlPbsc4ItY6 /opt/pbs/libexec/pbs_init.d start ---------------------------------------- 2017-10-24 23:23:23,330 INFOCLI2 linux-kt3k: /usr/bin/rm /tmp/PtlPbsTyk16Y 2017-10-24 23:23:23,357 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/server_priv/server.lock 2017-10-24 23:23:23,381 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-24 23:23:23,390 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-24 23:23:23,399 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 got: state = job-exclusive,maintenance 2017-10-24 23:23:23,901 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-24 23:23:23,912 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 2 got: state = job-exclusive,maintenance 2017-10-24 23:23:24,415 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-24 23:23:24,427 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 3 got: state = job-exclusive,maintenance 2017-10-24 23:23:24,929 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-24 23:23:24,940 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 4 got: state = job-exclusive,maintenance 2017-10-24 23:23:25,441 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-24 23:23:25,452 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 5 got: state = job-exclusive,maintenance 2017-10-24 23:23:25,954 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-24 23:23:25,964 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 6 got: state = job-exclusive,maintenance 2017-10-24 23:23:26,465 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-24 23:23:26,476 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 7 got: state = job-exclusive,maintenance 2017-10-24 23:23:26,977 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-24 23:23:26,989 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 8 got: state = job-exclusive,maintenance 2017-10-24 23:23:27,490 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-24 23:23:27,501 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 9 got: state = job-exclusive,maintenance 2017-10-24 23:23:28,002 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-24 23:23:28,013 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 10 got: state = job-exclusive,maintenance 2017-10-24 23:23:28,514 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-24 23:23:28,525 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 11 got: state = job-exclusive,maintenance 2017-10-24 23:23:29,027 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-24 23:23:29,038 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 12 got: state = job-exclusive,maintenance 2017-10-24 23:23:29,539 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-24 23:23:29,549 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 13 got: state = job-exclusive,maintenance 2017-10-24 23:23:30,051 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-24 23:23:30,062 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 14 got: state = job-exclusive,maintenance 2017-10-24 23:23:30,563 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-24 23:23:30,574 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 15 got: state = job-exclusive,maintenance 2017-10-24 23:23:31,076 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-24 23:23:31,088 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 16 got: state = job-exclusive,maintenance 2017-10-24 23:23:31,590 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-24 23:23:31,600 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 17 got: state = job-exclusive,maintenance 2017-10-24 23:23:32,102 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-24 23:23:32,111 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 18 got: state = job-exclusive,maintenance 2017-10-24 23:23:32,612 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-24 23:23:32,623 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 19 got: state = job-exclusive,maintenance 2017-10-24 23:23:33,124 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-24 23:23:33,137 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 20 got: state = job-exclusive,maintenance 2017-10-24 23:23:33,639 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-24 23:23:33,648 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 21 got: state = job-exclusive,maintenance 2017-10-24 23:23:34,150 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-24 23:23:34,160 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 22 got: state = job-exclusive,maintenance 2017-10-24 23:23:34,662 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-24 23:23:34,673 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 23 got: state = job-exclusive,maintenance 2017-10-24 23:23:35,175 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-24 23:23:35,187 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 24 got: state = job-exclusive,maintenance 2017-10-24 23:23:35,689 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-24 23:23:35,699 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 25 got: state = job-exclusive,maintenance 2017-10-24 23:23:36,200 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-24 23:23:36,210 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 26 got: state = job-exclusive,maintenance 2017-10-24 23:23:36,712 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-24 23:23:36,723 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 27 got: state = job-exclusive,maintenance 2017-10-24 23:23:37,225 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-24 23:23:37,237 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 28 got: state = job-exclusive,maintenance 2017-10-24 23:23:37,739 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-24 23:23:37,749 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 29 got: state = job-exclusive,maintenance 2017-10-24 23:23:38,251 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-24 23:23:38,261 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 30 got: state = job-exclusive,maintenance 2017-10-24 23:23:38,762 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-24 23:23:38,771 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 31 got: state = job-exclusive,maintenance 2017-10-24 23:23:39,273 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-24 23:23:39,282 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 32 got: state = job-exclusive,maintenance 2017-10-24 23:23:39,783 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-24 23:23:39,792 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 33 got: state = job-exclusive,maintenance 2017-10-24 23:23:40,295 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-24 23:23:40,305 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 34 got: state = job-exclusive,maintenance 2017-10-24 23:23:40,808 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-24 23:23:40,817 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 35 got: state = job-exclusive,maintenance 2017-10-24 23:23:41,318 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-24 23:23:41,329 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 36 got: state = job-exclusive,maintenance 2017-10-24 23:23:41,830 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-24 23:23:41,843 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 37 got: state = job-exclusive,maintenance 2017-10-24 23:23:42,344 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-24 23:23:42,354 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 38 got: state = job-exclusive,maintenance 2017-10-24 23:23:42,856 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-24 23:23:42,866 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 39 got: state = job-exclusive,maintenance 2017-10-24 23:23:43,368 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-24 23:23:43,377 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 40 got: state = job-exclusive,maintenance 2017-10-24 23:23:43,879 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-24 23:23:43,889 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 41 got: state = job-exclusive,maintenance 2017-10-24 23:23:44,391 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-24 23:23:44,402 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 42 got: state = job-exclusive,maintenance 2017-10-24 23:23:44,904 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-24 23:23:44,915 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 43 got: state = job-exclusive,maintenance 2017-10-24 23:23:45,417 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-24 23:23:45,429 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 44 got: state = job-exclusive,maintenance 2017-10-24 23:23:45,931 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-24 23:23:45,941 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 45 got: state = job-exclusive,maintenance 2017-10-24 23:23:46,443 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-24 23:23:46,455 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 46 got: state = job-exclusive,maintenance 2017-10-24 23:23:46,956 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-24 23:23:46,967 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 47 got: state = job-exclusive,maintenance 2017-10-24 23:23:47,468 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-24 23:23:47,478 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 48 got: state = job-exclusive,maintenance 2017-10-24 23:23:47,979 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-24 23:23:47,988 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 49 got: state = job-exclusive,maintenance 2017-10-24 23:23:48,490 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-24 23:23:48,500 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 50 got: state = job-exclusive,maintenance 2017-10-24 23:23:49,001 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-24 23:23:49,012 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 51 got: state = job-exclusive,maintenance 2017-10-24 23:23:49,514 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-24 23:23:49,527 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 52 got: state = job-exclusive,maintenance 2017-10-24 23:23:50,029 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-24 23:23:50,039 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 53 got: state = job-exclusive,maintenance 2017-10-24 23:23:50,541 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-24 23:23:50,551 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 54 got: state = job-exclusive,maintenance 2017-10-24 23:23:51,052 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-24 23:23:51,063 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 55 got: state = job-exclusive,maintenance 2017-10-24 23:23:51,564 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-24 23:23:51,575 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 56 got: state = job-exclusive,maintenance 2017-10-24 23:23:52,077 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-24 23:23:52,089 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 57 got: state = job-exclusive,maintenance 2017-10-24 23:23:52,591 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-24 23:23:52,602 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 58 got: state = job-exclusive,maintenance 2017-10-24 23:23:53,104 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-24 23:23:53,116 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 59 got: state = job-exclusive,maintenance 2017-10-24 23:23:53,618 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-24 23:23:53,628 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 60 got: state = job-exclusive,maintenance 2017-10-24 23:23:54,136 INFO FAILED 2017-10-24 23:23:54,137 INFO ========================================= 2017-10-24 23:23:54,137 INFO Entered TestSuspendResumeOnCray tearDown 2017-10-24 23:23:54,137 INFO ========================================= 2017-10-24 23:23:54,137 INFO ========================================== 2017-10-24 23:23:54,138 INFO Completed TestSuspendResumeOnCray tearDown 2017-10-24 23:23:54,138 INFO ========================================== 2017-10-24 23:23:54,138 INFO test name: test_default_restrict_res_to_release_on_suspend_setting (tests.functional.pbs_cray_suspend_resume.TestSuspendResumeOnCray)... 2017-10-24 23:23:54,138 INFO test start time: Tue Oct 24 23:23:54 2017 2017-10-24 23:23:54,139 INFO test docstring: Check that on Cray restrict_res_to_release_on_suspend is always set to 'ncpus' by default 2017-10-24 23:23:54,139 INFO ====================================== 2017-10-24 23:23:54,139 INFO Entered TestSuspendResumeOnCray setUp 2017-10-24 23:23:54,139 INFO ====================================== 2017-10-24 23:23:54,139 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-24 23:23:54,148 INFO status on linux-kt3k: server 2017-10-24 23:23:54,148 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-24 23:23:54,156 INFO manager on linux-kt3k: set server {'managers': (3, 'root@*')} 2017-10-24 23:23:54,157 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c set server managers-=root@* 2017-10-24 23:23:54,191 INFO manager on linux-kt3k: set server {'managers': (2, 'root@*')} 2017-10-24 23:23:54,191 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c set server managers+=root@* 2017-10-24 23:23:54,219 INFO server linux-kt3k: reverting configuration to defaults 2017-10-24 23:23:54,220 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-24 23:23:54,228 INFO select on linux-kt3k: __ALL__ 2017-10-24 23:23:54,228 INFOCLI linux-kt3k: /opt/pbs/bin/qselect 2017-10-24 23:23:54,237 INFO delete job on linux-kt3k: 129.linux-kt3k 2017-10-24 23:23:54,237 INFOCLI linux-kt3k: /opt/pbs/bin/qdel -W force 129.linux-kt3k 2017-10-24 23:23:54,262 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 129.linux-kt3k 2017-10-24 23:23:54,285 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f @linux-kt3k 2017-10-24 23:23:54,294 INFO expect on server linux-kt3k: job_state set 0 job ... OK 2017-10-24 23:23:54,295 INFOCLI linux-kt3k: /opt/pbs/bin/pbs_rstat -f 2017-10-24 23:23:54,306 INFO manager on linux-kt3k: unset server ['pbs_license_min', 'restrict_res_to_release_on_suspend', 'pbs_license_max'] 2017-10-24 23:23:54,306 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c unset server pbs_license_min,restrict_res_to_release_on_suspend,pbs_license_max 2017-10-24 23:23:54,332 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-10-24 23:23:54,372 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Qf @linux-kt3k 2017-10-24 23:23:54,382 INFO status on linux-kt3k: node 2017-10-24 23:23:54,382 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v -a 2017-10-24 23:23:54,390 INFO manager on linux-kt3k: delete queue workq 2017-10-24 23:23:54,391 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c delete queue workq 2017-10-24 23:23:54,400 INFO server linux-kt3k: expect offset set to 0.5 2017-10-24 23:23:54,902 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Qf workq@linux-kt3k 2017-10-24 23:23:54,912 INFO expect on server linux-kt3k: unset queue workq ... OK 2017-10-24 23:23:54,912 INFO manager on linux-kt3k: create queue workq {'started': 'True', 'queue_type': 'Execution', 'enabled': 'True'} 2017-10-24 23:23:54,912 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c create queue workq started=True,queue_type=Execution,enabled=True 2017-10-24 23:23:54,923 INFO server linux-kt3k: expect offset set to 0.5 2017-10-24 23:23:55,424 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Qf workq@linux-kt3k 2017-10-24 23:23:55,435 INFO expect on server linux-kt3k: started set True || queue_type set Execution || enabled set True queue workq ... OK 2017-10-24 23:23:55,435 INFOCLI2 linux-kt3k: sudo -H cmp /opt/pbs/lib/python/altair/pbs_hooks/PBS_translate_mpp.HK /var/spool/pbs/server_priv/hooks/PBS_translate_mpp.HK 2017-10-24 23:23:55,458 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /opt/pbs/lib/python/altair/pbs_hooks/PBS_translate_mpp.HK /var/spool/pbs/server_priv/hooks/PBS_translate_mpp.HK 2017-10-24 23:23:55,478 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/server_priv/hooks/PBS_translate_mpp.HK 2017-10-24 23:23:55,498 INFO server linux-kt3k: sent signal -HUP 2017-10-24 23:23:55,498 INFOCLI2 linux-kt3k: sudo -H kill -HUP 80177 2017-10-24 23:23:55,520 INFO manager on linux-kt3k: set pbshook PBS_translate_mpp {'enabled': 'true'} 2017-10-24 23:23:55,521 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c set pbshook PBS_translate_mpp enabled=true 2017-10-24 23:23:55,549 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-10-24 23:23:55,575 INFO manager on linux-kt3k: set server {'default_queue': 'workq', 'restrict_res_to_release_on_suspend': 'ncpus'} 2017-10-24 23:23:55,575 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server default_queue=workq,restrict_res_to_release_on_suspend=ncpus 2017-10-24 23:23:55,593 INFO status on linux-kt3k: resource 2017-10-24 23:23:55,593 INFO manager on linux-kt3k: list resource 2017-10-24 23:23:55,593 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource 2017-10-24 23:23:55,602 INFO manager on linux-kt3k: delete resource PBScraylabel_G34_IL,PBScrayorder,PBScrayhost,PBScraynid 2017-10-24 23:23:55,603 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c delete resource PBScraylabel_G34_IL,PBScrayorder,PBScrayhost,PBScraynid 2017-10-24 23:23:55,687 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource PBScraylabel_G34_IL 2017-10-24 23:23:55,696 INFO expect on server linux-kt3k: unset resource PBScraylabel_G34_IL ... OK 2017-10-24 23:23:55,696 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource PBScrayorder 2017-10-24 23:23:55,704 INFO expect on server linux-kt3k: unset resource PBScrayorder ... OK 2017-10-24 23:23:55,704 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource PBScrayhost 2017-10-24 23:23:55,713 INFO expect on server linux-kt3k: unset resource PBScrayhost ... OK 2017-10-24 23:23:55,714 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource PBScraynid 2017-10-24 23:23:55,728 INFO expect on server linux-kt3k: unset resource PBScraynid ... OK 2017-10-24 23:23:55,728 INFOCLI status on linux-kt3k: server license_count 2017-10-24 23:23:55,728 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-24 23:23:55,737 INFO server: linux-kt3k licensed 2017-10-24 23:23:55,748 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/server_priv/comm.lock 2017-10-24 23:23:55,781 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-10-24 23:23:55,812 INFO scheduler linux-kt3k: reverting configuration to defaults 2017-10-24 23:23:55,812 INFO manager on linux-kt3k: delete sched @default 2017-10-24 23:23:55,812 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c delete sched @default 2017-10-24 23:23:55,823 INFO manager on linux-kt3k: list sched 2017-10-24 23:23:55,823 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list sched 2017-10-24 23:23:55,834 INFO manager on linux-kt3k: unset sched ['sched_cycle_length'] 2017-10-24 23:23:55,834 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c unset sched sched_cycle_length 2017-10-24 23:23:55,850 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/dedicated_time 2017-10-24 23:23:55,876 INFOCLI2 linux-kt3k: cmp /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-10-24 23:23:55,878 INFO scheduler linux-kt3k: reverting holidays file to default 2017-10-24 23:23:55,878 INFOCLI2 linux-kt3k: cmp /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-10-24 23:23:55,881 INFOCLI2 linux-kt3k: cmp /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-10-24 23:23:55,884 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-10-24 23:23:55,910 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-10-24 23:23:55,930 INFO scheduler linux-kt3k: sent signal -HUP 2017-10-24 23:23:55,931 INFOCLI2 linux-kt3k: sudo -H kill -HUP 96312 2017-10-24 23:23:55,949 INFOCLI2 linux-kt3k: sudo -H /opt/pbs/sbin/pbsfs -e 2017-10-24 23:23:55,999 INFO scheduler linux-kt3k: config {'resources': '"ncpus, mem, arch, host, vnode, aoe, eoe, vntype"'} 2017-10-24 23:23:55,999 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /var/spool/pbs/sched_priv/sched_config /var/spool/pbs/sched_priv/sched_config.bak 2017-10-24 23:23:56,022 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /tmp/PtlPbsfBSvg1 /var/spool/pbs/sched_priv/sched_config 2017-10-24 23:23:56,044 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-10-24 23:23:56,064 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chown root /var/spool/pbs/sched_priv/sched_config 2017-10-24 23:23:56,086 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chgrp root /var/spool/pbs/sched_priv/sched_config 2017-10-24 23:23:56,110 INFO scheduler linux-kt3k: sent signal -HUP 2017-10-24 23:23:56,110 INFOCLI2 linux-kt3k: sudo -H kill -HUP 96312 2017-10-24 23:23:56,132 INFO scheduler linux-kt3k log match: searching for "Error reading line" - No match 2017-10-24 23:23:57,144 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-10-24 23:23:57,175 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-10-24 23:23:57,195 INFO mom linux-kt3k: reverting configuration to defaults 2017-10-24 23:23:57,195 INFOCLI2 linux-kt3k: sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2017-10-24 23:23:57,213 INFOCLI2 linux-kt3k: sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2017-10-24 23:23:57,233 INFOCLI linux-kt3k: sudo -H /opt/pbs/sbin/pbs_mom -s list 2017-10-24 23:23:57,271 INFO mom linux-kt3k: sent signal -HUP 2017-10-24 23:23:57,272 INFOCLI2 linux-kt3k: sudo -H kill -HUP 96290 2017-10-24 23:23:57,303 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-10-24 23:23:57,327 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v -a 2017-10-24 23:23:57,339 INFO manager on linux-kt3k: delete node 2017-10-24 23:23:57,339 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c delete node @default 2017-10-24 23:23:57,355 INFO manager on linux-kt3k: create node linux-kt3k 2017-10-24 23:23:57,355 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c create node linux-kt3k 2017-10-24 23:23:57,367 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k 2017-10-24 23:23:57,389 INFO expect on server linux-kt3k: state = free node linux-kt3k got: state = state-unknown,down 2017-10-24 23:23:58,391 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k 2017-10-24 23:23:58,402 INFO expect on server linux-kt3k: state = free node linux-kt3k attempt: 2 got: state = state-unknown,down 2017-10-24 23:23:59,408 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k 2017-10-24 23:23:59,440 INFO expect on server linux-kt3k: state = free node linux-kt3k attempt: 3 ... OK 2017-10-24 23:23:59,440 INFO ======================================= 2017-10-24 23:23:59,441 INFO Completed TestSuspendResumeOnCray setUp 2017-10-24 23:23:59,441 INFO ======================================= 2017-10-24 23:23:59,441 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-24 23:23:59,449 INFO expect on server linux-kt3k: restrict_res_to_release_on_suspend = ncpus server linux-kt3k ... OK 2017-10-24 23:23:59,449 INFO ========================================= 2017-10-24 23:23:59,450 INFO Entered TestSuspendResumeOnCray tearDown 2017-10-24 23:23:59,450 INFO ========================================= 2017-10-24 23:23:59,450 INFO ========================================== 2017-10-24 23:23:59,450 INFO Completed TestSuspendResumeOnCray tearDown 2017-10-24 23:23:59,450 INFO ========================================== 2017-10-24 23:23:59,450 INFO ok 2017-10-24 23:23:59,450 INFO test name: test_exclusive_job_not_suspended (tests.functional.pbs_cray_suspend_resume.TestSuspendResumeOnCray)... 2017-10-24 23:23:59,450 INFO test start time: Tue Oct 24 23:23:59 2017 2017-10-24 23:23:59,450 INFO test docstring: If a running job is a job with exclusive placement then this job can not be suspended. This test is checking for a log message which is an unstable interface and may need change in future when interface changes. 2017-10-24 23:23:59,450 INFO ====================================== 2017-10-24 23:23:59,450 INFO Entered TestSuspendResumeOnCray setUp 2017-10-24 23:23:59,451 INFO ====================================== 2017-10-24 23:23:59,451 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-24 23:23:59,459 INFO status on linux-kt3k: server 2017-10-24 23:23:59,460 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-24 23:23:59,469 INFO manager on linux-kt3k: set server {'managers': (3, 'root@*')} 2017-10-24 23:23:59,469 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c set server managers-=root@* 2017-10-24 23:23:59,500 INFO manager on linux-kt3k: set server {'managers': (2, 'root@*')} 2017-10-24 23:23:59,500 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c set server managers+=root@* 2017-10-24 23:23:59,528 INFO server linux-kt3k: reverting configuration to defaults 2017-10-24 23:23:59,528 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-24 23:23:59,537 INFO select on linux-kt3k: __ALL__ 2017-10-24 23:23:59,537 INFOCLI linux-kt3k: /opt/pbs/bin/qselect 2017-10-24 23:23:59,545 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f @linux-kt3k 2017-10-24 23:23:59,553 INFO expect on server linux-kt3k: job_state set 0 job ... OK 2017-10-24 23:23:59,553 INFOCLI linux-kt3k: /opt/pbs/bin/pbs_rstat -f 2017-10-24 23:23:59,561 INFO manager on linux-kt3k: unset server ['restrict_res_to_release_on_suspend'] 2017-10-24 23:23:59,561 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c unset server restrict_res_to_release_on_suspend 2017-10-24 23:23:59,571 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-10-24 23:23:59,596 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Qf @linux-kt3k 2017-10-24 23:23:59,606 INFO status on linux-kt3k: node 2017-10-24 23:23:59,606 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v -a 2017-10-24 23:23:59,614 INFO manager on linux-kt3k: delete queue workq 2017-10-24 23:23:59,614 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c delete queue workq 2017-10-24 23:23:59,626 INFO server linux-kt3k: expect offset set to 0.5 2017-10-24 23:24:00,128 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Qf workq@linux-kt3k 2017-10-24 23:24:00,136 INFO expect on server linux-kt3k: unset queue workq ... OK 2017-10-24 23:24:00,137 INFO manager on linux-kt3k: create queue workq {'started': 'True', 'queue_type': 'Execution', 'enabled': 'True'} 2017-10-24 23:24:00,137 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c create queue workq started=True,queue_type=Execution,enabled=True 2017-10-24 23:24:00,146 INFO server linux-kt3k: expect offset set to 0.5 2017-10-24 23:24:00,648 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Qf workq@linux-kt3k 2017-10-24 23:24:00,659 INFO expect on server linux-kt3k: started set True || queue_type set Execution || enabled set True queue workq ... OK 2017-10-24 23:24:00,659 INFOCLI2 linux-kt3k: sudo -H cmp /opt/pbs/lib/python/altair/pbs_hooks/PBS_translate_mpp.HK /var/spool/pbs/server_priv/hooks/PBS_translate_mpp.HK 2017-10-24 23:24:00,678 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /opt/pbs/lib/python/altair/pbs_hooks/PBS_translate_mpp.HK /var/spool/pbs/server_priv/hooks/PBS_translate_mpp.HK 2017-10-24 23:24:00,697 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/server_priv/hooks/PBS_translate_mpp.HK 2017-10-24 23:24:00,717 INFO server linux-kt3k: sent signal -HUP 2017-10-24 23:24:00,717 INFOCLI2 linux-kt3k: sudo -H kill -HUP 80177 2017-10-24 23:24:00,734 INFO manager on linux-kt3k: set pbshook PBS_translate_mpp {'enabled': 'true'} 2017-10-24 23:24:00,735 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c set pbshook PBS_translate_mpp enabled=true 2017-10-24 23:24:00,765 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-10-24 23:24:00,789 INFO manager on linux-kt3k: set server {'default_queue': 'workq', 'restrict_res_to_release_on_suspend': 'ncpus'} 2017-10-24 23:24:00,790 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server default_queue=workq,restrict_res_to_release_on_suspend=ncpus 2017-10-24 23:24:00,800 INFO status on linux-kt3k: resource 2017-10-24 23:24:00,800 INFO manager on linux-kt3k: list resource 2017-10-24 23:24:00,801 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource 2017-10-24 23:24:00,810 INFO manager on linux-kt3k: delete resource PBScraylabel_G34_IL,PBScrayorder,PBScrayhost,PBScraynid 2017-10-24 23:24:00,810 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c delete resource PBScraylabel_G34_IL,PBScrayorder,PBScrayhost,PBScraynid 2017-10-24 23:24:00,891 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource PBScraylabel_G34_IL 2017-10-24 23:24:00,903 INFO expect on server linux-kt3k: unset resource PBScraylabel_G34_IL ... OK 2017-10-24 23:24:00,903 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource PBScrayorder 2017-10-24 23:24:00,913 INFO expect on server linux-kt3k: unset resource PBScrayorder ... OK 2017-10-24 23:24:00,914 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource PBScrayhost 2017-10-24 23:24:00,922 INFO expect on server linux-kt3k: unset resource PBScrayhost ... OK 2017-10-24 23:24:00,923 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource PBScraynid 2017-10-24 23:24:00,935 INFO expect on server linux-kt3k: unset resource PBScraynid ... OK 2017-10-24 23:24:00,935 INFOCLI status on linux-kt3k: server license_count 2017-10-24 23:24:00,936 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-24 23:24:00,946 INFO server: linux-kt3k licensed 2017-10-24 23:24:00,954 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/server_priv/comm.lock 2017-10-24 23:24:00,989 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-10-24 23:24:01,016 INFO scheduler linux-kt3k: reverting configuration to defaults 2017-10-24 23:24:01,016 INFO manager on linux-kt3k: delete sched @default 2017-10-24 23:24:01,016 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c delete sched @default 2017-10-24 23:24:01,026 INFO manager on linux-kt3k: list sched 2017-10-24 23:24:01,026 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list sched 2017-10-24 23:24:01,034 INFO manager on linux-kt3k: unset sched ['sched_cycle_length'] 2017-10-24 23:24:01,034 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c unset sched sched_cycle_length 2017-10-24 23:24:01,045 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/dedicated_time 2017-10-24 23:24:01,076 INFOCLI2 linux-kt3k: cmp /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-10-24 23:24:01,078 INFO scheduler linux-kt3k: reverting holidays file to default 2017-10-24 23:24:01,079 INFOCLI2 linux-kt3k: cmp /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-10-24 23:24:01,081 INFOCLI2 linux-kt3k: cmp /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-10-24 23:24:01,083 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-10-24 23:24:01,108 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-10-24 23:24:01,134 INFO scheduler linux-kt3k: sent signal -HUP 2017-10-24 23:24:01,134 INFOCLI2 linux-kt3k: sudo -H kill -HUP 96312 2017-10-24 23:24:01,155 INFOCLI2 linux-kt3k: sudo -H /opt/pbs/sbin/pbsfs -e 2017-10-24 23:24:01,199 INFO scheduler linux-kt3k: config {'resources': '"ncpus, mem, arch, host, vnode, aoe, eoe, vntype"'} 2017-10-24 23:24:01,200 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /var/spool/pbs/sched_priv/sched_config /var/spool/pbs/sched_priv/sched_config.bak 2017-10-24 23:24:01,225 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /tmp/PtlPbsWdXS42 /var/spool/pbs/sched_priv/sched_config 2017-10-24 23:24:01,252 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-10-24 23:24:01,273 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chown root /var/spool/pbs/sched_priv/sched_config 2017-10-24 23:24:01,295 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chgrp root /var/spool/pbs/sched_priv/sched_config 2017-10-24 23:24:01,315 INFO scheduler linux-kt3k: sent signal -HUP 2017-10-24 23:24:01,315 INFOCLI2 linux-kt3k: sudo -H kill -HUP 96312 2017-10-24 23:24:01,336 INFO scheduler linux-kt3k log match: searching for "Error reading line" - No match 2017-10-24 23:24:02,348 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-10-24 23:24:02,375 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-10-24 23:24:02,396 INFO mom linux-kt3k: reverting configuration to defaults 2017-10-24 23:24:02,397 INFOCLI2 linux-kt3k: sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2017-10-24 23:24:02,416 INFOCLI2 linux-kt3k: sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2017-10-24 23:24:02,436 INFOCLI linux-kt3k: sudo -H /opt/pbs/sbin/pbs_mom -s list 2017-10-24 23:24:02,474 INFO mom linux-kt3k: sent signal -HUP 2017-10-24 23:24:02,474 INFOCLI2 linux-kt3k: sudo -H kill -HUP 96290 2017-10-24 23:24:02,504 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-10-24 23:24:02,530 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v -a 2017-10-24 23:24:02,542 INFO manager on linux-kt3k: delete node 2017-10-24 23:24:02,542 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c delete node @default 2017-10-24 23:24:02,560 INFO manager on linux-kt3k: create node linux-kt3k 2017-10-24 23:24:02,560 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c create node linux-kt3k 2017-10-24 23:24:02,580 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k 2017-10-24 23:24:02,601 INFO expect on server linux-kt3k: state = free node linux-kt3k got: state = state-unknown,down 2017-10-24 23:24:03,602 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k 2017-10-24 23:24:03,612 INFO expect on server linux-kt3k: state = free node linux-kt3k attempt: 2 got: state = state-unknown,down 2017-10-24 23:24:04,620 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k 2017-10-24 23:24:04,652 INFO expect on server linux-kt3k: state = free node linux-kt3k attempt: 3 ... OK 2017-10-24 23:24:04,652 INFO ======================================= 2017-10-24 23:24:04,652 INFO Completed TestSuspendResumeOnCray setUp 2017-10-24 23:24:04,652 INFO ======================================= 2017-10-24 23:24:04,652 INFO job: executable set to /bin/sleep with arguments: 100 2017-10-24 23:24:04,652 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0755 /tmp/PtlPbsJobScriptwKvws9 2017-10-24 23:24:04,673 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chown pbsuser /tmp/PtlPbsJobScriptwKvws9 2017-10-24 23:24:04,696 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chgrp tstgrp00 /tmp/PtlPbsJobScriptwKvws9 2017-10-24 23:24:04,717 INFOCLI linux-kt3k: sudo -H -u pbsuser /opt/pbs/bin/qsub -l place=excl -l select=1:ncpus=1:vntype=cray_compute /tmp/PtlPbsJobScriptwKvws9 2017-10-24 23:24:04,750 INFO submit to linux-kt3k as pbsuser: job 130.linux-kt3k OrderedDict([('Resource_List.place', 'excl'), ('Resource_List.select', '1:ncpus=1:vntype=cray_compute')]) 2017-10-24 23:24:04,750 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 130.linux-kt3k 2017-10-24 23:24:04,760 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 130.linux-kt3k got: job_state = Q 2017-10-24 23:24:05,261 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:24:05,276 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 130.linux-kt3k 2017-10-24 23:24:05,288 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 130.linux-kt3k attempt: 2 ... OK 2017-10-24 23:24:05,288 INFO signal on linux-kt3k: 130.linux-kt3k with signal = suspend 2017-10-24 23:24:05,288 INFOCLI linux-kt3k: /opt/pbs/bin/qsig -s suspend 130.linux-kt3k 2017-10-24 23:24:05,350 ERROR err: ['qsig: Switching ALPS reservation failed 130.linux-kt3k'] 2017-10-24 23:24:05,350 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 130.linux-kt3k 2017-10-24 23:24:05,360 INFO expect on server linux-kt3k: exec_host set job 130.linux-kt3k ... OK 2017-10-24 23:24:05,360 INFO status on linux-kt3k: job 130.linux-kt3k 2017-10-24 23:24:05,360 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 130.linux-kt3k 2017-10-24 23:24:05,371 INFO mom linux-kt3k log match: searching for "BASIL;ERROR: ALPS error: apsched: at least resid .* is exclusive" - using regular expression ... OK 2017-10-24 23:24:05,371 INFO ========================================= 2017-10-24 23:24:05,371 INFO Entered TestSuspendResumeOnCray tearDown 2017-10-24 23:24:05,371 INFO ========================================= 2017-10-24 23:24:05,371 INFO ========================================== 2017-10-24 23:24:05,371 INFO Completed TestSuspendResumeOnCray tearDown 2017-10-24 23:24:05,371 INFO ========================================== 2017-10-24 23:24:05,372 INFO ok 2017-10-24 23:24:05,372 INFO test name: test_multi_express (tests.functional.pbs_cray_suspend_resume.TestSuspendResumeOnCray)... 2017-10-24 23:24:05,372 INFO test start time: Tue Oct 24 23:24:05 2017 2017-10-24 23:24:05,373 INFO test docstring: Test of multiple express queues of different priorities. See that jobs from the higher express queues preempt jobs from lower express queues. Also see when express jobs finish (or are deleted), suspended jobs restart. Make sure loadLimit is set to 4 on the server node: # apmgr config loadLimit 4 2017-10-24 23:24:05,373 INFO ====================================== 2017-10-24 23:24:05,373 INFO Entered TestSuspendResumeOnCray setUp 2017-10-24 23:24:05,373 INFO ====================================== 2017-10-24 23:24:05,373 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-24 23:24:05,382 INFO status on linux-kt3k: server 2017-10-24 23:24:05,382 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-24 23:24:05,393 INFO manager on linux-kt3k: set server {'managers': (3, 'root@*')} 2017-10-24 23:24:05,394 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c set server managers-=root@* 2017-10-24 23:24:05,422 INFO manager on linux-kt3k: set server {'managers': (2, 'root@*')} 2017-10-24 23:24:05,422 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c set server managers+=root@* 2017-10-24 23:24:05,450 INFO server linux-kt3k: reverting configuration to defaults 2017-10-24 23:24:05,450 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-24 23:24:05,459 INFO select on linux-kt3k: __ALL__ 2017-10-24 23:24:05,460 INFOCLI linux-kt3k: /opt/pbs/bin/qselect 2017-10-24 23:24:05,467 INFO delete job on linux-kt3k: 130.linux-kt3k 2017-10-24 23:24:05,467 INFOCLI linux-kt3k: /opt/pbs/bin/qdel -W force 130.linux-kt3k 2017-10-24 23:24:05,484 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 130.linux-kt3k 2017-10-24 23:24:05,508 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f @linux-kt3k 2017-10-24 23:24:05,519 INFO expect on server linux-kt3k: job_state set 0 job ... OK 2017-10-24 23:24:05,519 INFOCLI linux-kt3k: /opt/pbs/bin/pbs_rstat -f 2017-10-24 23:24:05,530 INFO manager on linux-kt3k: unset server ['restrict_res_to_release_on_suspend'] 2017-10-24 23:24:05,531 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c unset server restrict_res_to_release_on_suspend 2017-10-24 23:24:05,550 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-10-24 23:24:05,596 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Qf @linux-kt3k 2017-10-24 23:24:05,606 INFO status on linux-kt3k: node 2017-10-24 23:24:05,607 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v -a 2017-10-24 23:24:05,618 INFO manager on linux-kt3k: delete queue workq 2017-10-24 23:24:05,618 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c delete queue workq 2017-10-24 23:24:05,629 INFO server linux-kt3k: expect offset set to 0.5 2017-10-24 23:24:06,130 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Qf workq@linux-kt3k 2017-10-24 23:24:06,142 INFO expect on server linux-kt3k: unset queue workq ... OK 2017-10-24 23:24:06,142 INFO manager on linux-kt3k: create queue workq {'started': 'True', 'queue_type': 'Execution', 'enabled': 'True'} 2017-10-24 23:24:06,142 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c create queue workq started=True,queue_type=Execution,enabled=True 2017-10-24 23:24:06,152 INFO server linux-kt3k: expect offset set to 0.5 2017-10-24 23:24:06,654 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Qf workq@linux-kt3k 2017-10-24 23:24:06,665 INFO expect on server linux-kt3k: started set True || queue_type set Execution || enabled set True queue workq ... OK 2017-10-24 23:24:06,666 INFOCLI2 linux-kt3k: sudo -H cmp /opt/pbs/lib/python/altair/pbs_hooks/PBS_translate_mpp.HK /var/spool/pbs/server_priv/hooks/PBS_translate_mpp.HK 2017-10-24 23:24:06,685 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /opt/pbs/lib/python/altair/pbs_hooks/PBS_translate_mpp.HK /var/spool/pbs/server_priv/hooks/PBS_translate_mpp.HK 2017-10-24 23:24:06,706 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/server_priv/hooks/PBS_translate_mpp.HK 2017-10-24 23:24:06,734 INFO server linux-kt3k: sent signal -HUP 2017-10-24 23:24:06,734 INFOCLI2 linux-kt3k: sudo -H kill -HUP 80177 2017-10-24 23:24:06,752 INFO manager on linux-kt3k: set pbshook PBS_translate_mpp {'enabled': 'true'} 2017-10-24 23:24:06,753 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c set pbshook PBS_translate_mpp enabled=true 2017-10-24 23:24:06,780 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-10-24 23:24:06,807 INFO manager on linux-kt3k: set server {'default_queue': 'workq', 'restrict_res_to_release_on_suspend': 'ncpus'} 2017-10-24 23:24:06,807 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server default_queue=workq,restrict_res_to_release_on_suspend=ncpus 2017-10-24 23:24:06,821 INFO status on linux-kt3k: resource 2017-10-24 23:24:06,821 INFO manager on linux-kt3k: list resource 2017-10-24 23:24:06,821 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource 2017-10-24 23:24:06,835 INFO manager on linux-kt3k: delete resource PBScraylabel_G34_IL,PBScrayorder,PBScrayhost,PBScraynid 2017-10-24 23:24:06,835 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c delete resource PBScraylabel_G34_IL,PBScrayorder,PBScrayhost,PBScraynid 2017-10-24 23:24:06,915 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource PBScraylabel_G34_IL 2017-10-24 23:24:06,924 INFO expect on server linux-kt3k: unset resource PBScraylabel_G34_IL ... OK 2017-10-24 23:24:06,924 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource PBScrayorder 2017-10-24 23:24:06,933 INFO expect on server linux-kt3k: unset resource PBScrayorder ... OK 2017-10-24 23:24:06,933 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource PBScrayhost 2017-10-24 23:24:06,941 INFO expect on server linux-kt3k: unset resource PBScrayhost ... OK 2017-10-24 23:24:06,941 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource PBScraynid 2017-10-24 23:24:06,950 INFO expect on server linux-kt3k: unset resource PBScraynid ... OK 2017-10-24 23:24:06,950 INFOCLI status on linux-kt3k: server license_count 2017-10-24 23:24:06,950 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-24 23:24:06,960 INFO server: linux-kt3k licensed 2017-10-24 23:24:06,969 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/server_priv/comm.lock 2017-10-24 23:24:06,998 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-10-24 23:24:07,015 INFO scheduler linux-kt3k: reverting configuration to defaults 2017-10-24 23:24:07,016 INFO manager on linux-kt3k: delete sched @default 2017-10-24 23:24:07,016 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c delete sched @default 2017-10-24 23:24:07,025 INFO manager on linux-kt3k: list sched 2017-10-24 23:24:07,025 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list sched 2017-10-24 23:24:07,034 INFO manager on linux-kt3k: unset sched ['sched_cycle_length'] 2017-10-24 23:24:07,034 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c unset sched sched_cycle_length 2017-10-24 23:24:07,043 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/dedicated_time 2017-10-24 23:24:07,063 INFOCLI2 linux-kt3k: cmp /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-10-24 23:24:07,065 INFO scheduler linux-kt3k: reverting holidays file to default 2017-10-24 23:24:07,066 INFOCLI2 linux-kt3k: cmp /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-10-24 23:24:07,068 INFOCLI2 linux-kt3k: cmp /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-10-24 23:24:07,070 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-10-24 23:24:07,089 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-10-24 23:24:07,109 INFO scheduler linux-kt3k: sent signal -HUP 2017-10-24 23:24:07,110 INFOCLI2 linux-kt3k: sudo -H kill -HUP 96312 2017-10-24 23:24:07,129 INFOCLI2 linux-kt3k: sudo -H /opt/pbs/sbin/pbsfs -e 2017-10-24 23:24:07,173 INFO scheduler linux-kt3k: config {'resources': '"ncpus, mem, arch, host, vnode, aoe, eoe, vntype"'} 2017-10-24 23:24:07,174 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /var/spool/pbs/sched_priv/sched_config /var/spool/pbs/sched_priv/sched_config.bak 2017-10-24 23:24:07,193 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /tmp/PtlPbseb0uS1 /var/spool/pbs/sched_priv/sched_config 2017-10-24 23:24:07,213 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-10-24 23:24:07,233 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chown root /var/spool/pbs/sched_priv/sched_config 2017-10-24 23:24:07,254 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chgrp root /var/spool/pbs/sched_priv/sched_config 2017-10-24 23:24:07,279 INFO scheduler linux-kt3k: sent signal -HUP 2017-10-24 23:24:07,281 INFOCLI2 linux-kt3k: sudo -H kill -HUP 96312 2017-10-24 23:24:07,301 INFO scheduler linux-kt3k log match: searching for "Error reading line" - No match 2017-10-24 23:24:08,316 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-10-24 23:24:08,346 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-10-24 23:24:08,368 INFO mom linux-kt3k: reverting configuration to defaults 2017-10-24 23:24:08,369 INFOCLI2 linux-kt3k: sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2017-10-24 23:24:08,389 INFOCLI2 linux-kt3k: sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2017-10-24 23:24:08,411 INFOCLI linux-kt3k: sudo -H /opt/pbs/sbin/pbs_mom -s list 2017-10-24 23:24:08,450 INFO mom linux-kt3k: sent signal -HUP 2017-10-24 23:24:08,450 INFOCLI2 linux-kt3k: sudo -H kill -HUP 96290 2017-10-24 23:24:08,483 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-10-24 23:24:08,510 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v -a 2017-10-24 23:24:08,524 INFO manager on linux-kt3k: delete node 2017-10-24 23:24:08,524 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c delete node @default 2017-10-24 23:24:08,539 INFO manager on linux-kt3k: create node linux-kt3k 2017-10-24 23:24:08,539 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c create node linux-kt3k 2017-10-24 23:24:08,562 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k 2017-10-24 23:24:08,584 INFO expect on server linux-kt3k: state = free node linux-kt3k got: state = state-unknown,down 2017-10-24 23:24:09,586 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k 2017-10-24 23:24:09,599 INFO expect on server linux-kt3k: state = free node linux-kt3k attempt: 2 got: state = state-unknown,down 2017-10-24 23:24:10,608 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k 2017-10-24 23:24:10,639 INFO expect on server linux-kt3k: state = free node linux-kt3k attempt: 3 ... OK 2017-10-24 23:24:10,639 INFO ======================================= 2017-10-24 23:24:10,639 INFO Completed TestSuspendResumeOnCray setUp 2017-10-24 23:24:10,639 INFO ======================================= 2017-10-24 23:24:10,639 INFO scheduler linux-kt3k: config {'preempt_prio': '"express_queue, normal_jobs, server_softlimits, queue_softlimits"'} 2017-10-24 23:24:10,640 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /var/spool/pbs/sched_priv/sched_config /var/spool/pbs/sched_priv/sched_config.bak 2017-10-24 23:24:10,662 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /tmp/PtlPbsehcJLh /var/spool/pbs/sched_priv/sched_config 2017-10-24 23:24:10,682 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-10-24 23:24:10,699 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chown root /var/spool/pbs/sched_priv/sched_config 2017-10-24 23:24:10,718 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chgrp root /var/spool/pbs/sched_priv/sched_config 2017-10-24 23:24:10,737 INFO scheduler linux-kt3k: sent signal -HUP 2017-10-24 23:24:10,737 INFOCLI2 linux-kt3k: sudo -H kill -HUP 96312 2017-10-24 23:24:10,758 INFO scheduler linux-kt3k log match: searching for "Error reading line" - No match 2017-10-24 23:24:11,760 INFO manager on linux-kt3k: create queue expressq {'started': 'True', 'Priority': 150, 'queue_type': 'e', 'enabled': 'True'} 2017-10-24 23:24:11,760 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c create queue expressq started=True,Priority=150,queue_type=e,enabled=True 2017-10-24 23:24:11,773 INFO manager on linux-kt3k: create queue expressq2 {'started': 'True', 'Priority': 160, 'queue_type': 'e', 'enabled': 'True'} 2017-10-24 23:24:11,773 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c create queue expressq2 started=True,Priority=160,queue_type=e,enabled=True 2017-10-24 23:24:11,784 INFO manager on linux-kt3k: create queue expressq3 {'started': 'True', 'Priority': 170, 'queue_type': 'e', 'enabled': 'True'} 2017-10-24 23:24:11,784 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c create queue expressq3 started=True,Priority=170,queue_type=e,enabled=True 2017-10-24 23:24:11,797 INFO filter: node {'resources_available.vntype': 'cray_compute'} 2017-10-24 23:24:11,797 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v -a 2017-10-24 23:24:11,806 INFO job: executable set to /bin/sleep with arguments: 100 2017-10-24 23:24:11,807 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0755 /tmp/PtlPbsJobScriptLfeeNS 2017-10-24 23:24:11,828 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chown pbsuser /tmp/PtlPbsJobScriptLfeeNS 2017-10-24 23:24:11,848 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chgrp tstgrp00 /tmp/PtlPbsJobScriptLfeeNS 2017-10-24 23:24:11,866 INFOCLI linux-kt3k: sudo -H -u pbsuser /opt/pbs/bin/qsub -l place=scatter -l walltime=3600 -l select=2:ncpus=1:vntype=cray_compute /tmp/PtlPbsJobScriptLfeeNS 2017-10-24 23:24:11,892 INFO submit to linux-kt3k as pbsuser: job 131.linux-kt3k OrderedDict([('Resource_List.place', 'scatter'), ('Resource_List.walltime', 3600), ('Resource_List.select', '2:ncpus=1:vntype=cray_compute')]) 2017-10-24 23:24:11,893 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 131.linux-kt3k 2017-10-24 23:24:11,901 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 131.linux-kt3k got: job_state = Q 2017-10-24 23:24:12,402 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:24:12,416 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 131.linux-kt3k 2017-10-24 23:24:12,424 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 131.linux-kt3k attempt: 2 ... OK 2017-10-24 23:24:12,425 INFO job: executable set to /bin/sleep with arguments: 100 2017-10-24 23:24:12,425 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0755 /tmp/PtlPbsJobScriptLmvc_E 2017-10-24 23:24:12,446 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chown pbsuser /tmp/PtlPbsJobScriptLmvc_E 2017-10-24 23:24:12,467 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chgrp tstgrp00 /tmp/PtlPbsJobScriptLmvc_E 2017-10-24 23:24:12,485 INFOCLI linux-kt3k: sudo -H -u pbsuser /opt/pbs/bin/qsub -q expressq -l place=scatter -l walltime=3600 -l select=2:ncpus=1:vntype=cray_compute /tmp/PtlPbsJobScriptLmvc_E 2017-10-24 23:24:12,507 INFO submit to linux-kt3k as pbsuser: job 132.linux-kt3k OrderedDict([('destination', 'expressq'), ('Resource_List.place', 'scatter'), ('Resource_List.walltime', 3600), ('Resource_List.select', '2:ncpus=1:vntype=cray_compute')]) 2017-10-24 23:24:12,507 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 131.linux-kt3k 2017-10-24 23:24:12,517 INFO expect on server linux-kt3k: job_state = S job 131.linux-kt3k got: job_state = R 2017-10-24 23:24:13,018 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:24:13,031 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 131.linux-kt3k 2017-10-24 23:24:13,039 INFO expect on server linux-kt3k: job_state = S job 131.linux-kt3k attempt: 2 got: job_state = R 2017-10-24 23:24:13,541 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:24:13,552 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 131.linux-kt3k 2017-10-24 23:24:13,561 INFO expect on server linux-kt3k: job_state = S job 131.linux-kt3k attempt: 3 got: job_state = R 2017-10-24 23:24:14,062 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:24:14,075 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 131.linux-kt3k 2017-10-24 23:24:14,083 INFO expect on server linux-kt3k: job_state = S job 131.linux-kt3k attempt: 4 got: job_state = R 2017-10-24 23:24:14,584 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:24:14,597 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 131.linux-kt3k 2017-10-24 23:24:14,606 INFO expect on server linux-kt3k: job_state = S job 131.linux-kt3k attempt: 5 got: job_state = R 2017-10-24 23:24:15,107 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:24:15,119 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 131.linux-kt3k 2017-10-24 23:24:15,129 INFO expect on server linux-kt3k: job_state = S job 131.linux-kt3k attempt: 6 got: job_state = R 2017-10-24 23:24:15,631 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:24:15,642 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 131.linux-kt3k 2017-10-24 23:24:15,652 INFO expect on server linux-kt3k: job_state = S job 131.linux-kt3k attempt: 7 got: job_state = R 2017-10-24 23:24:16,153 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:24:16,166 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 131.linux-kt3k 2017-10-24 23:24:16,176 INFO expect on server linux-kt3k: job_state = S job 131.linux-kt3k attempt: 8 got: job_state = R 2017-10-24 23:24:16,677 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:24:16,689 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 131.linux-kt3k 2017-10-24 23:24:16,698 INFO expect on server linux-kt3k: job_state = S job 131.linux-kt3k attempt: 9 got: job_state = R 2017-10-24 23:24:17,199 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:24:17,211 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 131.linux-kt3k 2017-10-24 23:24:17,222 INFO expect on server linux-kt3k: job_state = S job 131.linux-kt3k attempt: 10 got: job_state = R 2017-10-24 23:24:17,724 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:24:17,736 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 131.linux-kt3k 2017-10-24 23:24:17,744 INFO expect on server linux-kt3k: job_state = S job 131.linux-kt3k attempt: 11 got: job_state = R 2017-10-24 23:24:18,246 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:24:18,259 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 131.linux-kt3k 2017-10-24 23:24:18,268 INFO expect on server linux-kt3k: job_state = S job 131.linux-kt3k attempt: 12 got: job_state = R 2017-10-24 23:24:18,769 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:24:18,780 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 131.linux-kt3k 2017-10-24 23:24:18,789 INFO expect on server linux-kt3k: job_state = S job 131.linux-kt3k attempt: 13 got: job_state = R 2017-10-24 23:24:19,291 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:24:19,302 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 131.linux-kt3k 2017-10-24 23:24:19,312 INFO expect on server linux-kt3k: job_state = S job 131.linux-kt3k attempt: 14 got: job_state = R 2017-10-24 23:24:19,814 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:24:19,826 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 131.linux-kt3k 2017-10-24 23:24:19,834 INFO expect on server linux-kt3k: job_state = S job 131.linux-kt3k attempt: 15 got: job_state = R 2017-10-24 23:24:20,336 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:24:20,348 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 131.linux-kt3k 2017-10-24 23:24:20,357 INFO expect on server linux-kt3k: job_state = S job 131.linux-kt3k attempt: 16 got: job_state = R 2017-10-24 23:24:20,858 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:24:20,868 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 131.linux-kt3k 2017-10-24 23:24:20,877 INFO expect on server linux-kt3k: job_state = S job 131.linux-kt3k attempt: 17 got: job_state = R 2017-10-24 23:24:21,378 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:24:21,388 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 131.linux-kt3k 2017-10-24 23:24:21,398 INFO expect on server linux-kt3k: job_state = S job 131.linux-kt3k attempt: 18 got: job_state = R 2017-10-24 23:24:21,899 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:24:21,911 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 131.linux-kt3k 2017-10-24 23:24:21,921 INFO expect on server linux-kt3k: job_state = S job 131.linux-kt3k attempt: 19 got: job_state = R 2017-10-24 23:24:22,423 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:24:22,436 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 131.linux-kt3k 2017-10-24 23:24:22,445 INFO expect on server linux-kt3k: job_state = S job 131.linux-kt3k attempt: 20 ... OK 2017-10-24 23:24:22,446 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 132.linux-kt3k 2017-10-24 23:24:22,455 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 132.linux-kt3k ... OK 2017-10-24 23:24:22,455 INFO job: executable set to /bin/sleep with arguments: 100 2017-10-24 23:24:22,456 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0755 /tmp/PtlPbsJobScriptHib9bP 2017-10-24 23:24:22,478 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chown pbsuser /tmp/PtlPbsJobScriptHib9bP 2017-10-24 23:24:22,500 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chgrp tstgrp00 /tmp/PtlPbsJobScriptHib9bP 2017-10-24 23:24:22,519 INFOCLI linux-kt3k: sudo -H -u pbsuser /opt/pbs/bin/qsub -q expressq2 -l place=scatter -l walltime=3600 -l select=2:ncpus=1:vntype=cray_compute /tmp/PtlPbsJobScriptHib9bP 2017-10-24 23:24:22,545 INFO submit to linux-kt3k as pbsuser: job 133.linux-kt3k OrderedDict([('destination', 'expressq2'), ('Resource_List.place', 'scatter'), ('Resource_List.walltime', 3600), ('Resource_List.select', '2:ncpus=1:vntype=cray_compute')]) 2017-10-24 23:24:22,546 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 132.linux-kt3k 2017-10-24 23:24:22,554 INFO expect on server linux-kt3k: job_state = S job 132.linux-kt3k got: job_state = R 2017-10-24 23:24:23,055 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:24:23,067 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 132.linux-kt3k 2017-10-24 23:24:23,079 INFO expect on server linux-kt3k: job_state = S job 132.linux-kt3k attempt: 2 got: job_state = R 2017-10-24 23:24:23,580 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:24:23,593 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 132.linux-kt3k 2017-10-24 23:24:23,602 INFO expect on server linux-kt3k: job_state = S job 132.linux-kt3k attempt: 3 got: job_state = R 2017-10-24 23:24:24,104 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:24:24,119 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 132.linux-kt3k 2017-10-24 23:24:24,130 INFO expect on server linux-kt3k: job_state = S job 132.linux-kt3k attempt: 4 got: job_state = R 2017-10-24 23:24:24,632 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:24:24,645 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 132.linux-kt3k 2017-10-24 23:24:24,655 INFO expect on server linux-kt3k: job_state = S job 132.linux-kt3k attempt: 5 got: job_state = R 2017-10-24 23:24:25,156 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:24:25,168 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 132.linux-kt3k 2017-10-24 23:24:25,177 INFO expect on server linux-kt3k: job_state = S job 132.linux-kt3k attempt: 6 got: job_state = R 2017-10-24 23:24:25,678 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:24:25,688 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 132.linux-kt3k 2017-10-24 23:24:25,697 INFO expect on server linux-kt3k: job_state = S job 132.linux-kt3k attempt: 7 got: job_state = R 2017-10-24 23:24:26,199 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:24:26,212 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 132.linux-kt3k 2017-10-24 23:24:26,222 INFO expect on server linux-kt3k: job_state = S job 132.linux-kt3k attempt: 8 got: job_state = R 2017-10-24 23:24:26,724 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:24:26,735 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 132.linux-kt3k 2017-10-24 23:24:26,745 INFO expect on server linux-kt3k: job_state = S job 132.linux-kt3k attempt: 9 got: job_state = R 2017-10-24 23:24:27,246 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:24:27,259 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 132.linux-kt3k 2017-10-24 23:24:27,268 INFO expect on server linux-kt3k: job_state = S job 132.linux-kt3k attempt: 10 got: job_state = R 2017-10-24 23:24:27,769 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:24:27,781 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 132.linux-kt3k 2017-10-24 23:24:27,790 INFO expect on server linux-kt3k: job_state = S job 132.linux-kt3k attempt: 11 got: job_state = R 2017-10-24 23:24:28,291 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:24:28,303 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 132.linux-kt3k 2017-10-24 23:24:28,317 INFO expect on server linux-kt3k: job_state = S job 132.linux-kt3k attempt: 12 got: job_state = R 2017-10-24 23:24:28,818 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:24:28,829 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 132.linux-kt3k 2017-10-24 23:24:28,841 INFO expect on server linux-kt3k: job_state = S job 132.linux-kt3k attempt: 13 got: job_state = R 2017-10-24 23:24:29,343 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:24:29,357 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 132.linux-kt3k 2017-10-24 23:24:29,366 INFO expect on server linux-kt3k: job_state = S job 132.linux-kt3k attempt: 14 got: job_state = R 2017-10-24 23:24:29,867 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:24:29,878 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 132.linux-kt3k 2017-10-24 23:24:29,886 INFO expect on server linux-kt3k: job_state = S job 132.linux-kt3k attempt: 15 got: job_state = R 2017-10-24 23:24:30,387 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:24:30,399 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 132.linux-kt3k 2017-10-24 23:24:30,411 INFO expect on server linux-kt3k: job_state = S job 132.linux-kt3k attempt: 16 got: job_state = R 2017-10-24 23:24:30,913 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:24:30,924 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 132.linux-kt3k 2017-10-24 23:24:30,932 INFO expect on server linux-kt3k: job_state = S job 132.linux-kt3k attempt: 17 got: job_state = R 2017-10-24 23:24:31,434 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:24:31,448 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 132.linux-kt3k 2017-10-24 23:24:31,456 INFO expect on server linux-kt3k: job_state = S job 132.linux-kt3k attempt: 18 got: job_state = R 2017-10-24 23:24:31,957 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:24:31,968 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 132.linux-kt3k 2017-10-24 23:24:31,977 INFO expect on server linux-kt3k: job_state = S job 132.linux-kt3k attempt: 19 got: job_state = R 2017-10-24 23:24:32,478 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:24:32,490 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 132.linux-kt3k 2017-10-24 23:24:32,499 INFO expect on server linux-kt3k: job_state = S job 132.linux-kt3k attempt: 20 ... OK 2017-10-24 23:24:32,499 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 133.linux-kt3k 2017-10-24 23:24:32,509 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 133.linux-kt3k ... OK 2017-10-24 23:24:32,509 INFO job: executable set to /bin/sleep with arguments: 100 2017-10-24 23:24:32,510 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0755 /tmp/PtlPbsJobScriptDfU_23 2017-10-24 23:24:32,531 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chown pbsuser /tmp/PtlPbsJobScriptDfU_23 2017-10-24 23:24:32,556 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chgrp tstgrp00 /tmp/PtlPbsJobScriptDfU_23 2017-10-24 23:24:32,574 INFOCLI linux-kt3k: sudo -H -u pbsuser /opt/pbs/bin/qsub -q expressq3 -l place=scatter -l walltime=3600 -l select=2:ncpus=1:vntype=cray_compute /tmp/PtlPbsJobScriptDfU_23 2017-10-24 23:24:32,597 INFO submit to linux-kt3k as pbsuser: job 134.linux-kt3k OrderedDict([('destination', 'expressq3'), ('Resource_List.place', 'scatter'), ('Resource_List.walltime', 3600), ('Resource_List.select', '2:ncpus=1:vntype=cray_compute')]) 2017-10-24 23:24:32,598 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 133.linux-kt3k 2017-10-24 23:24:32,607 INFO expect on server linux-kt3k: job_state = S job 133.linux-kt3k got: job_state = R 2017-10-24 23:24:33,108 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:24:33,119 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 133.linux-kt3k 2017-10-24 23:24:33,128 INFO expect on server linux-kt3k: job_state = S job 133.linux-kt3k attempt: 2 got: job_state = R 2017-10-24 23:24:33,630 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:24:33,640 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 133.linux-kt3k 2017-10-24 23:24:33,648 INFO expect on server linux-kt3k: job_state = S job 133.linux-kt3k attempt: 3 got: job_state = R 2017-10-24 23:24:34,150 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:24:34,164 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 133.linux-kt3k 2017-10-24 23:24:34,173 INFO expect on server linux-kt3k: job_state = S job 133.linux-kt3k attempt: 4 got: job_state = R 2017-10-24 23:24:34,674 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:24:34,685 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 133.linux-kt3k 2017-10-24 23:24:34,695 INFO expect on server linux-kt3k: job_state = S job 133.linux-kt3k attempt: 5 got: job_state = R 2017-10-24 23:24:35,195 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:24:35,209 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 133.linux-kt3k 2017-10-24 23:24:35,221 INFO expect on server linux-kt3k: job_state = S job 133.linux-kt3k attempt: 6 got: job_state = R 2017-10-24 23:24:35,723 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:24:35,734 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 133.linux-kt3k 2017-10-24 23:24:35,742 INFO expect on server linux-kt3k: job_state = S job 133.linux-kt3k attempt: 7 got: job_state = R 2017-10-24 23:24:36,243 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:24:36,255 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 133.linux-kt3k 2017-10-24 23:24:36,263 INFO expect on server linux-kt3k: job_state = S job 133.linux-kt3k attempt: 8 got: job_state = R 2017-10-24 23:24:36,764 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:24:36,775 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 133.linux-kt3k 2017-10-24 23:24:36,783 INFO expect on server linux-kt3k: job_state = S job 133.linux-kt3k attempt: 9 got: job_state = R 2017-10-24 23:24:37,285 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:24:37,297 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 133.linux-kt3k 2017-10-24 23:24:37,307 INFO expect on server linux-kt3k: job_state = S job 133.linux-kt3k attempt: 10 got: job_state = R 2017-10-24 23:24:37,809 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:24:37,820 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 133.linux-kt3k 2017-10-24 23:24:37,828 INFO expect on server linux-kt3k: job_state = S job 133.linux-kt3k attempt: 11 got: job_state = R 2017-10-24 23:24:38,329 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:24:38,344 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 133.linux-kt3k 2017-10-24 23:24:38,353 INFO expect on server linux-kt3k: job_state = S job 133.linux-kt3k attempt: 12 got: job_state = R 2017-10-24 23:24:38,854 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:24:38,865 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 133.linux-kt3k 2017-10-24 23:24:38,873 INFO expect on server linux-kt3k: job_state = S job 133.linux-kt3k attempt: 13 got: job_state = R 2017-10-24 23:24:39,374 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:24:39,387 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 133.linux-kt3k 2017-10-24 23:24:39,396 INFO expect on server linux-kt3k: job_state = S job 133.linux-kt3k attempt: 14 got: job_state = R 2017-10-24 23:24:39,898 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:24:39,910 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 133.linux-kt3k 2017-10-24 23:24:39,918 INFO expect on server linux-kt3k: job_state = S job 133.linux-kt3k attempt: 15 got: job_state = R 2017-10-24 23:24:40,420 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:24:40,431 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 133.linux-kt3k 2017-10-24 23:24:40,440 INFO expect on server linux-kt3k: job_state = S job 133.linux-kt3k attempt: 16 got: job_state = R 2017-10-24 23:24:40,942 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:24:40,961 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 133.linux-kt3k 2017-10-24 23:24:40,971 INFO expect on server linux-kt3k: job_state = S job 133.linux-kt3k attempt: 17 got: job_state = R 2017-10-24 23:24:41,472 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:24:41,485 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 133.linux-kt3k 2017-10-24 23:24:41,493 INFO expect on server linux-kt3k: job_state = S job 133.linux-kt3k attempt: 18 got: job_state = R 2017-10-24 23:24:41,994 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:24:42,007 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 133.linux-kt3k 2017-10-24 23:24:42,015 INFO expect on server linux-kt3k: job_state = S job 133.linux-kt3k attempt: 19 got: job_state = R 2017-10-24 23:24:42,517 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:24:42,529 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 133.linux-kt3k 2017-10-24 23:24:42,538 INFO expect on server linux-kt3k: job_state = S job 133.linux-kt3k attempt: 20 ... OK 2017-10-24 23:24:42,538 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 134.linux-kt3k 2017-10-24 23:24:42,547 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 134.linux-kt3k ... OK 2017-10-24 23:24:42,547 INFO delete on linux-kt3k: 134.linux-kt3k 2017-10-24 23:24:42,547 INFO delete job on linux-kt3k: 134.linux-kt3k 2017-10-24 23:24:42,547 INFOCLI linux-kt3k: /opt/pbs/bin/qdel 134.linux-kt3k 2017-10-24 23:24:42,564 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 133.linux-kt3k 2017-10-24 23:24:42,588 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 133.linux-kt3k got: job_state = S 2017-10-24 23:24:43,090 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:24:43,101 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 133.linux-kt3k 2017-10-24 23:24:43,110 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 133.linux-kt3k attempt: 2 ... OK 2017-10-24 23:24:43,110 INFO ========================================= 2017-10-24 23:24:43,110 INFO Entered TestSuspendResumeOnCray tearDown 2017-10-24 23:24:43,110 INFO ========================================= 2017-10-24 23:24:43,110 INFO ========================================== 2017-10-24 23:24:43,110 INFO Completed TestSuspendResumeOnCray tearDown 2017-10-24 23:24:43,110 INFO ========================================== 2017-10-24 23:24:43,110 INFO ok 2017-10-24 23:24:43,111 INFO test name: test_preempt_STF (tests.functional.pbs_cray_suspend_resume.TestSuspendResumeOnCray)... 2017-10-24 23:24:43,111 INFO test start time: Tue Oct 24 23:24:43 2017 2017-10-24 23:24:43,111 INFO test docstring: Test shrink to fit by creating a reservation for all compute nodes starting in 100 sec. with a duration of two hours. A preempted STF job with min_walltime of 1 min. and max_walltime of 2 hours will stay suspended after higher priority job goes away if its min_walltime can't be satisfied. 2017-10-24 23:24:43,111 INFO ====================================== 2017-10-24 23:24:43,111 INFO Entered TestSuspendResumeOnCray setUp 2017-10-24 23:24:43,111 INFO ====================================== 2017-10-24 23:24:43,111 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-24 23:24:43,120 INFO status on linux-kt3k: server 2017-10-24 23:24:43,120 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-24 23:24:43,129 INFO manager on linux-kt3k: set server {'managers': (3, 'root@*')} 2017-10-24 23:24:43,129 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c set server managers-=root@* 2017-10-24 23:24:43,159 INFO manager on linux-kt3k: set server {'managers': (2, 'root@*')} 2017-10-24 23:24:43,159 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c set server managers+=root@* 2017-10-24 23:24:43,184 INFO server linux-kt3k: reverting configuration to defaults 2017-10-24 23:24:43,185 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-24 23:24:43,193 INFO select on linux-kt3k: __ALL__ 2017-10-24 23:24:43,193 INFOCLI linux-kt3k: /opt/pbs/bin/qselect 2017-10-24 23:24:43,201 INFO delete job on linux-kt3k: 131.linux-kt3k, 132.linux-kt3k, 133.linux-kt3k 2017-10-24 23:24:43,201 INFOCLI linux-kt3k: /opt/pbs/bin/qdel -W force 131.linux-kt3k 132.linux-kt3k 133.linux-kt3k 2017-10-24 23:24:43,263 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 131.linux-kt3k 2017-10-24 23:24:43,295 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 132.linux-kt3k 2017-10-24 23:24:43,309 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 133.linux-kt3k 2017-10-24 23:24:43,321 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f @linux-kt3k 2017-10-24 23:24:43,335 INFO expect on server linux-kt3k: job_state set 0 job ... OK 2017-10-24 23:24:43,335 INFOCLI linux-kt3k: /opt/pbs/bin/pbs_rstat -f 2017-10-24 23:24:43,347 INFO manager on linux-kt3k: unset server ['restrict_res_to_release_on_suspend'] 2017-10-24 23:24:43,347 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c unset server restrict_res_to_release_on_suspend 2017-10-24 23:24:43,370 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-10-24 23:24:43,406 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Qf @linux-kt3k 2017-10-24 23:24:43,418 INFO status on linux-kt3k: node 2017-10-24 23:24:43,418 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v -a 2017-10-24 23:24:43,437 INFO manager on linux-kt3k: delete queue workq,expressq,expressq2,expressq3 2017-10-24 23:24:43,437 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c delete queue workq,expressq,expressq2,expressq3 2017-10-24 23:24:43,450 INFO server linux-kt3k: expect offset set to 0.5 2017-10-24 23:24:43,952 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Qf workq@linux-kt3k 2017-10-24 23:24:43,962 INFO expect on server linux-kt3k: unset queue workq ... OK 2017-10-24 23:24:43,962 INFO server linux-kt3k: expect offset set to 0.5 2017-10-24 23:24:44,463 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Qf expressq@linux-kt3k 2017-10-24 23:24:44,472 INFO expect on server linux-kt3k: unset queue expressq ... OK 2017-10-24 23:24:44,472 INFO server linux-kt3k: expect offset set to 0.5 2017-10-24 23:24:44,973 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Qf expressq2@linux-kt3k 2017-10-24 23:24:44,983 INFO expect on server linux-kt3k: unset queue expressq2 ... OK 2017-10-24 23:24:44,983 INFO server linux-kt3k: expect offset set to 0.5 2017-10-24 23:24:45,485 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Qf expressq3@linux-kt3k 2017-10-24 23:24:45,496 INFO expect on server linux-kt3k: unset queue expressq3 ... OK 2017-10-24 23:24:45,496 INFO manager on linux-kt3k: create queue workq {'started': 'True', 'queue_type': 'Execution', 'enabled': 'True'} 2017-10-24 23:24:45,496 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c create queue workq started=True,queue_type=Execution,enabled=True 2017-10-24 23:24:45,506 INFO server linux-kt3k: expect offset set to 0.5 2017-10-24 23:24:46,007 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Qf workq@linux-kt3k 2017-10-24 23:24:46,017 INFO expect on server linux-kt3k: started set True || queue_type set Execution || enabled set True queue workq ... OK 2017-10-24 23:24:46,017 INFOCLI2 linux-kt3k: sudo -H cmp /opt/pbs/lib/python/altair/pbs_hooks/PBS_translate_mpp.HK /var/spool/pbs/server_priv/hooks/PBS_translate_mpp.HK 2017-10-24 23:24:46,035 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /opt/pbs/lib/python/altair/pbs_hooks/PBS_translate_mpp.HK /var/spool/pbs/server_priv/hooks/PBS_translate_mpp.HK 2017-10-24 23:24:46,056 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/server_priv/hooks/PBS_translate_mpp.HK 2017-10-24 23:24:46,074 INFO server linux-kt3k: sent signal -HUP 2017-10-24 23:24:46,074 INFOCLI2 linux-kt3k: sudo -H kill -HUP 80177 2017-10-24 23:24:46,094 INFO manager on linux-kt3k: set pbshook PBS_translate_mpp {'enabled': 'true'} 2017-10-24 23:24:46,094 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c set pbshook PBS_translate_mpp enabled=true 2017-10-24 23:24:46,121 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-10-24 23:24:46,145 INFO manager on linux-kt3k: set server {'default_queue': 'workq', 'restrict_res_to_release_on_suspend': 'ncpus'} 2017-10-24 23:24:46,146 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server default_queue=workq,restrict_res_to_release_on_suspend=ncpus 2017-10-24 23:24:46,156 INFO status on linux-kt3k: resource 2017-10-24 23:24:46,156 INFO manager on linux-kt3k: list resource 2017-10-24 23:24:46,157 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource 2017-10-24 23:24:46,166 INFO manager on linux-kt3k: delete resource PBScraylabel_G34_IL,PBScrayorder,PBScrayhost,PBScraynid 2017-10-24 23:24:46,166 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c delete resource PBScraylabel_G34_IL,PBScrayorder,PBScrayhost,PBScraynid 2017-10-24 23:24:46,243 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource PBScraylabel_G34_IL 2017-10-24 23:24:46,253 INFO expect on server linux-kt3k: unset resource PBScraylabel_G34_IL ... OK 2017-10-24 23:24:46,253 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource PBScrayorder 2017-10-24 23:24:46,261 INFO expect on server linux-kt3k: unset resource PBScrayorder ... OK 2017-10-24 23:24:46,261 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource PBScrayhost 2017-10-24 23:24:46,270 INFO expect on server linux-kt3k: unset resource PBScrayhost ... OK 2017-10-24 23:24:46,270 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource PBScraynid 2017-10-24 23:24:46,279 INFO expect on server linux-kt3k: unset resource PBScraynid ... OK 2017-10-24 23:24:46,279 INFOCLI status on linux-kt3k: server license_count 2017-10-24 23:24:46,279 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-24 23:24:46,287 INFO server: linux-kt3k licensed 2017-10-24 23:24:46,296 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/server_priv/comm.lock 2017-10-24 23:24:46,322 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-10-24 23:24:46,339 INFO scheduler linux-kt3k: reverting configuration to defaults 2017-10-24 23:24:46,340 INFO manager on linux-kt3k: delete sched @default 2017-10-24 23:24:46,340 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c delete sched @default 2017-10-24 23:24:46,349 INFO manager on linux-kt3k: list sched 2017-10-24 23:24:46,349 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list sched 2017-10-24 23:24:46,357 INFO manager on linux-kt3k: unset sched ['sched_cycle_length'] 2017-10-24 23:24:46,357 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c unset sched sched_cycle_length 2017-10-24 23:24:46,366 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/dedicated_time 2017-10-24 23:24:46,385 INFOCLI2 linux-kt3k: cmp /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-10-24 23:24:46,387 INFO scheduler linux-kt3k: reverting holidays file to default 2017-10-24 23:24:46,387 INFOCLI2 linux-kt3k: cmp /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-10-24 23:24:46,389 INFOCLI2 linux-kt3k: cmp /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-10-24 23:24:46,391 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-10-24 23:24:46,415 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-10-24 23:24:46,434 INFO scheduler linux-kt3k: sent signal -HUP 2017-10-24 23:24:46,434 INFOCLI2 linux-kt3k: sudo -H kill -HUP 96312 2017-10-24 23:24:46,458 INFOCLI2 linux-kt3k: sudo -H /opt/pbs/sbin/pbsfs -e 2017-10-24 23:24:46,507 INFO scheduler linux-kt3k: config {'resources': '"ncpus, mem, arch, host, vnode, aoe, eoe, vntype"'} 2017-10-24 23:24:46,508 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /var/spool/pbs/sched_priv/sched_config /var/spool/pbs/sched_priv/sched_config.bak 2017-10-24 23:24:46,529 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /tmp/PtlPbs4bwlmk /var/spool/pbs/sched_priv/sched_config 2017-10-24 23:24:46,546 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-10-24 23:24:46,564 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chown root /var/spool/pbs/sched_priv/sched_config 2017-10-24 23:24:46,582 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chgrp root /var/spool/pbs/sched_priv/sched_config 2017-10-24 23:24:46,602 INFO scheduler linux-kt3k: sent signal -HUP 2017-10-24 23:24:46,602 INFOCLI2 linux-kt3k: sudo -H kill -HUP 96312 2017-10-24 23:24:46,626 INFO scheduler linux-kt3k log match: searching for "Error reading line" - No match 2017-10-24 23:24:47,638 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-10-24 23:24:47,665 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-10-24 23:24:47,684 INFO mom linux-kt3k: reverting configuration to defaults 2017-10-24 23:24:47,684 INFOCLI2 linux-kt3k: sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2017-10-24 23:24:47,702 INFOCLI2 linux-kt3k: sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2017-10-24 23:24:47,721 INFOCLI linux-kt3k: sudo -H /opt/pbs/sbin/pbs_mom -s list 2017-10-24 23:24:47,758 INFO mom linux-kt3k: sent signal -HUP 2017-10-24 23:24:47,758 INFOCLI2 linux-kt3k: sudo -H kill -HUP 96290 2017-10-24 23:24:47,786 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-10-24 23:24:47,812 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v -a 2017-10-24 23:24:47,828 INFO manager on linux-kt3k: delete node 2017-10-24 23:24:47,829 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c delete node @default 2017-10-24 23:24:47,844 INFO manager on linux-kt3k: create node linux-kt3k 2017-10-24 23:24:47,845 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c create node linux-kt3k 2017-10-24 23:24:47,863 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k 2017-10-24 23:24:47,881 INFO expect on server linux-kt3k: state = free node linux-kt3k got: state = state-unknown,down 2017-10-24 23:24:48,883 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k 2017-10-24 23:24:48,895 INFO expect on server linux-kt3k: state = free node linux-kt3k attempt: 2 got: state = state-unknown,down 2017-10-24 23:24:49,896 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k 2017-10-24 23:24:49,936 INFO expect on server linux-kt3k: state = free node linux-kt3k attempt: 3 ... OK 2017-10-24 23:24:49,937 INFO ======================================= 2017-10-24 23:24:49,937 INFO Completed TestSuspendResumeOnCray setUp 2017-10-24 23:24:49,937 INFO ======================================= 2017-10-24 23:24:49,937 INFO manager on linux-kt3k: create queue highp {'queue_type': 'execution'} 2017-10-24 23:24:49,937 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c create queue highp queue_type=execution 2017-10-24 23:24:49,948 INFO manager on linux-kt3k: set queue highp {'started': 'True', 'priority': '150', 'enabled': 'True'} 2017-10-24 23:24:49,948 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set queue highp started=True,priority=150,enabled=True 2017-10-24 23:24:49,959 INFO filter: node {'resources_available.vntype': 'cray_compute'} 2017-10-24 23:24:49,960 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v -a 2017-10-24 23:24:49,968 INFOCLI linux-kt3k: sudo -H -u pbsuser /opt/pbs/bin/pbs_rsub -R 201710242326.29 -l place=scatter -D 7200 -l select=2:ncpus=1:vntype=cray_compute 2017-10-24 23:24:50,003 INFO submit to linux-kt3k as pbsuser: reservation R135.linux-kt3k UNCONFIRMED OrderedDict([('reserve_start', 1508867789), ('Resource_List.place', 'scatter'), ('reserve_duration', 7200), ('Resource_List.select', '2:ncpus=1:vntype=cray_compute')]) 2017-10-24 23:24:50,003 INFOCLI linux-kt3k: /opt/pbs/bin/pbs_rstat -f R135.linux-kt3k 2017-10-24 23:24:50,014 INFO expect on server linux-kt3k: reserve_state ~ RESV_CONFIRMED|2 reservation R135.linux-kt3k got: reserve_state = RESV_UNCONFIRMED 2017-10-24 23:24:50,516 INFOCLI linux-kt3k: /opt/pbs/bin/pbs_rstat -f R135.linux-kt3k 2017-10-24 23:24:50,529 INFO expect on server linux-kt3k: reserve_state ~ RESV_CONFIRMED|2 reservation R135.linux-kt3k attempt: 2 ... OK 2017-10-24 23:24:50,530 INFO job: executable set to /bin/sleep with arguments: 100 2017-10-24 23:24:50,530 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0755 /tmp/PtlPbsJobScriptBatGjP 2017-10-24 23:24:50,552 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chown pbsuser /tmp/PtlPbsJobScriptBatGjP 2017-10-24 23:24:50,577 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chgrp tstgrp00 /tmp/PtlPbsJobScriptBatGjP 2017-10-24 23:24:50,595 INFOCLI linux-kt3k: sudo -H -u pbsuser /opt/pbs/bin/qsub -l min_walltime=00:01:00 -l place=scatter -l max_walltime=02:00:00 -l select=2:ncpus=1:vntype=cray_compute /tmp/PtlPbsJobScriptBatGjP 2017-10-24 23:24:50,619 INFO submit to linux-kt3k as pbsuser: job 136.linux-kt3k OrderedDict([('Resource_List.min_walltime', '00:01:00'), ('Resource_List.place', 'scatter'), ('Resource_List.max_walltime', '02:00:00'), ('Resource_List.select', '2:ncpus=1:vntype=cray_compute')]) 2017-10-24 23:24:50,619 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 136.linux-kt3k 2017-10-24 23:24:50,629 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 136.linux-kt3k got: job_state = Q 2017-10-24 23:24:51,130 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:24:51,146 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 136.linux-kt3k 2017-10-24 23:24:51,156 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 136.linux-kt3k attempt: 2 ... OK 2017-10-24 23:24:51,157 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 136.linux-kt3k 2017-10-24 23:24:51,166 INFO expect on server linux-kt3k: Resource_List.walltime <= 00:01:40 job 136.linux-kt3k ... OK 2017-10-24 23:24:51,167 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 136.linux-kt3k 2017-10-24 23:24:51,177 INFO expect on server linux-kt3k: Resource_List.walltime >= 00:01:00 job 136.linux-kt3k ... OK 2017-10-24 23:25:26,214 INFO job: executable set to /bin/sleep with arguments: 100 2017-10-24 23:25:26,215 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0755 /tmp/PtlPbsJobScript4Hst5h 2017-10-24 23:25:26,237 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chown pbsuser /tmp/PtlPbsJobScript4Hst5h 2017-10-24 23:25:26,257 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chgrp tstgrp00 /tmp/PtlPbsJobScript4Hst5h 2017-10-24 23:25:26,279 INFOCLI linux-kt3k: sudo -H -u pbsuser /opt/pbs/bin/qsub -q highp -l place=scatter -l walltime=00:01:00 -l select=2:ncpus=1:vntype=cray_compute /tmp/PtlPbsJobScript4Hst5h 2017-10-24 23:25:26,305 INFO submit to linux-kt3k as pbsuser: job 137.linux-kt3k OrderedDict([('destination', 'highp'), ('Resource_List.place', 'scatter'), ('Resource_List.walltime', '00:01:00'), ('Resource_List.select', '2:ncpus=1:vntype=cray_compute')]) 2017-10-24 23:25:26,305 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 137.linux-kt3k 2017-10-24 23:25:26,314 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 137.linux-kt3k got: job_state = Q 2017-10-24 23:25:26,815 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:25:26,828 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 137.linux-kt3k 2017-10-24 23:25:26,835 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 137.linux-kt3k attempt: 2 got: job_state = Q 2017-10-24 23:25:27,336 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:25:27,348 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 137.linux-kt3k 2017-10-24 23:25:27,356 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 137.linux-kt3k attempt: 3 got: job_state = Q 2017-10-24 23:25:27,858 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:25:27,870 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 137.linux-kt3k 2017-10-24 23:25:27,877 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 137.linux-kt3k attempt: 4 got: job_state = Q 2017-10-24 23:25:28,380 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:25:28,392 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 137.linux-kt3k 2017-10-24 23:25:28,401 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 137.linux-kt3k attempt: 5 got: job_state = Q 2017-10-24 23:25:28,902 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:25:28,913 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 137.linux-kt3k 2017-10-24 23:25:28,921 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 137.linux-kt3k attempt: 6 got: job_state = Q 2017-10-24 23:25:29,422 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:25:29,434 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 137.linux-kt3k 2017-10-24 23:25:29,442 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 137.linux-kt3k attempt: 7 got: job_state = Q 2017-10-24 23:25:29,944 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:25:29,957 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 137.linux-kt3k 2017-10-24 23:25:29,966 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 137.linux-kt3k attempt: 8 got: job_state = Q 2017-10-24 23:25:30,468 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:25:30,480 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 137.linux-kt3k 2017-10-24 23:25:30,489 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 137.linux-kt3k attempt: 9 got: job_state = Q 2017-10-24 23:25:30,990 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:25:31,002 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 137.linux-kt3k 2017-10-24 23:25:31,010 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 137.linux-kt3k attempt: 10 got: job_state = Q 2017-10-24 23:25:31,512 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:25:31,525 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 137.linux-kt3k 2017-10-24 23:25:31,535 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 137.linux-kt3k attempt: 11 got: job_state = Q 2017-10-24 23:25:32,036 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:25:32,047 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 137.linux-kt3k 2017-10-24 23:25:32,057 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 137.linux-kt3k attempt: 12 got: job_state = Q 2017-10-24 23:25:32,558 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:25:32,571 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 137.linux-kt3k 2017-10-24 23:25:32,580 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 137.linux-kt3k attempt: 13 got: job_state = Q 2017-10-24 23:25:33,081 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:25:33,092 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 137.linux-kt3k 2017-10-24 23:25:33,100 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 137.linux-kt3k attempt: 14 got: job_state = Q 2017-10-24 23:25:33,601 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:25:33,614 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 137.linux-kt3k 2017-10-24 23:25:33,624 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 137.linux-kt3k attempt: 15 got: job_state = Q 2017-10-24 23:25:34,126 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:25:34,139 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 137.linux-kt3k 2017-10-24 23:25:34,150 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 137.linux-kt3k attempt: 16 got: job_state = Q 2017-10-24 23:25:34,652 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:25:34,665 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 137.linux-kt3k 2017-10-24 23:25:34,673 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 137.linux-kt3k attempt: 17 got: job_state = Q 2017-10-24 23:25:35,174 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:25:35,186 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 137.linux-kt3k 2017-10-24 23:25:35,195 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 137.linux-kt3k attempt: 18 got: job_state = Q 2017-10-24 23:25:35,696 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:25:35,709 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 137.linux-kt3k 2017-10-24 23:25:35,719 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 137.linux-kt3k attempt: 19 got: job_state = Q 2017-10-24 23:25:36,221 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:25:36,232 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 137.linux-kt3k 2017-10-24 23:25:36,241 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 137.linux-kt3k attempt: 20 ... OK 2017-10-24 23:25:36,241 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 136.linux-kt3k 2017-10-24 23:25:36,253 INFO expect on server linux-kt3k: job_state = S job 136.linux-kt3k ... OK 2017-10-24 23:26:26,280 INFO delete on linux-kt3k: 137.linux-kt3k 2017-10-24 23:26:26,281 INFO delete job on linux-kt3k: 137.linux-kt3k 2017-10-24 23:26:26,281 INFOCLI linux-kt3k: /opt/pbs/bin/qdel 137.linux-kt3k 2017-10-24 23:26:26,304 INFO manager on linux-kt3k: set server {'scheduling': 'True'} 2017-10-24 23:26:26,304 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:26:26,333 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-24 23:26:26,354 INFO expect on server linux-kt3k: server_state = Active server linux-kt3k got: server_state = Scheduling 2017-10-24 23:26:26,855 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-24 23:26:26,868 INFO expect on server linux-kt3k: server_state = Active server linux-kt3k attempt: 2 ... OK 2017-10-24 23:26:26,869 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 136.linux-kt3k 2017-10-24 23:26:26,877 INFO expect on server linux-kt3k: job_state = S job 136.linux-kt3k ... OK 2017-10-24 23:26:26,877 INFO ========================================= 2017-10-24 23:26:26,878 INFO Entered TestSuspendResumeOnCray tearDown 2017-10-24 23:26:26,878 INFO ========================================= 2017-10-24 23:26:26,878 INFO ========================================== 2017-10-24 23:26:26,878 INFO Completed TestSuspendResumeOnCray tearDown 2017-10-24 23:26:26,878 INFO ========================================== 2017-10-24 23:26:26,878 INFO ok 2017-10-24 23:26:26,879 INFO test name: test_preempted_topjob_calendared (tests.functional.pbs_cray_suspend_resume.TestSuspendResumeOnCray)... 2017-10-24 23:26:26,879 INFO test start time: Tue Oct 24 23:26:26 2017 2017-10-24 23:26:26,879 INFO test docstring: That even if topjob_ineligible is set for a preempted job and sched_preempt_enforce_resumption is set true, the preempted job will be calendared 2017-10-24 23:26:26,879 INFO ====================================== 2017-10-24 23:26:26,879 INFO Entered TestSuspendResumeOnCray setUp 2017-10-24 23:26:26,879 INFO ====================================== 2017-10-24 23:26:26,879 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-24 23:26:26,889 INFO status on linux-kt3k: server 2017-10-24 23:26:26,889 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-24 23:26:26,898 INFO manager on linux-kt3k: set server {'managers': (3, 'root@*')} 2017-10-24 23:26:26,898 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c set server managers-=root@* 2017-10-24 23:26:26,931 INFO manager on linux-kt3k: set server {'managers': (2, 'root@*')} 2017-10-24 23:26:26,931 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c set server managers+=root@* 2017-10-24 23:26:26,961 INFO server linux-kt3k: reverting configuration to defaults 2017-10-24 23:26:26,961 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-24 23:26:26,971 INFO select on linux-kt3k: __ALL__ 2017-10-24 23:26:26,971 INFOCLI linux-kt3k: /opt/pbs/bin/qselect 2017-10-24 23:26:26,980 INFO delete job on linux-kt3k: 136.linux-kt3k 2017-10-24 23:26:26,980 INFOCLI linux-kt3k: /opt/pbs/bin/qdel -W force 136.linux-kt3k 2017-10-24 23:26:27,003 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 136.linux-kt3k 2017-10-24 23:26:27,026 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f @linux-kt3k 2017-10-24 23:26:27,038 INFO expect on server linux-kt3k: job_state set 0 job ... OK 2017-10-24 23:26:27,038 INFOCLI linux-kt3k: /opt/pbs/bin/pbs_rstat -f 2017-10-24 23:26:27,050 INFO delete resv on linux-kt3k: R135.linux-kt3k 2017-10-24 23:26:27,051 INFOCLI linux-kt3k: /opt/pbs/bin/pbs_rdel R135.linux-kt3k 2017-10-24 23:26:27,096 INFOCLI linux-kt3k: /opt/pbs/bin/pbs_rstat -f 2017-10-24 23:26:27,119 INFO manager on linux-kt3k: unset server ['restrict_res_to_release_on_suspend'] 2017-10-24 23:26:27,119 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c unset server restrict_res_to_release_on_suspend 2017-10-24 23:26:27,130 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-10-24 23:26:27,156 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Qf @linux-kt3k 2017-10-24 23:26:27,164 INFO status on linux-kt3k: node 2017-10-24 23:26:27,164 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v -a 2017-10-24 23:26:27,173 INFO manager on linux-kt3k: delete queue workq,highp 2017-10-24 23:26:27,174 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c delete queue workq,highp 2017-10-24 23:26:27,185 INFO server linux-kt3k: expect offset set to 0.5 2017-10-24 23:26:27,687 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Qf workq@linux-kt3k 2017-10-24 23:26:27,698 INFO expect on server linux-kt3k: unset queue workq ... OK 2017-10-24 23:26:27,698 INFO server linux-kt3k: expect offset set to 0.5 2017-10-24 23:26:28,200 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Qf highp@linux-kt3k 2017-10-24 23:26:28,211 INFO expect on server linux-kt3k: unset queue highp ... OK 2017-10-24 23:26:28,211 INFO manager on linux-kt3k: create queue workq {'started': 'True', 'queue_type': 'Execution', 'enabled': 'True'} 2017-10-24 23:26:28,211 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c create queue workq started=True,queue_type=Execution,enabled=True 2017-10-24 23:26:28,223 INFO server linux-kt3k: expect offset set to 0.5 2017-10-24 23:26:28,725 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Qf workq@linux-kt3k 2017-10-24 23:26:28,740 INFO expect on server linux-kt3k: started set True || queue_type set Execution || enabled set True queue workq ... OK 2017-10-24 23:26:28,740 INFOCLI2 linux-kt3k: sudo -H cmp /opt/pbs/lib/python/altair/pbs_hooks/PBS_translate_mpp.HK /var/spool/pbs/server_priv/hooks/PBS_translate_mpp.HK 2017-10-24 23:26:28,761 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /opt/pbs/lib/python/altair/pbs_hooks/PBS_translate_mpp.HK /var/spool/pbs/server_priv/hooks/PBS_translate_mpp.HK 2017-10-24 23:26:28,782 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/server_priv/hooks/PBS_translate_mpp.HK 2017-10-24 23:26:28,805 INFO server linux-kt3k: sent signal -HUP 2017-10-24 23:26:28,806 INFOCLI2 linux-kt3k: sudo -H kill -HUP 80177 2017-10-24 23:26:28,828 INFO manager on linux-kt3k: set pbshook PBS_translate_mpp {'enabled': 'true'} 2017-10-24 23:26:28,828 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c set pbshook PBS_translate_mpp enabled=true 2017-10-24 23:26:28,857 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-10-24 23:26:28,882 INFO manager on linux-kt3k: set server {'default_queue': 'workq', 'restrict_res_to_release_on_suspend': 'ncpus'} 2017-10-24 23:26:28,883 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server default_queue=workq,restrict_res_to_release_on_suspend=ncpus 2017-10-24 23:26:28,897 INFO status on linux-kt3k: resource 2017-10-24 23:26:28,897 INFO manager on linux-kt3k: list resource 2017-10-24 23:26:28,897 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource 2017-10-24 23:26:28,907 INFO manager on linux-kt3k: delete resource PBScraylabel_G34_IL,PBScrayorder,PBScrayhost,PBScraynid 2017-10-24 23:26:28,907 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c delete resource PBScraylabel_G34_IL,PBScrayorder,PBScrayhost,PBScraynid 2017-10-24 23:26:28,988 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource PBScraylabel_G34_IL 2017-10-24 23:26:29,001 INFO expect on server linux-kt3k: unset resource PBScraylabel_G34_IL ... OK 2017-10-24 23:26:29,001 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource PBScrayorder 2017-10-24 23:26:29,010 INFO expect on server linux-kt3k: unset resource PBScrayorder ... OK 2017-10-24 23:26:29,011 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource PBScrayhost 2017-10-24 23:26:29,019 INFO expect on server linux-kt3k: unset resource PBScrayhost ... OK 2017-10-24 23:26:29,020 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource PBScraynid 2017-10-24 23:26:29,028 INFO expect on server linux-kt3k: unset resource PBScraynid ... OK 2017-10-24 23:26:29,028 INFOCLI status on linux-kt3k: server license_count 2017-10-24 23:26:29,028 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-24 23:26:29,037 INFO server: linux-kt3k licensed 2017-10-24 23:26:29,046 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/server_priv/comm.lock 2017-10-24 23:26:29,074 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-10-24 23:26:29,092 INFO scheduler linux-kt3k: reverting configuration to defaults 2017-10-24 23:26:29,092 INFO manager on linux-kt3k: delete sched @default 2017-10-24 23:26:29,092 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c delete sched @default 2017-10-24 23:26:29,101 INFO manager on linux-kt3k: list sched 2017-10-24 23:26:29,102 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list sched 2017-10-24 23:26:29,110 INFO manager on linux-kt3k: unset sched ['sched_cycle_length'] 2017-10-24 23:26:29,111 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c unset sched sched_cycle_length 2017-10-24 23:26:29,120 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/dedicated_time 2017-10-24 23:26:29,144 INFOCLI2 linux-kt3k: cmp /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-10-24 23:26:29,146 INFO scheduler linux-kt3k: reverting holidays file to default 2017-10-24 23:26:29,146 INFOCLI2 linux-kt3k: cmp /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-10-24 23:26:29,148 INFOCLI2 linux-kt3k: cmp /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-10-24 23:26:29,150 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-10-24 23:26:29,169 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-10-24 23:26:29,187 INFO scheduler linux-kt3k: sent signal -HUP 2017-10-24 23:26:29,188 INFOCLI2 linux-kt3k: sudo -H kill -HUP 96312 2017-10-24 23:26:29,208 INFOCLI2 linux-kt3k: sudo -H /opt/pbs/sbin/pbsfs -e 2017-10-24 23:26:29,250 INFO scheduler linux-kt3k: config {'resources': '"ncpus, mem, arch, host, vnode, aoe, eoe, vntype"'} 2017-10-24 23:26:29,251 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /var/spool/pbs/sched_priv/sched_config /var/spool/pbs/sched_priv/sched_config.bak 2017-10-24 23:26:29,272 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /tmp/PtlPbsIzuv3S /var/spool/pbs/sched_priv/sched_config 2017-10-24 23:26:29,293 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-10-24 23:26:29,313 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chown root /var/spool/pbs/sched_priv/sched_config 2017-10-24 23:26:29,337 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chgrp root /var/spool/pbs/sched_priv/sched_config 2017-10-24 23:26:29,358 INFO scheduler linux-kt3k: sent signal -HUP 2017-10-24 23:26:29,358 INFOCLI2 linux-kt3k: sudo -H kill -HUP 96312 2017-10-24 23:26:29,378 INFO scheduler linux-kt3k log match: searching for "Error reading line" - No match 2017-10-24 23:26:30,390 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-10-24 23:26:30,420 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-10-24 23:26:30,439 INFO mom linux-kt3k: reverting configuration to defaults 2017-10-24 23:26:30,440 INFOCLI2 linux-kt3k: sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2017-10-24 23:26:30,464 INFOCLI2 linux-kt3k: sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2017-10-24 23:26:30,483 INFOCLI linux-kt3k: sudo -H /opt/pbs/sbin/pbs_mom -s list 2017-10-24 23:26:30,521 INFO mom linux-kt3k: sent signal -HUP 2017-10-24 23:26:30,522 INFOCLI2 linux-kt3k: sudo -H kill -HUP 96290 2017-10-24 23:26:30,558 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-10-24 23:26:30,589 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v -a 2017-10-24 23:26:30,598 INFO manager on linux-kt3k: delete node 2017-10-24 23:26:30,599 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c delete node @default 2017-10-24 23:26:30,616 INFO manager on linux-kt3k: create node linux-kt3k 2017-10-24 23:26:30,616 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c create node linux-kt3k 2017-10-24 23:26:30,646 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k 2017-10-24 23:26:30,672 INFO expect on server linux-kt3k: state = free node linux-kt3k got: state = state-unknown,down 2017-10-24 23:26:31,673 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k 2017-10-24 23:26:31,683 INFO expect on server linux-kt3k: state = free node linux-kt3k attempt: 2 got: state = state-unknown,down 2017-10-24 23:26:32,688 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k 2017-10-24 23:26:32,724 INFO expect on server linux-kt3k: state = free node linux-kt3k attempt: 3 ... OK 2017-10-24 23:26:32,724 INFO ======================================= 2017-10-24 23:26:32,724 INFO Completed TestSuspendResumeOnCray setUp 2017-10-24 23:26:32,724 INFO ======================================= 2017-10-24 23:26:32,724 INFO manager on linux-kt3k: set sched {'sched_preempt_enforce_resumption': 'true'} 2017-10-24 23:26:32,724 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set sched sched_preempt_enforce_resumption=true 2017-10-24 23:26:32,739 INFO manager on linux-kt3k: set server {'backfill_depth': '2'} 2017-10-24 23:26:32,739 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server backfill_depth=2 2017-10-24 23:26:32,749 INFO filter: node {'resources_available.vntype': 'cray_compute'} 2017-10-24 23:26:32,749 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v -a 2017-10-24 23:26:32,763 INFO job: executable set to /bin/sleep with arguments: 100 2017-10-24 23:26:32,764 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0755 /tmp/PtlPbsJobScriptMpjgwM 2017-10-24 23:26:32,783 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chown pbsuser /tmp/PtlPbsJobScriptMpjgwM 2017-10-24 23:26:32,803 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chgrp tstgrp00 /tmp/PtlPbsJobScriptMpjgwM 2017-10-24 23:26:32,825 INFOCLI linux-kt3k: sudo -H -u pbsuser /opt/pbs/bin/qsub -l place=scatter -l walltime=120 -l select=2:ncpus=1:vntype=cray_compute /tmp/PtlPbsJobScriptMpjgwM 2017-10-24 23:26:32,867 INFO submit to linux-kt3k as pbsuser: job 138.linux-kt3k OrderedDict([('Resource_List.place', 'scatter'), ('Resource_List.walltime', '120'), ('Resource_List.select', '2:ncpus=1:vntype=cray_compute')]) 2017-10-24 23:26:32,868 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 138.linux-kt3k 2017-10-24 23:26:32,877 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 138.linux-kt3k got: job_state = Q 2017-10-24 23:26:33,379 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:26:33,390 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 138.linux-kt3k 2017-10-24 23:26:33,399 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 138.linux-kt3k attempt: 2 ... OK 2017-10-24 23:26:33,399 INFO alter on linux-kt3k as root: 138.linux-kt3k {'Additional_Attributes': 'topjob_ineligible = true'} 2017-10-24 23:26:33,400 INFOCLI linux-kt3k: /opt/pbs/bin/qalter -W topjob_ineligible = true 138.linux-kt3k 2017-10-24 23:26:33,411 INFO manager on linux-kt3k: create queue highp {'started': 't', 'priority': '150', 'queue_type': 'e', 'enabled': 'True'} 2017-10-24 23:26:33,412 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c create queue highp started=t,priority=150,queue_type=e,enabled=True 2017-10-24 23:26:33,423 INFO job: executable set to /bin/sleep with arguments: 100 2017-10-24 23:26:33,423 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0755 /tmp/PtlPbsJobScriptJ4jvv_ 2017-10-24 23:26:33,445 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chown pbsuser /tmp/PtlPbsJobScriptJ4jvv_ 2017-10-24 23:26:33,463 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chgrp tstgrp00 /tmp/PtlPbsJobScriptJ4jvv_ 2017-10-24 23:26:33,483 INFOCLI linux-kt3k: sudo -H -u pbsuser /opt/pbs/bin/qsub -q highp -l walltime=60 -l vntype=cray_compute /tmp/PtlPbsJobScriptJ4jvv_ 2017-10-24 23:26:33,506 INFO submit to linux-kt3k as pbsuser: job 139.linux-kt3k OrderedDict([('queue', 'highp'), ('Resource_List.walltime', '60'), ('Resource_List.vntype', 'cray_compute')]) 2017-10-24 23:26:33,506 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 139.linux-kt3k 2017-10-24 23:26:33,517 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 139.linux-kt3k got: job_state = Q 2017-10-24 23:26:34,019 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:26:34,030 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 139.linux-kt3k 2017-10-24 23:26:34,039 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 139.linux-kt3k attempt: 2 got: job_state = Q 2017-10-24 23:26:34,540 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:26:34,551 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 139.linux-kt3k 2017-10-24 23:26:34,563 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 139.linux-kt3k attempt: 3 got: job_state = Q 2017-10-24 23:26:35,065 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:26:35,076 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 139.linux-kt3k 2017-10-24 23:26:35,085 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 139.linux-kt3k attempt: 4 got: job_state = Q 2017-10-24 23:26:35,586 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:26:35,596 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 139.linux-kt3k 2017-10-24 23:26:35,604 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 139.linux-kt3k attempt: 5 got: job_state = Q 2017-10-24 23:26:36,105 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:26:36,119 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 139.linux-kt3k 2017-10-24 23:26:36,128 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 139.linux-kt3k attempt: 6 got: job_state = Q 2017-10-24 23:26:36,629 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:26:36,640 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 139.linux-kt3k 2017-10-24 23:26:36,650 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 139.linux-kt3k attempt: 7 got: job_state = Q 2017-10-24 23:26:37,152 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:26:37,162 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 139.linux-kt3k 2017-10-24 23:26:37,171 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 139.linux-kt3k attempt: 8 got: job_state = Q 2017-10-24 23:26:37,673 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:26:37,684 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 139.linux-kt3k 2017-10-24 23:26:37,693 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 139.linux-kt3k attempt: 9 got: job_state = Q 2017-10-24 23:26:38,195 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:26:38,208 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 139.linux-kt3k 2017-10-24 23:26:38,218 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 139.linux-kt3k attempt: 10 got: job_state = Q 2017-10-24 23:26:38,719 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:26:38,734 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 139.linux-kt3k 2017-10-24 23:26:38,746 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 139.linux-kt3k attempt: 11 got: job_state = Q 2017-10-24 23:26:39,248 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:26:39,264 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 139.linux-kt3k 2017-10-24 23:26:39,274 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 139.linux-kt3k attempt: 12 got: job_state = Q 2017-10-24 23:26:39,775 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:26:39,788 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 139.linux-kt3k 2017-10-24 23:26:39,797 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 139.linux-kt3k attempt: 13 got: job_state = Q 2017-10-24 23:26:40,298 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:26:40,311 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 139.linux-kt3k 2017-10-24 23:26:40,319 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 139.linux-kt3k attempt: 14 got: job_state = Q 2017-10-24 23:26:40,821 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:26:40,836 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 139.linux-kt3k 2017-10-24 23:26:40,849 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 139.linux-kt3k attempt: 15 got: job_state = Q 2017-10-24 23:26:41,351 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:26:41,363 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 139.linux-kt3k 2017-10-24 23:26:41,374 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 139.linux-kt3k attempt: 16 got: job_state = Q 2017-10-24 23:26:41,876 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:26:41,888 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 139.linux-kt3k 2017-10-24 23:26:41,898 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 139.linux-kt3k attempt: 17 got: job_state = Q 2017-10-24 23:26:42,399 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:26:42,412 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 139.linux-kt3k 2017-10-24 23:26:42,420 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 139.linux-kt3k attempt: 18 got: job_state = Q 2017-10-24 23:26:42,921 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:26:42,932 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 139.linux-kt3k 2017-10-24 23:26:42,940 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 139.linux-kt3k attempt: 19 got: job_state = Q 2017-10-24 23:26:43,441 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-24 23:26:43,452 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 139.linux-kt3k 2017-10-24 23:26:43,462 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 139.linux-kt3k attempt: 20 ... OK 2017-10-24 23:26:43,462 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 138.linux-kt3k 2017-10-24 23:26:43,470 INFO expect on server linux-kt3k: estimated.start_time set job 138.linux-kt3k ... OK 2017-10-24 23:26:43,471 INFO status on linux-kt3k: job 138.linux-kt3k estimated.start_time 2017-10-24 23:26:43,471 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 138.linux-kt3k 2017-10-24 23:26:43,481 INFO scheduler linux-kt3k log match: searching for "138.linux-kt3k;Job is a top job"... OK 2017-10-24 23:26:43,482 INFO ========================================= 2017-10-24 23:26:43,482 INFO Entered TestSuspendResumeOnCray tearDown 2017-10-24 23:26:43,482 INFO ========================================= 2017-10-24 23:26:43,482 INFO ========================================== 2017-10-24 23:26:43,482 INFO Completed TestSuspendResumeOnCray tearDown 2017-10-24 23:26:43,482 INFO ========================================== 2017-10-24 23:26:43,483 INFO ok 2017-10-24 23:26:43,483 INFO 2017-10-24 23:26:43,483 INFO ====================================================================== 2017-10-24 23:26:43,483 INFO FAILED: test_basic_admin_suspend_restart (tests.functional.pbs_cray_suspend_resume.TestSuspendResumeOnCray) 2017-10-24 23:26:43,483 INFO ___m_oo_m___ 2017-10-24 23:26:43,483 INFO Traceback (most recent call last): File "/tmp/PBSPro_18.2.0/tests/functional/pbs_cray_suspend_resume.py", line 121, in test_basic_admin_suspend_restart self.server.expect(NODE, {'state': 'maintenance'}, id=vname) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7792, in expect raise PtlExpectError(rc=1, rv=False, msg=_msg) PtlExpectError: rc=1, rv=False, msg=expected on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 60 got: state = job-exclusive,maintenance 2017-10-24 23:26:43,486 INFO ================================================================================ failed: test_basic_admin_suspend_restart (tests.functional.pbs_cray_suspend_resume.TestSuspendResumeOnCray) Test cases with failures: TestSuspendResumeOnCray.test_basic_admin_suspend_restart Test suites with failures: TestSuspendResumeOnCray run: 7, succeeded: 6, failed: 1, errors: 0, skipped: 0, timedout: 0 Tests run in 0:04:11.539038 2017-10-24 23:26:43,486 INFO Cleaning up temporary files