2017-10-21 05:11:47,099 INFO input command: pbs_benchpress -f tests/functional/pbs_cray_suspend_resume.py -o pbs_pp864_test.txt 2017-10-21 05:11:47,100 INFO param: None 2017-10-21 05:11:47,100 INFO ptl version: 17.1.0 2017-10-21 05:11:47,100 INFO platform: Linux linux-kt3k 3.12.28-4-default #1 SMP Thu Sep 25 17:02:34 UTC 2014 (9879bd4) x86_64 x86_64 2017-10-21 05:11:47,100 INFO python version: 2.7.12 2017-10-21 05:11:47,100 INFO user: root 2017-10-21 05:11:47,100 INFO -------------------------------------------------------------------------------- 2017-10-21 05:11:47,100 INFO Cleaning up temporary files 2017-10-21 05:11:47,111 INFO ====================================================================== 2017-10-21 05:11:47,111 INFO suite name: TestSuspendResumeOnCray 2017-10-21 05:11:47,111 INFO suite docstring: Test special cases where suspend/resume functionality differs on cray as compared to other platforms. This test suite expects the platform to be 'cray' and assumes that suspend/resume feature is enabled on it. 2017-10-21 05:11:47,112 INFO ====================================================================== 2017-10-21 05:11:47,112 INFO =========================================== 2017-10-21 05:11:47,112 INFO Entered TestSuspendResumeOnCray setUpClass 2017-10-21 05:11:47,112 INFO =========================================== 2017-10-21 05:11:47,112 INFOCLI2 linux-kt3k: id pbsuser 2017-10-21 05:11:47,116 INFOCLI2 linux-kt3k: id pbsuser1 2017-10-21 05:11:47,118 INFOCLI2 linux-kt3k: id pbsuser2 2017-10-21 05:11:47,121 INFOCLI2 linux-kt3k: id pbsuser3 2017-10-21 05:11:47,130 INFO server linux-kt3k: server operating mode set to cli 2017-10-21 05:11:47,130 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-21 05:11:47,139 INFO server linux-kt3k: version 18.2.0.20171019011017 2017-10-21 05:11:47,139 INFO expect action: created new action kicksched 2017-10-21 05:11:47,139 INFO expect action: added action kicksched to server linux-kt3k 2017-10-21 05:11:47,173 INFOCLI2 linux-kt3k: sudo -H /opt/pbs/sbin/pbsfs 2017-10-21 05:11:47,197 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2017-10-21 05:11:47,215 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2017-10-21 05:11:47,233 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list sched default 2017-10-21 05:11:47,267 INFOCLI2 linux-kt3k: sudo -H /opt/pbs/sbin/pbsfs 2017-10-21 05:11:47,295 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2017-10-21 05:11:47,315 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2017-10-21 05:11:47,339 INFO ============================================ 2017-10-21 05:11:47,340 INFO Completed TestSuspendResumeOnCray setUpClass 2017-10-21 05:11:47,340 INFO ============================================ 2017-10-21 05:11:47,340 INFO test name: test_admin_suspend_wrong_state (tests.functional.pbs_cray_suspend_resume.TestSuspendResumeOnCray)... 2017-10-21 05:11:47,340 INFO test start time: Sat Oct 21 05:11:47 2017 2017-10-21 05:11:47,340 INFO test docstring: Check that wrong 'resume' signal is correctly rejected. 2017-10-21 05:11:47,340 INFO ====================================== 2017-10-21 05:11:47,340 INFO Entered TestSuspendResumeOnCray setUp 2017-10-21 05:11:47,340 INFO ====================================== 2017-10-21 05:11:47,340 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-21 05:11:47,347 INFO status on linux-kt3k: server 2017-10-21 05:11:47,348 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-21 05:11:47,359 INFO manager on linux-kt3k: set server {'managers': (3, 'root@*')} 2017-10-21 05:11:47,359 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c set server managers-=root@* 2017-10-21 05:11:47,394 INFO manager on linux-kt3k: set server {'managers': (2, 'root@*')} 2017-10-21 05:11:47,395 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c set server managers+=root@* 2017-10-21 05:11:47,422 INFO server linux-kt3k: reverting configuration to defaults 2017-10-21 05:11:47,422 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-21 05:11:47,431 INFO select on linux-kt3k: __ALL__ 2017-10-21 05:11:47,431 INFOCLI linux-kt3k: /opt/pbs/bin/qselect 2017-10-21 05:11:47,442 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f @linux-kt3k 2017-10-21 05:11:47,453 INFO expect on server linux-kt3k: job_state set 0 job ... OK 2017-10-21 05:11:47,453 INFOCLI linux-kt3k: /opt/pbs/bin/pbs_rstat -f 2017-10-21 05:11:47,462 INFO manager on linux-kt3k: unset server ['backfill_depth', 'restrict_res_to_release_on_suspend'] 2017-10-21 05:11:47,462 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c unset server backfill_depth,restrict_res_to_release_on_suspend 2017-10-21 05:11:47,473 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-10-21 05:11:47,499 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Qf @linux-kt3k 2017-10-21 05:11:47,507 INFO status on linux-kt3k: node 2017-10-21 05:11:47,508 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v -a 2017-10-21 05:11:47,519 INFO manager on linux-kt3k: delete queue workq,highp 2017-10-21 05:11:47,519 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c delete queue workq,highp 2017-10-21 05:11:47,531 INFO server linux-kt3k: expect offset set to 0.5 2017-10-21 05:11:48,032 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Qf workq@linux-kt3k 2017-10-21 05:11:48,043 INFO expect on server linux-kt3k: unset queue workq ... OK 2017-10-21 05:11:48,043 INFO server linux-kt3k: expect offset set to 0.5 2017-10-21 05:11:48,545 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Qf highp@linux-kt3k 2017-10-21 05:11:48,556 INFO expect on server linux-kt3k: unset queue highp ... OK 2017-10-21 05:11:48,556 INFO manager on linux-kt3k: create queue workq {'started': 'True', 'queue_type': 'Execution', 'enabled': 'True'} 2017-10-21 05:11:48,556 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c create queue workq started=True,queue_type=Execution,enabled=True 2017-10-21 05:11:48,568 INFO server linux-kt3k: expect offset set to 0.5 2017-10-21 05:11:49,070 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Qf workq@linux-kt3k 2017-10-21 05:11:49,081 INFO expect on server linux-kt3k: started set True || queue_type set Execution || enabled set True queue workq ... OK 2017-10-21 05:11:49,081 INFOCLI2 linux-kt3k: sudo -H cmp /opt/pbs/lib/python/altair/pbs_hooks/PBS_translate_mpp.HK /var/spool/pbs/server_priv/hooks/PBS_translate_mpp.HK 2017-10-21 05:11:49,100 INFOCLI2 linux-kt3k: which cp 2017-10-21 05:11:49,102 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /opt/pbs/lib/python/altair/pbs_hooks/PBS_translate_mpp.HK /var/spool/pbs/server_priv/hooks/PBS_translate_mpp.HK 2017-10-21 05:11:49,124 INFOCLI2 linux-kt3k: which chmod 2017-10-21 05:11:49,126 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/server_priv/hooks/PBS_translate_mpp.HK 2017-10-21 05:11:49,145 INFO server linux-kt3k: sent signal -HUP 2017-10-21 05:11:49,145 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/server_priv/server.lock 2017-10-21 05:11:49,163 INFOCLI2 linux-kt3k: sudo -H kill -HUP 16810 2017-10-21 05:11:49,184 INFO manager on linux-kt3k: set pbshook PBS_translate_mpp {'enabled': 'true'} 2017-10-21 05:11:49,185 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c set pbshook PBS_translate_mpp enabled=true 2017-10-21 05:11:49,212 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-10-21 05:11:49,236 INFO manager on linux-kt3k: set server {'default_queue': 'workq', 'restrict_res_to_release_on_suspend': 'ncpus'} 2017-10-21 05:11:49,237 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server default_queue=workq,restrict_res_to_release_on_suspend=ncpus 2017-10-21 05:11:49,250 INFO status on linux-kt3k: resource 2017-10-21 05:11:49,250 INFO manager on linux-kt3k: list resource 2017-10-21 05:11:49,250 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource 2017-10-21 05:11:49,259 INFO manager on linux-kt3k: delete resource PBScraylabel_G34_IL,PBScrayorder,PBScrayhost,PBScraynid 2017-10-21 05:11:49,259 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c delete resource PBScraylabel_G34_IL,PBScrayorder,PBScrayhost,PBScraynid 2017-10-21 05:11:49,345 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource PBScraylabel_G34_IL 2017-10-21 05:11:49,354 INFO expect on server linux-kt3k: unset resource PBScraylabel_G34_IL ... OK 2017-10-21 05:11:49,355 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource PBScrayorder 2017-10-21 05:11:49,364 INFO expect on server linux-kt3k: unset resource PBScrayorder ... OK 2017-10-21 05:11:49,364 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource PBScrayhost 2017-10-21 05:11:49,371 INFO expect on server linux-kt3k: unset resource PBScrayhost ... OK 2017-10-21 05:11:49,372 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource PBScraynid 2017-10-21 05:11:49,379 INFO expect on server linux-kt3k: unset resource PBScraynid ... OK 2017-10-21 05:11:49,379 INFOCLI status on linux-kt3k: server license_count 2017-10-21 05:11:49,379 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-21 05:11:49,387 INFO server: linux-kt3k licensed 2017-10-21 05:11:49,397 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/server_priv/comm.lock 2017-10-21 05:11:49,431 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-10-21 05:11:49,449 INFO scheduler linux-kt3k: reverting configuration to defaults 2017-10-21 05:11:49,449 INFO manager on linux-kt3k: delete sched @default 2017-10-21 05:11:49,450 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c delete sched @default 2017-10-21 05:11:49,459 INFO manager on linux-kt3k: list sched 2017-10-21 05:11:49,459 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list sched 2017-10-21 05:11:49,468 INFO manager on linux-kt3k: unset sched ['sched_preempt_enforce_resumption', 'sched_cycle_length'] 2017-10-21 05:11:49,468 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c unset sched sched_preempt_enforce_resumption,sched_cycle_length 2017-10-21 05:11:49,480 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/dedicated_time 2017-10-21 05:11:49,498 INFOCLI2 linux-kt3k: cmp /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-10-21 05:11:49,500 INFO scheduler linux-kt3k: reverting holidays file to default 2017-10-21 05:11:49,500 INFOCLI2 linux-kt3k: cmp /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-10-21 05:11:49,502 INFOCLI2 linux-kt3k: cmp /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-10-21 05:11:49,504 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-10-21 05:11:49,523 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-10-21 05:11:49,543 INFO scheduler linux-kt3k: sent signal -HUP 2017-10-21 05:11:49,543 INFOCLI2 linux-kt3k: sudo -H kill -HUP 96312 2017-10-21 05:11:49,567 INFOCLI2 linux-kt3k: sudo -H /opt/pbs/sbin/pbsfs -e 2017-10-21 05:11:49,621 INFO scheduler linux-kt3k: config {'resources': '"ncpus, mem, arch, host, vnode, aoe, eoe, vntype"'} 2017-10-21 05:11:49,622 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /var/spool/pbs/sched_priv/sched_config /var/spool/pbs/sched_priv/sched_config.bak 2017-10-21 05:11:49,643 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /tmp/PtlPbsmOtJwH /var/spool/pbs/sched_priv/sched_config 2017-10-21 05:11:49,662 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-10-21 05:11:49,681 INFOCLI2 linux-kt3k: which chown 2017-10-21 05:11:49,683 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chown root /var/spool/pbs/sched_priv/sched_config 2017-10-21 05:11:49,708 INFOCLI2 linux-kt3k: which chgrp 2017-10-21 05:11:49,710 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chgrp root /var/spool/pbs/sched_priv/sched_config 2017-10-21 05:11:49,729 INFO scheduler linux-kt3k: sent signal -HUP 2017-10-21 05:11:49,730 INFOCLI2 linux-kt3k: sudo -H kill -HUP 96312 2017-10-21 05:11:49,751 INFO scheduler linux-kt3k log match: searching for "Error reading line" - No match 2017-10-21 05:11:50,761 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-10-21 05:11:50,791 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-10-21 05:11:50,813 INFOCLI2 linux-kt3k: sudo -H /opt/pbs/sbin/pbs_mom --version 2017-10-21 05:11:50,845 INFO mom linux-kt3k: reverting configuration to defaults 2017-10-21 05:11:50,846 INFOCLI2 linux-kt3k: which rm 2017-10-21 05:11:50,847 INFOCLI2 linux-kt3k: sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2017-10-21 05:11:50,877 INFOCLI2 linux-kt3k: sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2017-10-21 05:11:50,895 INFOCLI linux-kt3k: sudo -H /opt/pbs/sbin/pbs_mom -s list 2017-10-21 05:11:50,930 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /tmp/PtlPbsIm7zz9 /var/spool/pbs/mom_priv/config 2017-10-21 05:11:50,953 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/mom_priv/config 2017-10-21 05:11:50,974 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chown root /var/spool/pbs/mom_priv/config 2017-10-21 05:11:50,994 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chgrp root /var/spool/pbs/mom_priv/config 2017-10-21 05:11:51,014 INFO mom linux-kt3k: sent signal -HUP 2017-10-21 05:11:51,014 INFOCLI2 linux-kt3k: sudo -H kill -HUP 96290 2017-10-21 05:11:51,045 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-10-21 05:11:51,069 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v -a 2017-10-21 05:11:51,082 INFO manager on linux-kt3k: delete node 2017-10-21 05:11:51,083 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c delete node @default 2017-10-21 05:11:51,096 INFO manager on linux-kt3k: create node linux-kt3k 2017-10-21 05:11:51,096 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c create node linux-kt3k 2017-10-21 05:11:51,116 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k 2017-10-21 05:11:51,137 INFO expect on server linux-kt3k: state = free node linux-kt3k got: state = state-unknown,down 2017-10-21 05:11:52,139 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k 2017-10-21 05:11:52,149 INFO expect on server linux-kt3k: state = free node linux-kt3k attempt: 2 got: state = state-unknown,down 2017-10-21 05:11:53,153 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k 2017-10-21 05:11:53,193 INFO expect on server linux-kt3k: state = free node linux-kt3k attempt: 3 ... OK 2017-10-21 05:11:53,193 INFO ======================================= 2017-10-21 05:11:53,193 INFO Completed TestSuspendResumeOnCray setUp 2017-10-21 05:11:53,193 INFO ======================================= 2017-10-21 05:11:53,193 INFO job: executable set to /bin/sleep with arguments: 100 2017-10-21 05:11:53,194 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0755 /tmp/PtlPbsJobScriptF0hTdk 2017-10-21 05:11:53,214 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chown pbsuser /tmp/PtlPbsJobScriptF0hTdk 2017-10-21 05:11:53,233 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chgrp tstgrp00 /tmp/PtlPbsJobScriptF0hTdk 2017-10-21 05:11:53,254 INFOCLI linux-kt3k: sudo -H -u pbsuser /opt/pbs/bin/qsub -l vntype=cray_compute /tmp/PtlPbsJobScriptF0hTdk 2017-10-21 05:11:53,293 INFO submit to linux-kt3k as pbsuser: job 92.linux-kt3k OrderedDict([('Resource_List.vntype', 'cray_compute')]) 2017-10-21 05:11:53,294 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 92.linux-kt3k 2017-10-21 05:11:53,302 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 92.linux-kt3k got: job_state = Q 2017-10-21 05:11:53,804 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:11:53,815 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 92.linux-kt3k 2017-10-21 05:11:53,823 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 92.linux-kt3k attempt: 2 ... OK 2017-10-21 05:11:53,824 INFO signal on linux-kt3k as root: 92.linux-kt3k with signal = suspend 2017-10-21 05:11:53,824 INFOCLI linux-kt3k: /opt/pbs/bin/qsig -s suspend 92.linux-kt3k 2017-10-21 05:12:03,070 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 92.linux-kt3k 2017-10-21 05:12:03,082 INFO expect on server linux-kt3k: job_state = S job 92.linux-kt3k ... OK 2017-10-21 05:12:03,082 INFO signal on linux-kt3k as root: 92.linux-kt3k with signal = admin-resume 2017-10-21 05:12:03,082 INFOCLI linux-kt3k: /opt/pbs/bin/qsig -s admin-resume 92.linux-kt3k 2017-10-21 05:12:03,092 ERROR err: ['qsig: Job can not be resumed with the requested resume signal 92.linux-kt3k'] 2017-10-21 05:12:03,092 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 92.linux-kt3k 2017-10-21 05:12:03,105 INFO expect on server linux-kt3k: job_state = S job 92.linux-kt3k ... OK 2017-10-21 05:12:03,105 INFO job: executable set to /bin/sleep with arguments: 100 2017-10-21 05:12:03,105 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0755 /tmp/PtlPbsJobScriptKHrQS1 2017-10-21 05:12:03,132 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chown pbsuser /tmp/PtlPbsJobScriptKHrQS1 2017-10-21 05:12:03,155 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chgrp tstgrp00 /tmp/PtlPbsJobScriptKHrQS1 2017-10-21 05:12:03,183 INFOCLI linux-kt3k: sudo -H -u pbsuser /opt/pbs/bin/qsub -l vntype=cray_compute /tmp/PtlPbsJobScriptKHrQS1 2017-10-21 05:12:03,219 INFO submit to linux-kt3k as pbsuser: job 93.linux-kt3k OrderedDict([('Resource_List.vntype', 'cray_compute')]) 2017-10-21 05:12:03,219 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 93.linux-kt3k 2017-10-21 05:12:03,232 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 93.linux-kt3k got: job_state = Q 2017-10-21 05:12:03,734 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:12:03,745 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 93.linux-kt3k 2017-10-21 05:12:03,753 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 93.linux-kt3k attempt: 2 ... OK 2017-10-21 05:12:03,753 INFO signal on linux-kt3k as root: 93.linux-kt3k with signal = admin-suspend 2017-10-21 05:12:03,753 INFOCLI linux-kt3k: /opt/pbs/bin/qsig -s admin-suspend 93.linux-kt3k 2017-10-21 05:12:13,101 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 93.linux-kt3k 2017-10-21 05:12:13,111 INFO expect on server linux-kt3k: job_state = S || substate = 43 job 93.linux-kt3k ... OK 2017-10-21 05:12:13,111 INFO signal on linux-kt3k as root: 93.linux-kt3k with signal = resume 2017-10-21 05:12:13,111 INFOCLI linux-kt3k: /opt/pbs/bin/qsig -s resume 93.linux-kt3k 2017-10-21 05:12:13,118 ERROR err: ['qsig: Job can not be resumed with the requested resume signal 93.linux-kt3k'] 2017-10-21 05:12:13,119 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 93.linux-kt3k 2017-10-21 05:12:13,130 INFO expect on server linux-kt3k: job_state = S || substate = 43 job 93.linux-kt3k ... OK 2017-10-21 05:12:13,131 INFO ========================================= 2017-10-21 05:12:13,131 INFO Entered TestSuspendResumeOnCray tearDown 2017-10-21 05:12:13,131 INFO ========================================= 2017-10-21 05:12:13,131 INFO ========================================== 2017-10-21 05:12:13,131 INFO Completed TestSuspendResumeOnCray tearDown 2017-10-21 05:12:13,131 INFO ========================================== 2017-10-21 05:12:13,132 INFO ok 2017-10-21 05:12:13,132 INFO test name: test_basic_admin_suspend_restart (tests.functional.pbs_cray_suspend_resume.TestSuspendResumeOnCray)... 2017-10-21 05:12:13,133 INFO test start time: Sat Oct 21 05:12:13 2017 2017-10-21 05:12:13,133 INFO test docstring: Test basic admin-suspend funcionality for jobs and array jobs with restart on Cray. The restart will test if the node recovers properly in maintenance. After turning off scheduling and a server restart, a subjob is always requeued and node shows up as free. 2017-10-21 05:12:13,133 INFO ====================================== 2017-10-21 05:12:13,133 INFO Entered TestSuspendResumeOnCray setUp 2017-10-21 05:12:13,134 INFO ====================================== 2017-10-21 05:12:13,134 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-21 05:12:13,143 INFO status on linux-kt3k: server 2017-10-21 05:12:13,143 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-21 05:12:13,152 INFO manager on linux-kt3k: set server {'managers': (3, 'root@*')} 2017-10-21 05:12:13,152 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c set server managers-=root@* 2017-10-21 05:12:13,186 INFO manager on linux-kt3k: set server {'managers': (2, 'root@*')} 2017-10-21 05:12:13,186 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c set server managers+=root@* 2017-10-21 05:12:13,219 INFO server linux-kt3k: reverting configuration to defaults 2017-10-21 05:12:13,220 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-21 05:12:13,230 INFO select on linux-kt3k: __ALL__ 2017-10-21 05:12:13,230 INFOCLI linux-kt3k: /opt/pbs/bin/qselect 2017-10-21 05:12:13,238 INFO delete job on linux-kt3k: 92.linux-kt3k, 93.linux-kt3k 2017-10-21 05:12:13,239 INFOCLI linux-kt3k: /opt/pbs/bin/qdel -W force 92.linux-kt3k 93.linux-kt3k 2017-10-21 05:12:13,276 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 92.linux-kt3k 2017-10-21 05:12:13,318 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 93.linux-kt3k 2017-10-21 05:12:13,369 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f @linux-kt3k 2017-10-21 05:12:13,380 INFO expect on server linux-kt3k: job_state set 0 job ... OK 2017-10-21 05:12:13,380 INFOCLI linux-kt3k: /opt/pbs/bin/pbs_rstat -f 2017-10-21 05:12:13,390 INFO manager on linux-kt3k: unset server ['restrict_res_to_release_on_suspend'] 2017-10-21 05:12:13,390 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c unset server restrict_res_to_release_on_suspend 2017-10-21 05:12:13,417 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-10-21 05:12:13,455 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Qf @linux-kt3k 2017-10-21 05:12:13,471 INFO status on linux-kt3k: node 2017-10-21 05:12:13,471 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v -a 2017-10-21 05:12:13,482 INFO manager on linux-kt3k: delete queue workq 2017-10-21 05:12:13,482 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c delete queue workq 2017-10-21 05:12:13,496 INFO server linux-kt3k: expect offset set to 0.5 2017-10-21 05:12:13,998 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Qf workq@linux-kt3k 2017-10-21 05:12:14,008 INFO expect on server linux-kt3k: unset queue workq ... OK 2017-10-21 05:12:14,008 INFO manager on linux-kt3k: create queue workq {'started': 'True', 'queue_type': 'Execution', 'enabled': 'True'} 2017-10-21 05:12:14,009 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c create queue workq started=True,queue_type=Execution,enabled=True 2017-10-21 05:12:14,018 INFO server linux-kt3k: expect offset set to 0.5 2017-10-21 05:12:14,519 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Qf workq@linux-kt3k 2017-10-21 05:12:14,529 INFO expect on server linux-kt3k: started set True || queue_type set Execution || enabled set True queue workq ... OK 2017-10-21 05:12:14,529 INFOCLI2 linux-kt3k: sudo -H cmp /opt/pbs/lib/python/altair/pbs_hooks/PBS_translate_mpp.HK /var/spool/pbs/server_priv/hooks/PBS_translate_mpp.HK 2017-10-21 05:12:14,548 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /opt/pbs/lib/python/altair/pbs_hooks/PBS_translate_mpp.HK /var/spool/pbs/server_priv/hooks/PBS_translate_mpp.HK 2017-10-21 05:12:14,570 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/server_priv/hooks/PBS_translate_mpp.HK 2017-10-21 05:12:14,589 INFO server linux-kt3k: sent signal -HUP 2017-10-21 05:12:14,589 INFOCLI2 linux-kt3k: sudo -H kill -HUP 16810 2017-10-21 05:12:14,608 INFO manager on linux-kt3k: set pbshook PBS_translate_mpp {'enabled': 'true'} 2017-10-21 05:12:14,608 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c set pbshook PBS_translate_mpp enabled=true 2017-10-21 05:12:14,637 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-10-21 05:12:14,664 INFO manager on linux-kt3k: set server {'default_queue': 'workq', 'restrict_res_to_release_on_suspend': 'ncpus'} 2017-10-21 05:12:14,664 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server default_queue=workq,restrict_res_to_release_on_suspend=ncpus 2017-10-21 05:12:14,678 INFO status on linux-kt3k: resource 2017-10-21 05:12:14,678 INFO manager on linux-kt3k: list resource 2017-10-21 05:12:14,678 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource 2017-10-21 05:12:14,688 INFO manager on linux-kt3k: delete resource PBScraylabel_G34_IL,PBScrayorder,PBScrayhost,PBScraynid 2017-10-21 05:12:14,688 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c delete resource PBScraylabel_G34_IL,PBScrayorder,PBScrayhost,PBScraynid 2017-10-21 05:12:14,767 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource PBScraylabel_G34_IL 2017-10-21 05:12:14,776 INFO expect on server linux-kt3k: unset resource PBScraylabel_G34_IL ... OK 2017-10-21 05:12:14,776 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource PBScrayorder 2017-10-21 05:12:14,784 INFO expect on server linux-kt3k: unset resource PBScrayorder ... OK 2017-10-21 05:12:14,785 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource PBScrayhost 2017-10-21 05:12:14,795 INFO expect on server linux-kt3k: unset resource PBScrayhost ... OK 2017-10-21 05:12:14,795 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource PBScraynid 2017-10-21 05:12:14,804 INFO expect on server linux-kt3k: unset resource PBScraynid ... OK 2017-10-21 05:12:14,805 INFOCLI status on linux-kt3k: server license_count 2017-10-21 05:12:14,805 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-21 05:12:14,815 INFO server: linux-kt3k licensed 2017-10-21 05:12:14,830 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/server_priv/comm.lock 2017-10-21 05:12:14,861 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-10-21 05:12:14,882 INFO scheduler linux-kt3k: reverting configuration to defaults 2017-10-21 05:12:14,882 INFO manager on linux-kt3k: delete sched @default 2017-10-21 05:12:14,882 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c delete sched @default 2017-10-21 05:12:14,891 INFO manager on linux-kt3k: list sched 2017-10-21 05:12:14,891 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list sched 2017-10-21 05:12:14,900 INFO manager on linux-kt3k: unset sched ['sched_preempt_enforce_resumption', 'sched_cycle_length'] 2017-10-21 05:12:14,900 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c unset sched sched_preempt_enforce_resumption,sched_cycle_length 2017-10-21 05:12:14,909 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/dedicated_time 2017-10-21 05:12:14,930 INFOCLI2 linux-kt3k: cmp /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-10-21 05:12:14,932 INFO scheduler linux-kt3k: reverting holidays file to default 2017-10-21 05:12:14,932 INFOCLI2 linux-kt3k: cmp /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-10-21 05:12:14,934 INFOCLI2 linux-kt3k: cmp /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-10-21 05:12:14,936 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-10-21 05:12:14,956 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-10-21 05:12:14,977 INFO scheduler linux-kt3k: sent signal -HUP 2017-10-21 05:12:14,978 INFOCLI2 linux-kt3k: sudo -H kill -HUP 96312 2017-10-21 05:12:14,998 INFOCLI2 linux-kt3k: sudo -H /opt/pbs/sbin/pbsfs -e 2017-10-21 05:12:15,045 INFO scheduler linux-kt3k: config {'resources': '"ncpus, mem, arch, host, vnode, aoe, eoe, vntype"'} 2017-10-21 05:12:15,046 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /var/spool/pbs/sched_priv/sched_config /var/spool/pbs/sched_priv/sched_config.bak 2017-10-21 05:12:15,070 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /tmp/PtlPbsEMWdjj /var/spool/pbs/sched_priv/sched_config 2017-10-21 05:12:15,094 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-10-21 05:12:15,116 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chown root /var/spool/pbs/sched_priv/sched_config 2017-10-21 05:12:15,141 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chgrp root /var/spool/pbs/sched_priv/sched_config 2017-10-21 05:12:15,168 INFO scheduler linux-kt3k: sent signal -HUP 2017-10-21 05:12:15,168 INFOCLI2 linux-kt3k: sudo -H kill -HUP 96312 2017-10-21 05:12:15,190 INFO scheduler linux-kt3k log match: searching for "Error reading line" - No match 2017-10-21 05:12:16,203 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-10-21 05:12:16,233 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-10-21 05:12:16,252 INFO mom linux-kt3k: reverting configuration to defaults 2017-10-21 05:12:16,252 INFOCLI2 linux-kt3k: sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2017-10-21 05:12:16,271 INFOCLI2 linux-kt3k: sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2017-10-21 05:12:16,290 INFOCLI linux-kt3k: sudo -H /opt/pbs/sbin/pbs_mom -s list 2017-10-21 05:12:16,325 INFO mom linux-kt3k: sent signal -HUP 2017-10-21 05:12:16,326 INFOCLI2 linux-kt3k: sudo -H kill -HUP 96290 2017-10-21 05:12:16,353 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-10-21 05:12:16,383 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v -a 2017-10-21 05:12:16,395 INFO manager on linux-kt3k: delete node 2017-10-21 05:12:16,395 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c delete node @default 2017-10-21 05:12:16,409 INFO manager on linux-kt3k: create node linux-kt3k 2017-10-21 05:12:16,410 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c create node linux-kt3k 2017-10-21 05:12:16,422 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k 2017-10-21 05:12:16,445 INFO expect on server linux-kt3k: state = free node linux-kt3k got: state = state-unknown,down 2017-10-21 05:12:17,448 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k 2017-10-21 05:12:17,458 INFO expect on server linux-kt3k: state = free node linux-kt3k attempt: 2 got: state = state-unknown,down 2017-10-21 05:12:18,465 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k 2017-10-21 05:12:18,495 INFO expect on server linux-kt3k: state = free node linux-kt3k attempt: 3 ... OK 2017-10-21 05:12:18,496 INFO ======================================= 2017-10-21 05:12:18,496 INFO Completed TestSuspendResumeOnCray setUp 2017-10-21 05:12:18,496 INFO ======================================= 2017-10-21 05:12:18,496 INFO job: executable set to /bin/sleep with arguments: 100 2017-10-21 05:12:18,496 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0755 /tmp/PtlPbsJobScriptdgsVVb 2017-10-21 05:12:18,517 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chown pbsuser /tmp/PtlPbsJobScriptdgsVVb 2017-10-21 05:12:18,536 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chgrp tstgrp00 /tmp/PtlPbsJobScriptdgsVVb 2017-10-21 05:12:18,558 INFOCLI linux-kt3k: sudo -H -u pbsuser /opt/pbs/bin/qsub -l vntype=cray_compute /tmp/PtlPbsJobScriptdgsVVb 2017-10-21 05:12:18,584 INFO submit to linux-kt3k as pbsuser: job 94.linux-kt3k OrderedDict([('Resource_List.vntype', 'cray_compute')]) 2017-10-21 05:12:18,584 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 94.linux-kt3k 2017-10-21 05:12:18,598 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 94.linux-kt3k got: job_state = Q 2017-10-21 05:12:19,100 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:12:19,113 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 94.linux-kt3k 2017-10-21 05:12:19,123 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 94.linux-kt3k attempt: 2 ... OK 2017-10-21 05:12:19,123 INFO status on linux-kt3k: job 94.linux-kt3k exec_vnode 2017-10-21 05:12:19,123 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 94.linux-kt3k 2017-10-21 05:12:19,131 INFO signal on linux-kt3k as root: 94.linux-kt3k with signal = admin-suspend 2017-10-21 05:12:19,132 INFOCLI linux-kt3k: /opt/pbs/bin/qsig -s admin-suspend 94.linux-kt3k 2017-10-21 05:12:29,057 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 94.linux-kt3k 2017-10-21 05:12:29,066 INFO expect on server linux-kt3k: job_state = S job 94.linux-kt3k ... OK 2017-10-21 05:12:29,067 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-21 05:12:29,076 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 ... OK 2017-10-21 05:12:29,076 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v -a 2017-10-21 05:12:29,084 INFO expect on server linux-kt3k: maintenance_jobs = 94.linux-kt3k node ... OK 2017-10-21 05:12:29,084 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-21 05:12:29,094 INFO Set {'PBS_START_MOM': 0, 'PBS_START_SERVER': 1, 'PBS_START_SCHED': 0, 'PBS_START_COMM': 0} in /tmp/PtlPbsOfQqUp 2017-10-21 05:12:29,097 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chown root /tmp/PtlPbsG201RV 2017-10-21 05:12:29,116 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chgrp root /tmp/PtlPbsG201RV 2017-10-21 05:12:29,153 INFOCLI2 linux-kt3k: sudo -H /usr/bin/rm /tmp/PtlPbsG201RV 2017-10-21 05:12:29,172 INFOCLI2 linux-kt3k: sudo -H ls -l /opt/pbs/libexec/pbs_init.d 2017-10-21 05:12:29,191 INFO running init script to stop pbs server on linux-kt3k using /etc/pbs.conf init_cmd=['sudo', 'PBS_CONF_FILE=/tmp/PtlPbsOfQqUp', '/opt/pbs/libexec/pbs_init.d', 'stop'] 2017-10-21 05:12:29,192 INFOCLI2 linux-kt3k: /tmp/PtlPbsrL8wob Contents of /tmp/PtlPbsrL8wob: ---------------------------------------- #!/bin/bash sudo PBS_CONF_FILE=/tmp/PtlPbsOfQqUp /opt/pbs/libexec/pbs_init.d stop ---------------------------------------- 2017-10-21 05:12:31,568 INFOCLI2 linux-kt3k: /usr/bin/rm /tmp/PtlPbsrL8wob 2017-10-21 05:12:31,572 INFO Set {'PBS_START_MOM': 0, 'PBS_START_SERVER': 1, 'PBS_START_SCHED': 0, 'PBS_START_COMM': 0} in /tmp/PtlPbsPdCsm8 2017-10-21 05:12:31,574 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chown root /tmp/PtlPbsB7Youe 2017-10-21 05:12:31,596 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chgrp root /tmp/PtlPbsB7Youe 2017-10-21 05:12:31,643 INFOCLI2 linux-kt3k: sudo -H /usr/bin/rm /tmp/PtlPbsB7Youe 2017-10-21 05:12:31,664 INFOCLI2 linux-kt3k: sudo -H ls -l /opt/pbs/libexec/pbs_init.d 2017-10-21 05:12:31,688 INFO running init script to start pbs server on linux-kt3k using /etc/pbs.conf init_cmd=['sudo', 'PBS_CONF_FILE=/tmp/PtlPbsPdCsm8', '/opt/pbs/libexec/pbs_init.d', 'start'] 2017-10-21 05:12:31,689 INFOCLI2 linux-kt3k: /tmp/PtlPbs0uUKrX Contents of /tmp/PtlPbs0uUKrX: ---------------------------------------- #!/bin/bash sudo PBS_CONF_FILE=/tmp/PtlPbsPdCsm8 /opt/pbs/libexec/pbs_init.d start ---------------------------------------- 2017-10-21 05:12:36,141 INFOCLI2 linux-kt3k: /usr/bin/rm /tmp/PtlPbs0uUKrX 2017-10-21 05:12:36,164 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/server_priv/server.lock 2017-10-21 05:12:36,183 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-21 05:12:36,192 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-21 05:12:36,201 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 got: state = job-exclusive,maintenance 2017-10-21 05:12:36,702 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-21 05:12:36,712 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 2 got: state = job-exclusive,maintenance 2017-10-21 05:12:37,214 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-21 05:12:37,226 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 3 got: state = job-exclusive,maintenance 2017-10-21 05:12:37,728 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-21 05:12:37,738 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 4 got: state = job-exclusive,maintenance 2017-10-21 05:12:38,240 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-21 05:12:38,252 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 5 got: state = job-exclusive,maintenance 2017-10-21 05:12:38,754 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-21 05:12:38,766 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 6 got: state = job-exclusive,maintenance 2017-10-21 05:12:39,268 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-21 05:12:39,278 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 7 got: state = job-exclusive,maintenance 2017-10-21 05:12:39,780 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-21 05:12:39,790 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 8 got: state = job-exclusive,maintenance 2017-10-21 05:12:40,292 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-21 05:12:40,303 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 9 got: state = job-exclusive,maintenance 2017-10-21 05:12:40,804 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-21 05:12:40,814 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 10 got: state = job-exclusive,maintenance 2017-10-21 05:12:41,315 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-21 05:12:41,325 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 11 got: state = job-exclusive,maintenance 2017-10-21 05:12:41,827 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-21 05:12:41,839 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 12 got: state = job-exclusive,maintenance 2017-10-21 05:12:42,341 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-21 05:12:42,354 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 13 got: state = job-exclusive,maintenance 2017-10-21 05:12:42,855 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-21 05:12:42,865 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 14 got: state = job-exclusive,maintenance 2017-10-21 05:12:43,367 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-21 05:12:43,377 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 15 got: state = job-exclusive,maintenance 2017-10-21 05:12:43,878 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-21 05:12:43,889 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 16 got: state = job-exclusive,maintenance 2017-10-21 05:12:44,390 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-21 05:12:44,399 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 17 got: state = job-exclusive,maintenance 2017-10-21 05:12:44,901 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-21 05:12:44,911 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 18 got: state = job-exclusive,maintenance 2017-10-21 05:12:45,412 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-21 05:12:45,423 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 19 got: state = job-exclusive,maintenance 2017-10-21 05:12:45,925 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-21 05:12:45,937 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 20 got: state = job-exclusive,maintenance 2017-10-21 05:12:46,438 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-21 05:12:46,450 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 21 got: state = job-exclusive,maintenance 2017-10-21 05:12:46,951 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-21 05:12:46,963 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 22 got: state = job-exclusive,maintenance 2017-10-21 05:12:47,466 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-21 05:12:47,476 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 23 got: state = job-exclusive,maintenance 2017-10-21 05:12:47,978 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-21 05:12:47,987 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 24 got: state = job-exclusive,maintenance 2017-10-21 05:12:48,489 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-21 05:12:48,498 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 25 got: state = job-exclusive,maintenance 2017-10-21 05:12:49,000 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-21 05:12:49,010 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 26 got: state = job-exclusive,maintenance 2017-10-21 05:12:49,512 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-21 05:12:49,524 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 27 got: state = job-exclusive,maintenance 2017-10-21 05:12:50,026 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-21 05:12:50,036 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 28 got: state = job-exclusive,maintenance 2017-10-21 05:12:50,538 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-21 05:12:50,549 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 29 got: state = job-exclusive,maintenance 2017-10-21 05:12:51,051 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-21 05:12:51,061 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 30 got: state = job-exclusive,maintenance 2017-10-21 05:12:51,563 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-21 05:12:51,574 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 31 got: state = job-exclusive,maintenance 2017-10-21 05:12:52,075 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-21 05:12:52,086 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 32 got: state = job-exclusive,maintenance 2017-10-21 05:12:52,588 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-21 05:12:52,598 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 33 got: state = job-exclusive,maintenance 2017-10-21 05:12:53,100 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-21 05:12:53,110 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 34 got: state = job-exclusive,maintenance 2017-10-21 05:12:53,612 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-21 05:12:53,622 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 35 got: state = job-exclusive,maintenance 2017-10-21 05:12:54,124 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-21 05:12:54,136 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 36 got: state = job-exclusive,maintenance 2017-10-21 05:12:54,637 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-21 05:12:54,646 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 37 got: state = job-exclusive,maintenance 2017-10-21 05:12:55,147 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-21 05:12:55,157 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 38 got: state = job-exclusive,maintenance 2017-10-21 05:12:55,659 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-21 05:12:55,670 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 39 got: state = job-exclusive,maintenance 2017-10-21 05:12:56,171 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-21 05:12:56,182 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 40 got: state = job-exclusive,maintenance 2017-10-21 05:12:56,684 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-21 05:12:56,696 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 41 got: state = job-exclusive,maintenance 2017-10-21 05:12:57,197 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-21 05:12:57,207 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 42 got: state = job-exclusive,maintenance 2017-10-21 05:12:57,709 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-21 05:12:57,719 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 43 got: state = job-exclusive,maintenance 2017-10-21 05:12:58,220 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-21 05:12:58,232 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 44 got: state = job-exclusive,maintenance 2017-10-21 05:12:58,734 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-21 05:12:58,745 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 45 got: state = job-exclusive,maintenance 2017-10-21 05:12:59,247 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-21 05:12:59,257 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 46 got: state = job-exclusive,maintenance 2017-10-21 05:12:59,758 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-21 05:12:59,767 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 47 got: state = job-exclusive,maintenance 2017-10-21 05:13:00,268 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-21 05:13:00,279 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 48 got: state = job-exclusive,maintenance 2017-10-21 05:13:00,780 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-21 05:13:00,791 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 49 got: state = job-exclusive,maintenance 2017-10-21 05:13:01,293 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-21 05:13:01,303 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 50 got: state = job-exclusive,maintenance 2017-10-21 05:13:01,804 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-21 05:13:01,815 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 51 got: state = job-exclusive,maintenance 2017-10-21 05:13:02,316 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-21 05:13:02,327 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 52 got: state = job-exclusive,maintenance 2017-10-21 05:13:02,829 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-21 05:13:02,840 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 53 got: state = job-exclusive,maintenance 2017-10-21 05:13:03,341 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-21 05:13:03,352 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 54 got: state = job-exclusive,maintenance 2017-10-21 05:13:03,854 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-21 05:13:03,866 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 55 got: state = job-exclusive,maintenance 2017-10-21 05:13:04,368 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-21 05:13:04,379 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 56 got: state = job-exclusive,maintenance 2017-10-21 05:13:04,881 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-21 05:13:04,893 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 57 got: state = job-exclusive,maintenance 2017-10-21 05:13:05,395 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-21 05:13:05,406 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 58 got: state = job-exclusive,maintenance 2017-10-21 05:13:05,908 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-21 05:13:05,917 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 59 got: state = job-exclusive,maintenance 2017-10-21 05:13:06,419 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k_2 2017-10-21 05:13:06,430 INFO expect on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 60 got: state = job-exclusive,maintenance 2017-10-21 05:13:06,934 INFO FAILED 2017-10-21 05:13:06,934 INFO ========================================= 2017-10-21 05:13:06,935 INFO Entered TestSuspendResumeOnCray tearDown 2017-10-21 05:13:06,935 INFO ========================================= 2017-10-21 05:13:06,935 INFO ========================================== 2017-10-21 05:13:06,935 INFO Completed TestSuspendResumeOnCray tearDown 2017-10-21 05:13:06,935 INFO ========================================== 2017-10-21 05:13:06,936 INFO test name: test_default_restrict_res_to_release_on_suspend_setting (tests.functional.pbs_cray_suspend_resume.TestSuspendResumeOnCray)... 2017-10-21 05:13:06,936 INFO test start time: Sat Oct 21 05:13:06 2017 2017-10-21 05:13:06,936 INFO test docstring: Check that on Cray restrict_res_to_release_on_suspend is always set to 'ncpus' by default 2017-10-21 05:13:06,936 INFO ====================================== 2017-10-21 05:13:06,937 INFO Entered TestSuspendResumeOnCray setUp 2017-10-21 05:13:06,937 INFO ====================================== 2017-10-21 05:13:06,937 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-21 05:13:06,947 INFO status on linux-kt3k: server 2017-10-21 05:13:06,947 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-21 05:13:06,955 INFO manager on linux-kt3k: set server {'managers': (3, 'root@*')} 2017-10-21 05:13:06,955 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c set server managers-=root@* 2017-10-21 05:13:06,985 INFO manager on linux-kt3k: set server {'managers': (2, 'root@*')} 2017-10-21 05:13:06,986 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c set server managers+=root@* 2017-10-21 05:13:07,016 INFO server linux-kt3k: reverting configuration to defaults 2017-10-21 05:13:07,016 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-21 05:13:07,025 INFO select on linux-kt3k: __ALL__ 2017-10-21 05:13:07,025 INFOCLI linux-kt3k: /opt/pbs/bin/qselect 2017-10-21 05:13:07,034 INFO delete job on linux-kt3k: 94.linux-kt3k 2017-10-21 05:13:07,034 INFOCLI linux-kt3k: /opt/pbs/bin/qdel -W force 94.linux-kt3k 2017-10-21 05:13:07,056 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 94.linux-kt3k 2017-10-21 05:13:07,077 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f @linux-kt3k 2017-10-21 05:13:07,087 INFO expect on server linux-kt3k: job_state set 0 job ... OK 2017-10-21 05:13:07,087 INFOCLI linux-kt3k: /opt/pbs/bin/pbs_rstat -f 2017-10-21 05:13:07,100 INFO manager on linux-kt3k: unset server ['pbs_license_min', 'restrict_res_to_release_on_suspend', 'pbs_license_max'] 2017-10-21 05:13:07,101 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c unset server pbs_license_min,restrict_res_to_release_on_suspend,pbs_license_max 2017-10-21 05:13:07,120 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-10-21 05:13:07,156 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Qf @linux-kt3k 2017-10-21 05:13:07,175 INFO status on linux-kt3k: node 2017-10-21 05:13:07,175 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v -a 2017-10-21 05:13:07,184 INFO manager on linux-kt3k: delete queue workq 2017-10-21 05:13:07,185 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c delete queue workq 2017-10-21 05:13:07,195 INFO server linux-kt3k: expect offset set to 0.5 2017-10-21 05:13:07,696 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Qf workq@linux-kt3k 2017-10-21 05:13:07,706 INFO expect on server linux-kt3k: unset queue workq ... OK 2017-10-21 05:13:07,706 INFO manager on linux-kt3k: create queue workq {'started': 'True', 'queue_type': 'Execution', 'enabled': 'True'} 2017-10-21 05:13:07,706 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c create queue workq started=True,queue_type=Execution,enabled=True 2017-10-21 05:13:07,716 INFO server linux-kt3k: expect offset set to 0.5 2017-10-21 05:13:08,218 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Qf workq@linux-kt3k 2017-10-21 05:13:08,228 INFO expect on server linux-kt3k: started set True || queue_type set Execution || enabled set True queue workq ... OK 2017-10-21 05:13:08,228 INFOCLI2 linux-kt3k: sudo -H cmp /opt/pbs/lib/python/altair/pbs_hooks/PBS_translate_mpp.HK /var/spool/pbs/server_priv/hooks/PBS_translate_mpp.HK 2017-10-21 05:13:08,247 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /opt/pbs/lib/python/altair/pbs_hooks/PBS_translate_mpp.HK /var/spool/pbs/server_priv/hooks/PBS_translate_mpp.HK 2017-10-21 05:13:08,267 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/server_priv/hooks/PBS_translate_mpp.HK 2017-10-21 05:13:08,288 INFO server linux-kt3k: sent signal -HUP 2017-10-21 05:13:08,288 INFOCLI2 linux-kt3k: sudo -H kill -HUP 31472 2017-10-21 05:13:08,311 INFO manager on linux-kt3k: set pbshook PBS_translate_mpp {'enabled': 'true'} 2017-10-21 05:13:08,311 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c set pbshook PBS_translate_mpp enabled=true 2017-10-21 05:13:08,341 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-10-21 05:13:08,367 INFO manager on linux-kt3k: set server {'default_queue': 'workq', 'restrict_res_to_release_on_suspend': 'ncpus'} 2017-10-21 05:13:08,368 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server default_queue=workq,restrict_res_to_release_on_suspend=ncpus 2017-10-21 05:13:08,377 INFO status on linux-kt3k: resource 2017-10-21 05:13:08,378 INFO manager on linux-kt3k: list resource 2017-10-21 05:13:08,378 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource 2017-10-21 05:13:08,386 INFO manager on linux-kt3k: delete resource PBScraylabel_G34_IL,PBScrayorder,PBScrayhost,PBScraynid 2017-10-21 05:13:08,387 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c delete resource PBScraylabel_G34_IL,PBScrayorder,PBScrayhost,PBScraynid 2017-10-21 05:13:08,458 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource PBScraylabel_G34_IL 2017-10-21 05:13:08,466 INFO expect on server linux-kt3k: unset resource PBScraylabel_G34_IL ... OK 2017-10-21 05:13:08,467 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource PBScrayorder 2017-10-21 05:13:08,475 INFO expect on server linux-kt3k: unset resource PBScrayorder ... OK 2017-10-21 05:13:08,475 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource PBScrayhost 2017-10-21 05:13:08,483 INFO expect on server linux-kt3k: unset resource PBScrayhost ... OK 2017-10-21 05:13:08,484 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource PBScraynid 2017-10-21 05:13:08,492 INFO expect on server linux-kt3k: unset resource PBScraynid ... OK 2017-10-21 05:13:08,492 INFOCLI status on linux-kt3k: server license_count 2017-10-21 05:13:08,492 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-21 05:13:08,501 INFO server: linux-kt3k licensed 2017-10-21 05:13:08,510 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/server_priv/comm.lock 2017-10-21 05:13:08,539 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-10-21 05:13:08,560 INFO scheduler linux-kt3k: reverting configuration to defaults 2017-10-21 05:13:08,560 INFO manager on linux-kt3k: delete sched @default 2017-10-21 05:13:08,560 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c delete sched @default 2017-10-21 05:13:08,569 INFO manager on linux-kt3k: list sched 2017-10-21 05:13:08,569 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list sched 2017-10-21 05:13:08,577 INFO manager on linux-kt3k: unset sched ['sched_preempt_enforce_resumption', 'sched_cycle_length'] 2017-10-21 05:13:08,577 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c unset sched sched_preempt_enforce_resumption,sched_cycle_length 2017-10-21 05:13:08,587 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/dedicated_time 2017-10-21 05:13:08,611 INFOCLI2 linux-kt3k: cmp /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-10-21 05:13:08,613 INFO scheduler linux-kt3k: reverting holidays file to default 2017-10-21 05:13:08,613 INFOCLI2 linux-kt3k: cmp /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-10-21 05:13:08,615 INFOCLI2 linux-kt3k: cmp /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-10-21 05:13:08,617 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-10-21 05:13:08,639 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-10-21 05:13:08,658 INFO scheduler linux-kt3k: sent signal -HUP 2017-10-21 05:13:08,658 INFOCLI2 linux-kt3k: sudo -H kill -HUP 96312 2017-10-21 05:13:08,679 INFOCLI2 linux-kt3k: sudo -H /opt/pbs/sbin/pbsfs -e 2017-10-21 05:13:08,721 INFO scheduler linux-kt3k: config {'resources': '"ncpus, mem, arch, host, vnode, aoe, eoe, vntype"'} 2017-10-21 05:13:08,721 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /var/spool/pbs/sched_priv/sched_config /var/spool/pbs/sched_priv/sched_config.bak 2017-10-21 05:13:08,741 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /tmp/PtlPbsNhYTGd /var/spool/pbs/sched_priv/sched_config 2017-10-21 05:13:08,759 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-10-21 05:13:08,783 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chown root /var/spool/pbs/sched_priv/sched_config 2017-10-21 05:13:08,802 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chgrp root /var/spool/pbs/sched_priv/sched_config 2017-10-21 05:13:08,820 INFO scheduler linux-kt3k: sent signal -HUP 2017-10-21 05:13:08,820 INFOCLI2 linux-kt3k: sudo -H kill -HUP 96312 2017-10-21 05:13:08,841 INFO scheduler linux-kt3k log match: searching for "Error reading line" - No match 2017-10-21 05:13:09,853 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-10-21 05:13:09,879 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-10-21 05:13:09,899 INFO mom linux-kt3k: reverting configuration to defaults 2017-10-21 05:13:09,899 INFOCLI2 linux-kt3k: sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2017-10-21 05:13:09,919 INFOCLI2 linux-kt3k: sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2017-10-21 05:13:09,937 INFOCLI linux-kt3k: sudo -H /opt/pbs/sbin/pbs_mom -s list 2017-10-21 05:13:09,977 INFO mom linux-kt3k: sent signal -HUP 2017-10-21 05:13:09,977 INFOCLI2 linux-kt3k: sudo -H kill -HUP 96290 2017-10-21 05:13:10,007 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-10-21 05:13:10,034 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v -a 2017-10-21 05:13:10,047 INFO manager on linux-kt3k: delete node 2017-10-21 05:13:10,047 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c delete node @default 2017-10-21 05:13:10,061 INFO manager on linux-kt3k: create node linux-kt3k 2017-10-21 05:13:10,061 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c create node linux-kt3k 2017-10-21 05:13:10,072 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k 2017-10-21 05:13:10,095 INFO expect on server linux-kt3k: state = free node linux-kt3k got: state = state-unknown,down 2017-10-21 05:13:11,097 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k 2017-10-21 05:13:11,107 INFO expect on server linux-kt3k: state = free node linux-kt3k attempt: 2 got: state = state-unknown,down 2017-10-21 05:13:12,112 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k 2017-10-21 05:13:12,157 INFO expect on server linux-kt3k: state = free node linux-kt3k attempt: 3 ... OK 2017-10-21 05:13:12,158 INFO ======================================= 2017-10-21 05:13:12,158 INFO Completed TestSuspendResumeOnCray setUp 2017-10-21 05:13:12,158 INFO ======================================= 2017-10-21 05:13:12,158 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-21 05:13:12,168 INFO expect on server linux-kt3k: restrict_res_to_release_on_suspend = ncpus server linux-kt3k ... OK 2017-10-21 05:13:12,168 INFO ========================================= 2017-10-21 05:13:12,168 INFO Entered TestSuspendResumeOnCray tearDown 2017-10-21 05:13:12,168 INFO ========================================= 2017-10-21 05:13:12,168 INFO ========================================== 2017-10-21 05:13:12,169 INFO Completed TestSuspendResumeOnCray tearDown 2017-10-21 05:13:12,169 INFO ========================================== 2017-10-21 05:13:12,169 INFO ok 2017-10-21 05:13:12,169 INFO test name: test_exclusive_job_not_suspended (tests.functional.pbs_cray_suspend_resume.TestSuspendResumeOnCray)... 2017-10-21 05:13:12,170 INFO test start time: Sat Oct 21 05:13:12 2017 2017-10-21 05:13:12,170 INFO test docstring: If a running job is a job with exclusive placement then this job can not be suspended. This test is checking for a log message which is an unstable interface and may need change in future when interface changes. 2017-10-21 05:13:12,170 INFO ====================================== 2017-10-21 05:13:12,170 INFO Entered TestSuspendResumeOnCray setUp 2017-10-21 05:13:12,170 INFO ====================================== 2017-10-21 05:13:12,170 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-21 05:13:12,182 INFO status on linux-kt3k: server 2017-10-21 05:13:12,182 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-21 05:13:12,193 INFO manager on linux-kt3k: set server {'managers': (3, 'root@*')} 2017-10-21 05:13:12,193 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c set server managers-=root@* 2017-10-21 05:13:12,240 INFO manager on linux-kt3k: set server {'managers': (2, 'root@*')} 2017-10-21 05:13:12,240 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c set server managers+=root@* 2017-10-21 05:13:12,273 INFO server linux-kt3k: reverting configuration to defaults 2017-10-21 05:13:12,274 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-21 05:13:12,284 INFO select on linux-kt3k: __ALL__ 2017-10-21 05:13:12,284 INFOCLI linux-kt3k: /opt/pbs/bin/qselect 2017-10-21 05:13:12,296 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f @linux-kt3k 2017-10-21 05:13:12,303 INFO expect on server linux-kt3k: job_state set 0 job ... OK 2017-10-21 05:13:12,303 INFOCLI linux-kt3k: /opt/pbs/bin/pbs_rstat -f 2017-10-21 05:13:12,314 INFO manager on linux-kt3k: unset server ['restrict_res_to_release_on_suspend'] 2017-10-21 05:13:12,314 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c unset server restrict_res_to_release_on_suspend 2017-10-21 05:13:12,329 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-10-21 05:13:12,366 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Qf @linux-kt3k 2017-10-21 05:13:12,374 INFO status on linux-kt3k: node 2017-10-21 05:13:12,374 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v -a 2017-10-21 05:13:12,382 INFO manager on linux-kt3k: delete queue workq 2017-10-21 05:13:12,383 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c delete queue workq 2017-10-21 05:13:12,392 INFO server linux-kt3k: expect offset set to 0.5 2017-10-21 05:13:12,894 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Qf workq@linux-kt3k 2017-10-21 05:13:12,903 INFO expect on server linux-kt3k: unset queue workq ... OK 2017-10-21 05:13:12,903 INFO manager on linux-kt3k: create queue workq {'started': 'True', 'queue_type': 'Execution', 'enabled': 'True'} 2017-10-21 05:13:12,903 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c create queue workq started=True,queue_type=Execution,enabled=True 2017-10-21 05:13:12,914 INFO server linux-kt3k: expect offset set to 0.5 2017-10-21 05:13:13,415 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Qf workq@linux-kt3k 2017-10-21 05:13:13,424 INFO expect on server linux-kt3k: started set True || queue_type set Execution || enabled set True queue workq ... OK 2017-10-21 05:13:13,424 INFOCLI2 linux-kt3k: sudo -H cmp /opt/pbs/lib/python/altair/pbs_hooks/PBS_translate_mpp.HK /var/spool/pbs/server_priv/hooks/PBS_translate_mpp.HK 2017-10-21 05:13:13,446 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /opt/pbs/lib/python/altair/pbs_hooks/PBS_translate_mpp.HK /var/spool/pbs/server_priv/hooks/PBS_translate_mpp.HK 2017-10-21 05:13:13,474 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/server_priv/hooks/PBS_translate_mpp.HK 2017-10-21 05:13:13,497 INFO server linux-kt3k: sent signal -HUP 2017-10-21 05:13:13,497 INFOCLI2 linux-kt3k: sudo -H kill -HUP 31472 2017-10-21 05:13:13,516 INFO manager on linux-kt3k: set pbshook PBS_translate_mpp {'enabled': 'true'} 2017-10-21 05:13:13,516 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c set pbshook PBS_translate_mpp enabled=true 2017-10-21 05:13:13,542 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-10-21 05:13:13,572 INFO manager on linux-kt3k: set server {'default_queue': 'workq', 'restrict_res_to_release_on_suspend': 'ncpus'} 2017-10-21 05:13:13,572 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server default_queue=workq,restrict_res_to_release_on_suspend=ncpus 2017-10-21 05:13:13,586 INFO status on linux-kt3k: resource 2017-10-21 05:13:13,587 INFO manager on linux-kt3k: list resource 2017-10-21 05:13:13,587 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource 2017-10-21 05:13:13,598 INFO manager on linux-kt3k: delete resource PBScraylabel_G34_IL,PBScrayorder,PBScrayhost,PBScraynid 2017-10-21 05:13:13,598 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c delete resource PBScraylabel_G34_IL,PBScrayorder,PBScrayhost,PBScraynid 2017-10-21 05:13:13,676 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource PBScraylabel_G34_IL 2017-10-21 05:13:13,685 INFO expect on server linux-kt3k: unset resource PBScraylabel_G34_IL ... OK 2017-10-21 05:13:13,686 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource PBScrayorder 2017-10-21 05:13:13,694 INFO expect on server linux-kt3k: unset resource PBScrayorder ... OK 2017-10-21 05:13:13,694 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource PBScrayhost 2017-10-21 05:13:13,703 INFO expect on server linux-kt3k: unset resource PBScrayhost ... OK 2017-10-21 05:13:13,703 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource PBScraynid 2017-10-21 05:13:13,710 INFO expect on server linux-kt3k: unset resource PBScraynid ... OK 2017-10-21 05:13:13,711 INFOCLI status on linux-kt3k: server license_count 2017-10-21 05:13:13,711 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-21 05:13:13,719 INFO server: linux-kt3k licensed 2017-10-21 05:13:13,728 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/server_priv/comm.lock 2017-10-21 05:13:13,754 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-10-21 05:13:13,772 INFO scheduler linux-kt3k: reverting configuration to defaults 2017-10-21 05:13:13,773 INFO manager on linux-kt3k: delete sched @default 2017-10-21 05:13:13,773 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c delete sched @default 2017-10-21 05:13:13,782 INFO manager on linux-kt3k: list sched 2017-10-21 05:13:13,782 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list sched 2017-10-21 05:13:13,791 INFO manager on linux-kt3k: unset sched ['sched_preempt_enforce_resumption', 'sched_cycle_length'] 2017-10-21 05:13:13,791 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c unset sched sched_preempt_enforce_resumption,sched_cycle_length 2017-10-21 05:13:13,801 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/dedicated_time 2017-10-21 05:13:13,824 INFOCLI2 linux-kt3k: cmp /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-10-21 05:13:13,826 INFO scheduler linux-kt3k: reverting holidays file to default 2017-10-21 05:13:13,826 INFOCLI2 linux-kt3k: cmp /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-10-21 05:13:13,828 INFOCLI2 linux-kt3k: cmp /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-10-21 05:13:13,830 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-10-21 05:13:13,851 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-10-21 05:13:13,869 INFO scheduler linux-kt3k: sent signal -HUP 2017-10-21 05:13:13,869 INFOCLI2 linux-kt3k: sudo -H kill -HUP 96312 2017-10-21 05:13:13,889 INFOCLI2 linux-kt3k: sudo -H /opt/pbs/sbin/pbsfs -e 2017-10-21 05:13:13,940 INFO scheduler linux-kt3k: config {'resources': '"ncpus, mem, arch, host, vnode, aoe, eoe, vntype"'} 2017-10-21 05:13:13,940 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /var/spool/pbs/sched_priv/sched_config /var/spool/pbs/sched_priv/sched_config.bak 2017-10-21 05:13:13,962 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /tmp/PtlPbs0T_ehp /var/spool/pbs/sched_priv/sched_config 2017-10-21 05:13:13,982 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-10-21 05:13:14,000 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chown root /var/spool/pbs/sched_priv/sched_config 2017-10-21 05:13:14,019 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chgrp root /var/spool/pbs/sched_priv/sched_config 2017-10-21 05:13:14,038 INFO scheduler linux-kt3k: sent signal -HUP 2017-10-21 05:13:14,038 INFOCLI2 linux-kt3k: sudo -H kill -HUP 96312 2017-10-21 05:13:14,060 INFO scheduler linux-kt3k log match: searching for "Error reading line" - No match 2017-10-21 05:13:15,073 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-10-21 05:13:15,102 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-10-21 05:13:15,122 INFO mom linux-kt3k: reverting configuration to defaults 2017-10-21 05:13:15,123 INFOCLI2 linux-kt3k: sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2017-10-21 05:13:15,146 INFOCLI2 linux-kt3k: sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2017-10-21 05:13:15,165 INFOCLI linux-kt3k: sudo -H /opt/pbs/sbin/pbs_mom -s list 2017-10-21 05:13:15,202 INFO mom linux-kt3k: sent signal -HUP 2017-10-21 05:13:15,203 INFOCLI2 linux-kt3k: sudo -H kill -HUP 96290 2017-10-21 05:13:15,238 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-10-21 05:13:15,263 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v -a 2017-10-21 05:13:15,273 INFO manager on linux-kt3k: delete node 2017-10-21 05:13:15,273 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c delete node @default 2017-10-21 05:13:15,288 INFO manager on linux-kt3k: create node linux-kt3k 2017-10-21 05:13:15,289 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c create node linux-kt3k 2017-10-21 05:13:15,308 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k 2017-10-21 05:13:15,326 INFO expect on server linux-kt3k: state = free node linux-kt3k got: state = state-unknown,down 2017-10-21 05:13:16,328 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k 2017-10-21 05:13:16,338 INFO expect on server linux-kt3k: state = free node linux-kt3k attempt: 2 got: state = state-unknown,down 2017-10-21 05:13:17,345 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k 2017-10-21 05:13:17,377 INFO expect on server linux-kt3k: state = free node linux-kt3k attempt: 3 ... OK 2017-10-21 05:13:17,377 INFO ======================================= 2017-10-21 05:13:17,377 INFO Completed TestSuspendResumeOnCray setUp 2017-10-21 05:13:17,378 INFO ======================================= 2017-10-21 05:13:17,378 INFO job: executable set to /bin/sleep with arguments: 100 2017-10-21 05:13:17,378 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0755 /tmp/PtlPbsJobScript84hgHe 2017-10-21 05:13:17,398 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chown pbsuser /tmp/PtlPbsJobScript84hgHe 2017-10-21 05:13:17,420 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chgrp tstgrp00 /tmp/PtlPbsJobScript84hgHe 2017-10-21 05:13:17,439 INFOCLI linux-kt3k: sudo -H -u pbsuser /opt/pbs/bin/qsub -l place=excl -l select=1:ncpus=1:vntype=cray_compute /tmp/PtlPbsJobScript84hgHe 2017-10-21 05:13:17,474 INFO submit to linux-kt3k as pbsuser: job 95.linux-kt3k OrderedDict([('Resource_List.place', 'excl'), ('Resource_List.select', '1:ncpus=1:vntype=cray_compute')]) 2017-10-21 05:13:17,475 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 95.linux-kt3k 2017-10-21 05:13:17,484 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 95.linux-kt3k got: job_state = Q 2017-10-21 05:13:17,986 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:13:17,998 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 95.linux-kt3k 2017-10-21 05:13:18,007 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 95.linux-kt3k attempt: 2 ... OK 2017-10-21 05:13:18,008 INFO signal on linux-kt3k: 95.linux-kt3k with signal = suspend 2017-10-21 05:13:18,008 INFOCLI linux-kt3k: /opt/pbs/bin/qsig -s suspend 95.linux-kt3k 2017-10-21 05:13:18,051 ERROR err: ['qsig: Switching ALPS reservation failed 95.linux-kt3k'] 2017-10-21 05:13:18,052 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 95.linux-kt3k 2017-10-21 05:13:18,061 INFO expect on server linux-kt3k: exec_host set job 95.linux-kt3k ... OK 2017-10-21 05:13:18,061 INFO status on linux-kt3k: job 95.linux-kt3k 2017-10-21 05:13:18,061 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 95.linux-kt3k 2017-10-21 05:13:18,070 INFO mom linux-kt3k log match: searching for "BASIL;ERROR: ALPS error: apsched: at least resid .* is exclusive" - using regular expression ... OK 2017-10-21 05:13:18,070 INFO ========================================= 2017-10-21 05:13:18,070 INFO Entered TestSuspendResumeOnCray tearDown 2017-10-21 05:13:18,070 INFO ========================================= 2017-10-21 05:13:18,070 INFO ========================================== 2017-10-21 05:13:18,070 INFO Completed TestSuspendResumeOnCray tearDown 2017-10-21 05:13:18,070 INFO ========================================== 2017-10-21 05:13:18,070 INFO ok 2017-10-21 05:13:18,071 INFO test name: test_job_accounted_limit_usage (tests.functional.pbs_cray_suspend_resume.TestSuspendResumeOnCray)... 2017-10-21 05:13:18,071 INFO test start time: Sat Oct 21 05:13:18 2017 2017-10-21 05:13:18,071 INFO test docstring: Test that a job is correctly accounted limit usage 2017-10-21 05:13:18,071 INFO ====================================== 2017-10-21 05:13:18,071 INFO Entered TestSuspendResumeOnCray setUp 2017-10-21 05:13:18,071 INFO ====================================== 2017-10-21 05:13:18,071 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-21 05:13:18,079 INFO status on linux-kt3k: server 2017-10-21 05:13:18,079 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-21 05:13:18,087 INFO manager on linux-kt3k: set server {'managers': (3, 'root@*')} 2017-10-21 05:13:18,088 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c set server managers-=root@* 2017-10-21 05:13:18,119 INFO manager on linux-kt3k: set server {'managers': (2, 'root@*')} 2017-10-21 05:13:18,119 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c set server managers+=root@* 2017-10-21 05:13:18,148 INFO server linux-kt3k: reverting configuration to defaults 2017-10-21 05:13:18,149 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-21 05:13:18,158 INFO select on linux-kt3k: __ALL__ 2017-10-21 05:13:18,158 INFOCLI linux-kt3k: /opt/pbs/bin/qselect 2017-10-21 05:13:18,168 INFO delete job on linux-kt3k: 95.linux-kt3k 2017-10-21 05:13:18,168 INFOCLI linux-kt3k: /opt/pbs/bin/qdel -W force 95.linux-kt3k 2017-10-21 05:13:18,188 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 95.linux-kt3k 2017-10-21 05:13:18,207 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f @linux-kt3k 2017-10-21 05:13:18,221 INFO expect on server linux-kt3k: job_state set 0 job ... OK 2017-10-21 05:13:18,222 INFOCLI linux-kt3k: /opt/pbs/bin/pbs_rstat -f 2017-10-21 05:13:18,239 INFO manager on linux-kt3k: unset server ['restrict_res_to_release_on_suspend'] 2017-10-21 05:13:18,239 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c unset server restrict_res_to_release_on_suspend 2017-10-21 05:13:18,257 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-10-21 05:13:18,288 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Qf @linux-kt3k 2017-10-21 05:13:18,305 INFO status on linux-kt3k: node 2017-10-21 05:13:18,305 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v -a 2017-10-21 05:13:18,315 INFO manager on linux-kt3k: delete queue workq 2017-10-21 05:13:18,315 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c delete queue workq 2017-10-21 05:13:18,328 INFO server linux-kt3k: expect offset set to 0.5 2017-10-21 05:13:18,829 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Qf workq@linux-kt3k 2017-10-21 05:13:18,839 INFO expect on server linux-kt3k: unset queue workq ... OK 2017-10-21 05:13:18,839 INFO manager on linux-kt3k: create queue workq {'started': 'True', 'queue_type': 'Execution', 'enabled': 'True'} 2017-10-21 05:13:18,839 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c create queue workq started=True,queue_type=Execution,enabled=True 2017-10-21 05:13:18,849 INFO server linux-kt3k: expect offset set to 0.5 2017-10-21 05:13:19,351 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Qf workq@linux-kt3k 2017-10-21 05:13:19,360 INFO expect on server linux-kt3k: started set True || queue_type set Execution || enabled set True queue workq ... OK 2017-10-21 05:13:19,360 INFOCLI2 linux-kt3k: sudo -H cmp /opt/pbs/lib/python/altair/pbs_hooks/PBS_translate_mpp.HK /var/spool/pbs/server_priv/hooks/PBS_translate_mpp.HK 2017-10-21 05:13:19,379 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /opt/pbs/lib/python/altair/pbs_hooks/PBS_translate_mpp.HK /var/spool/pbs/server_priv/hooks/PBS_translate_mpp.HK 2017-10-21 05:13:19,399 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/server_priv/hooks/PBS_translate_mpp.HK 2017-10-21 05:13:19,425 INFO server linux-kt3k: sent signal -HUP 2017-10-21 05:13:19,425 INFOCLI2 linux-kt3k: sudo -H kill -HUP 31472 2017-10-21 05:13:19,444 INFO manager on linux-kt3k: set pbshook PBS_translate_mpp {'enabled': 'true'} 2017-10-21 05:13:19,444 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c set pbshook PBS_translate_mpp enabled=true 2017-10-21 05:13:19,470 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-10-21 05:13:19,495 INFO manager on linux-kt3k: set server {'default_queue': 'workq', 'restrict_res_to_release_on_suspend': 'ncpus'} 2017-10-21 05:13:19,495 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server default_queue=workq,restrict_res_to_release_on_suspend=ncpus 2017-10-21 05:13:19,505 INFO status on linux-kt3k: resource 2017-10-21 05:13:19,506 INFO manager on linux-kt3k: list resource 2017-10-21 05:13:19,506 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource 2017-10-21 05:13:19,518 INFO manager on linux-kt3k: delete resource PBScraylabel_G34_IL,PBScrayorder,PBScrayhost,PBScraynid 2017-10-21 05:13:19,519 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c delete resource PBScraylabel_G34_IL,PBScrayorder,PBScrayhost,PBScraynid 2017-10-21 05:13:19,607 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource PBScraylabel_G34_IL 2017-10-21 05:13:19,619 INFO expect on server linux-kt3k: unset resource PBScraylabel_G34_IL ... OK 2017-10-21 05:13:19,619 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource PBScrayorder 2017-10-21 05:13:19,632 INFO expect on server linux-kt3k: unset resource PBScrayorder ... OK 2017-10-21 05:13:19,633 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource PBScrayhost 2017-10-21 05:13:19,641 INFO expect on server linux-kt3k: unset resource PBScrayhost ... OK 2017-10-21 05:13:19,641 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource PBScraynid 2017-10-21 05:13:19,650 INFO expect on server linux-kt3k: unset resource PBScraynid ... OK 2017-10-21 05:13:19,651 INFOCLI status on linux-kt3k: server license_count 2017-10-21 05:13:19,652 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-21 05:13:19,664 INFO server: linux-kt3k licensed 2017-10-21 05:13:19,675 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/server_priv/comm.lock 2017-10-21 05:13:19,702 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-10-21 05:13:19,729 INFO scheduler linux-kt3k: reverting configuration to defaults 2017-10-21 05:13:19,729 INFO manager on linux-kt3k: delete sched @default 2017-10-21 05:13:19,729 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c delete sched @default 2017-10-21 05:13:19,741 INFO manager on linux-kt3k: list sched 2017-10-21 05:13:19,741 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list sched 2017-10-21 05:13:19,751 INFO manager on linux-kt3k: unset sched ['sched_preempt_enforce_resumption', 'sched_cycle_length'] 2017-10-21 05:13:19,752 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c unset sched sched_preempt_enforce_resumption,sched_cycle_length 2017-10-21 05:13:19,765 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/dedicated_time 2017-10-21 05:13:19,794 INFOCLI2 linux-kt3k: cmp /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-10-21 05:13:19,796 INFO scheduler linux-kt3k: reverting holidays file to default 2017-10-21 05:13:19,797 INFOCLI2 linux-kt3k: cmp /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-10-21 05:13:19,798 INFOCLI2 linux-kt3k: cmp /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-10-21 05:13:19,800 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-10-21 05:13:19,822 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-10-21 05:13:19,853 INFO scheduler linux-kt3k: sent signal -HUP 2017-10-21 05:13:19,853 INFOCLI2 linux-kt3k: sudo -H kill -HUP 96312 2017-10-21 05:13:19,874 INFOCLI2 linux-kt3k: sudo -H /opt/pbs/sbin/pbsfs -e 2017-10-21 05:13:19,925 INFO scheduler linux-kt3k: config {'resources': '"ncpus, mem, arch, host, vnode, aoe, eoe, vntype"'} 2017-10-21 05:13:19,925 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /var/spool/pbs/sched_priv/sched_config /var/spool/pbs/sched_priv/sched_config.bak 2017-10-21 05:13:19,946 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /tmp/PtlPbsbyxK3p /var/spool/pbs/sched_priv/sched_config 2017-10-21 05:13:19,979 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-10-21 05:13:20,006 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chown root /var/spool/pbs/sched_priv/sched_config 2017-10-21 05:13:20,030 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chgrp root /var/spool/pbs/sched_priv/sched_config 2017-10-21 05:13:20,049 INFO scheduler linux-kt3k: sent signal -HUP 2017-10-21 05:13:20,050 INFOCLI2 linux-kt3k: sudo -H kill -HUP 96312 2017-10-21 05:13:20,076 INFO scheduler linux-kt3k log match: searching for "Error reading line" - No match 2017-10-21 05:13:21,090 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-10-21 05:13:21,128 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-10-21 05:13:21,151 INFO mom linux-kt3k: reverting configuration to defaults 2017-10-21 05:13:21,152 INFOCLI2 linux-kt3k: sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2017-10-21 05:13:21,184 INFOCLI2 linux-kt3k: sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2017-10-21 05:13:21,215 INFOCLI linux-kt3k: sudo -H /opt/pbs/sbin/pbs_mom -s list 2017-10-21 05:13:21,261 INFO mom linux-kt3k: sent signal -HUP 2017-10-21 05:13:21,261 INFOCLI2 linux-kt3k: sudo -H kill -HUP 96290 2017-10-21 05:13:21,298 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-10-21 05:13:21,336 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v -a 2017-10-21 05:13:21,353 INFO manager on linux-kt3k: delete node 2017-10-21 05:13:21,353 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c delete node @default 2017-10-21 05:13:21,379 INFO manager on linux-kt3k: create node linux-kt3k 2017-10-21 05:13:21,380 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c create node linux-kt3k 2017-10-21 05:13:21,416 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k 2017-10-21 05:13:21,444 INFO expect on server linux-kt3k: state = free node linux-kt3k got: state = state-unknown,down 2017-10-21 05:13:22,446 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k 2017-10-21 05:13:22,458 INFO expect on server linux-kt3k: state = free node linux-kt3k attempt: 2 got: state = state-unknown,down 2017-10-21 05:13:23,465 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k 2017-10-21 05:13:23,495 INFO expect on server linux-kt3k: state = free node linux-kt3k attempt: 3 ... OK 2017-10-21 05:13:23,495 INFO ======================================= 2017-10-21 05:13:23,495 INFO Completed TestSuspendResumeOnCray setUp 2017-10-21 05:13:23,495 INFO ======================================= 2017-10-21 05:13:23,495 INFO status on linux-kt3k: queue expressq 2017-10-21 05:13:23,495 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Qf expressq@linux-kt3k 2017-10-21 05:13:23,503 INFO manager on linux-kt3k: create queue expressq {'priority': 150, 'started': 'True', 'queue_type': 'execution', 'enabled': 'True'} 2017-10-21 05:13:23,503 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c create queue expressq priority=150,started=True,queue_type=execution,enabled=True 2017-10-21 05:13:23,514 INFO manager on linux-kt3k: set server {'eligible_time_enable': 1, 'max_run_res_soft.ncpus': '[p:PBS_GENERIC=1]'} 2017-10-21 05:13:23,514 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server eligible_time_enable=1,max_run_res_soft.ncpus=[p:PBS_GENERIC=1] 2017-10-21 05:13:23,529 INFO job: executable set to /bin/sleep with arguments: 100 2017-10-21 05:13:23,530 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0755 /tmp/PtlPbsJobScriptXe0wvQ 2017-10-21 05:13:23,550 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chown pbsuser /tmp/PtlPbsJobScriptXe0wvQ 2017-10-21 05:13:23,568 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chgrp tstgrp00 /tmp/PtlPbsJobScriptXe0wvQ 2017-10-21 05:13:23,592 INFOCLI linux-kt3k: sudo -H -u pbsuser /opt/pbs/bin/qsub -P A -l walltime=3600 -l select=1:ncpus=1:vntype=cray_compute /tmp/PtlPbsJobScriptXe0wvQ 2017-10-21 05:13:23,620 INFO submit to linux-kt3k as pbsuser: job 96.linux-kt3k OrderedDict([('project', 'A'), ('Resource_List.walltime', '3600'), ('Resource_List.select', '1:ncpus=1:vntype=cray_compute')]) 2017-10-21 05:13:24,623 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 96.linux-kt3k 2017-10-21 05:13:24,633 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 96.linux-kt3k ... OK 2017-10-21 05:13:24,633 INFO job: executable set to /bin/sleep with arguments: 100 2017-10-21 05:13:24,633 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0755 /tmp/PtlPbsJobScript6wfYae 2017-10-21 05:13:24,652 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chown pbsuser /tmp/PtlPbsJobScript6wfYae 2017-10-21 05:13:24,678 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chgrp tstgrp00 /tmp/PtlPbsJobScript6wfYae 2017-10-21 05:13:24,698 INFOCLI linux-kt3k: sudo -H -u pbsuser /opt/pbs/bin/qsub -P A -l walltime=3600 -l select=1:ncpus=1:vntype=cray_compute /tmp/PtlPbsJobScript6wfYae 2017-10-21 05:13:24,723 INFO submit to linux-kt3k as pbsuser: job 97.linux-kt3k OrderedDict([('project', 'A'), ('Resource_List.walltime', '3600'), ('Resource_List.select', '1:ncpus=1:vntype=cray_compute')]) 2017-10-21 05:13:24,723 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 97.linux-kt3k 2017-10-21 05:13:24,735 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 97.linux-kt3k got: job_state = Q 2017-10-21 05:13:25,236 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:13:25,248 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 97.linux-kt3k 2017-10-21 05:13:25,259 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 97.linux-kt3k attempt: 2 ... OK 2017-10-21 05:13:25,259 INFO job: executable set to /bin/sleep with arguments: 100 2017-10-21 05:13:25,260 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0755 /tmp/PtlPbsJobScript3Xm8OV 2017-10-21 05:13:25,280 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chown pbsuser /tmp/PtlPbsJobScript3Xm8OV 2017-10-21 05:13:25,303 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chgrp tstgrp00 /tmp/PtlPbsJobScript3Xm8OV 2017-10-21 05:13:25,323 INFOCLI linux-kt3k: sudo -H -u pbsuser /opt/pbs/bin/qsub -P B -q expressq -l walltime=3600 -l select=1:ncpus=1:vntype=cray_compute /tmp/PtlPbsJobScript3Xm8OV 2017-10-21 05:13:25,347 INFO submit to linux-kt3k as pbsuser: job 98.linux-kt3k OrderedDict([('project', 'B'), ('queue', 'expressq'), ('Resource_List.walltime', '3600'), ('Resource_List.select', '1:ncpus=1:vntype=cray_compute')]) 2017-10-21 05:13:25,348 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 98.linux-kt3k 2017-10-21 05:13:25,356 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 98.linux-kt3k got: job_state = Q 2017-10-21 05:13:25,858 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:13:25,869 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 98.linux-kt3k 2017-10-21 05:13:25,878 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 98.linux-kt3k attempt: 2 got: job_state = Q 2017-10-21 05:13:26,379 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:13:26,391 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 98.linux-kt3k 2017-10-21 05:13:26,402 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 98.linux-kt3k attempt: 3 got: job_state = Q 2017-10-21 05:13:26,905 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:13:26,916 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 98.linux-kt3k 2017-10-21 05:13:26,925 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 98.linux-kt3k attempt: 4 got: job_state = Q 2017-10-21 05:13:27,427 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:13:27,438 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 98.linux-kt3k 2017-10-21 05:13:27,446 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 98.linux-kt3k attempt: 5 got: job_state = Q 2017-10-21 05:13:27,947 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:13:27,958 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 98.linux-kt3k 2017-10-21 05:13:27,967 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 98.linux-kt3k attempt: 6 got: job_state = Q 2017-10-21 05:13:28,468 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:13:28,481 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 98.linux-kt3k 2017-10-21 05:13:28,491 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 98.linux-kt3k attempt: 7 got: job_state = Q 2017-10-21 05:13:28,993 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:13:29,006 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 98.linux-kt3k 2017-10-21 05:13:29,014 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 98.linux-kt3k attempt: 8 got: job_state = Q 2017-10-21 05:13:29,516 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:13:29,528 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 98.linux-kt3k 2017-10-21 05:13:29,537 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 98.linux-kt3k attempt: 9 got: job_state = Q 2017-10-21 05:13:30,038 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:13:30,050 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 98.linux-kt3k 2017-10-21 05:13:30,059 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 98.linux-kt3k attempt: 10 got: job_state = Q 2017-10-21 05:13:30,560 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:13:30,574 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 98.linux-kt3k 2017-10-21 05:13:30,586 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 98.linux-kt3k attempt: 11 got: job_state = Q 2017-10-21 05:13:31,087 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:13:31,099 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 98.linux-kt3k 2017-10-21 05:13:31,108 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 98.linux-kt3k attempt: 12 got: job_state = Q 2017-10-21 05:13:31,610 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:13:31,622 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 98.linux-kt3k 2017-10-21 05:13:31,631 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 98.linux-kt3k attempt: 13 got: job_state = Q 2017-10-21 05:13:32,133 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:13:32,144 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 98.linux-kt3k 2017-10-21 05:13:32,153 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 98.linux-kt3k attempt: 14 got: job_state = Q 2017-10-21 05:13:32,655 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:13:32,665 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 98.linux-kt3k 2017-10-21 05:13:32,678 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 98.linux-kt3k attempt: 15 got: job_state = Q 2017-10-21 05:13:33,180 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:13:33,193 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 98.linux-kt3k 2017-10-21 05:13:33,202 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 98.linux-kt3k attempt: 16 got: job_state = Q 2017-10-21 05:13:33,703 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:13:33,715 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 98.linux-kt3k 2017-10-21 05:13:33,726 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 98.linux-kt3k attempt: 17 got: job_state = Q 2017-10-21 05:13:34,227 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:13:34,238 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 98.linux-kt3k 2017-10-21 05:13:34,247 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 98.linux-kt3k attempt: 18 got: job_state = Q 2017-10-21 05:13:34,749 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:13:34,761 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 98.linux-kt3k 2017-10-21 05:13:34,774 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 98.linux-kt3k attempt: 19 got: job_state = Q 2017-10-21 05:13:35,277 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:13:35,304 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 98.linux-kt3k 2017-10-21 05:13:35,316 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 98.linux-kt3k attempt: 20 ... OK 2017-10-21 05:13:35,317 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 97.linux-kt3k 2017-10-21 05:13:35,331 INFO expect on server linux-kt3k: job_state = S job 97.linux-kt3k ... OK 2017-10-21 05:13:35,331 INFO job: executable set to /bin/sleep with arguments: 100 2017-10-21 05:13:35,332 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0755 /tmp/PtlPbsJobScriptucwghk 2017-10-21 05:13:35,369 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chown pbsuser /tmp/PtlPbsJobScriptucwghk 2017-10-21 05:13:35,403 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chgrp tstgrp00 /tmp/PtlPbsJobScriptucwghk 2017-10-21 05:13:35,433 INFOCLI linux-kt3k: sudo -H -u pbsuser /opt/pbs/bin/qsub -P B -l walltime=3600 -l select=1:ncpus=1:vntype=cray_compute /tmp/PtlPbsJobScriptucwghk 2017-10-21 05:13:35,459 INFO submit to linux-kt3k as pbsuser: job 99.linux-kt3k OrderedDict([('project', 'B'), ('Resource_List.walltime', '3600'), ('Resource_List.select', '1:ncpus=1:vntype=cray_compute')]) 2017-10-21 05:13:35,459 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 99.linux-kt3k 2017-10-21 05:13:35,469 INFO expect on server linux-kt3k: job_state = Q job 99.linux-kt3k ... OK 2017-10-21 05:13:35,470 INFO ========================================= 2017-10-21 05:13:35,470 INFO Entered TestSuspendResumeOnCray tearDown 2017-10-21 05:13:35,470 INFO ========================================= 2017-10-21 05:13:35,470 INFO ========================================== 2017-10-21 05:13:35,470 INFO Completed TestSuspendResumeOnCray tearDown 2017-10-21 05:13:35,470 INFO ========================================== 2017-10-21 05:13:35,470 INFO ok 2017-10-21 05:13:35,470 INFO test name: test_multi_express (tests.functional.pbs_cray_suspend_resume.TestSuspendResumeOnCray)... 2017-10-21 05:13:35,470 INFO test start time: Sat Oct 21 05:13:35 2017 2017-10-21 05:13:35,471 INFO test docstring: Test of multiple express queues of different priorities. See that jobs from the higher express queues preempt jobs from lower express queues. Also see when express jobs finish (or are deleted), suspended jobs restart. Make sure loadLimit is set to 4 on the server node: # apmgr config loadLimit 4 2017-10-21 05:13:35,471 INFO ====================================== 2017-10-21 05:13:35,471 INFO Entered TestSuspendResumeOnCray setUp 2017-10-21 05:13:35,471 INFO ====================================== 2017-10-21 05:13:35,471 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-21 05:13:35,480 INFO status on linux-kt3k: server 2017-10-21 05:13:35,480 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-21 05:13:35,490 INFO manager on linux-kt3k: set server {'managers': (3, 'root@*')} 2017-10-21 05:13:35,490 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c set server managers-=root@* 2017-10-21 05:13:35,520 INFO manager on linux-kt3k: set server {'managers': (2, 'root@*')} 2017-10-21 05:13:35,521 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c set server managers+=root@* 2017-10-21 05:13:35,547 INFO server linux-kt3k: reverting configuration to defaults 2017-10-21 05:13:35,548 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-21 05:13:35,557 INFO select on linux-kt3k: __ALL__ 2017-10-21 05:13:35,557 INFOCLI linux-kt3k: /opt/pbs/bin/qselect 2017-10-21 05:13:35,566 INFO delete job on linux-kt3k: 96.linux-kt3k, 97.linux-kt3k, 98.linux-kt3k, 99.linux-kt3k 2017-10-21 05:13:35,566 INFOCLI linux-kt3k: /opt/pbs/bin/qdel -W force 96.linux-kt3k 97.linux-kt3k 98.linux-kt3k 99.linux-kt3k 2017-10-21 05:13:35,654 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 96.linux-kt3k 2017-10-21 05:13:35,680 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 97.linux-kt3k 2017-10-21 05:13:35,697 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 98.linux-kt3k 2017-10-21 05:13:35,720 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 99.linux-kt3k 2017-10-21 05:13:35,739 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f @linux-kt3k 2017-10-21 05:13:35,747 INFO expect on server linux-kt3k: job_state set 0 job ... OK 2017-10-21 05:13:35,747 INFOCLI linux-kt3k: /opt/pbs/bin/pbs_rstat -f 2017-10-21 05:13:35,759 INFO manager on linux-kt3k: unset server ['restrict_res_to_release_on_suspend', 'max_run_res_soft.ncpus'] 2017-10-21 05:13:35,759 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c unset server restrict_res_to_release_on_suspend,max_run_res_soft.ncpus 2017-10-21 05:13:35,785 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-10-21 05:13:35,827 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Qf @linux-kt3k 2017-10-21 05:13:35,842 INFO status on linux-kt3k: node 2017-10-21 05:13:35,843 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v -a 2017-10-21 05:13:35,851 INFO manager on linux-kt3k: delete queue workq,expressq 2017-10-21 05:13:35,852 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c delete queue workq,expressq 2017-10-21 05:13:35,863 INFO server linux-kt3k: expect offset set to 0.5 2017-10-21 05:13:36,364 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Qf workq@linux-kt3k 2017-10-21 05:13:36,373 INFO expect on server linux-kt3k: unset queue workq ... OK 2017-10-21 05:13:36,373 INFO server linux-kt3k: expect offset set to 0.5 2017-10-21 05:13:36,874 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Qf expressq@linux-kt3k 2017-10-21 05:13:36,885 INFO expect on server linux-kt3k: unset queue expressq ... OK 2017-10-21 05:13:36,885 INFO manager on linux-kt3k: create queue workq {'started': 'True', 'queue_type': 'Execution', 'enabled': 'True'} 2017-10-21 05:13:36,885 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c create queue workq started=True,queue_type=Execution,enabled=True 2017-10-21 05:13:36,898 INFO server linux-kt3k: expect offset set to 0.5 2017-10-21 05:13:37,401 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Qf workq@linux-kt3k 2017-10-21 05:13:37,413 INFO expect on server linux-kt3k: started set True || queue_type set Execution || enabled set True queue workq ... OK 2017-10-21 05:13:37,413 INFOCLI2 linux-kt3k: sudo -H cmp /opt/pbs/lib/python/altair/pbs_hooks/PBS_translate_mpp.HK /var/spool/pbs/server_priv/hooks/PBS_translate_mpp.HK 2017-10-21 05:13:37,435 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /opt/pbs/lib/python/altair/pbs_hooks/PBS_translate_mpp.HK /var/spool/pbs/server_priv/hooks/PBS_translate_mpp.HK 2017-10-21 05:13:37,456 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/server_priv/hooks/PBS_translate_mpp.HK 2017-10-21 05:13:37,476 INFO server linux-kt3k: sent signal -HUP 2017-10-21 05:13:37,476 INFOCLI2 linux-kt3k: sudo -H kill -HUP 31472 2017-10-21 05:13:37,501 INFO manager on linux-kt3k: set pbshook PBS_translate_mpp {'enabled': 'true'} 2017-10-21 05:13:37,501 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c set pbshook PBS_translate_mpp enabled=true 2017-10-21 05:13:37,535 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-10-21 05:13:37,562 INFO manager on linux-kt3k: set server {'eligible_time_enable': 'False', 'default_queue': 'workq', 'restrict_res_to_release_on_suspend': 'ncpus'} 2017-10-21 05:13:37,563 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server eligible_time_enable=False,default_queue=workq,restrict_res_to_release_on_suspend=ncpus 2017-10-21 05:13:37,577 INFO status on linux-kt3k: resource 2017-10-21 05:13:37,578 INFO manager on linux-kt3k: list resource 2017-10-21 05:13:37,578 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource 2017-10-21 05:13:37,592 INFO manager on linux-kt3k: delete resource PBScraylabel_G34_IL,PBScrayorder,PBScrayhost,PBScraynid 2017-10-21 05:13:37,592 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c delete resource PBScraylabel_G34_IL,PBScrayorder,PBScrayhost,PBScraynid 2017-10-21 05:13:37,681 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource PBScraylabel_G34_IL 2017-10-21 05:13:37,691 INFO expect on server linux-kt3k: unset resource PBScraylabel_G34_IL ... OK 2017-10-21 05:13:37,691 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource PBScrayorder 2017-10-21 05:13:37,701 INFO expect on server linux-kt3k: unset resource PBScrayorder ... OK 2017-10-21 05:13:37,701 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource PBScrayhost 2017-10-21 05:13:37,710 INFO expect on server linux-kt3k: unset resource PBScrayhost ... OK 2017-10-21 05:13:37,710 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource PBScraynid 2017-10-21 05:13:37,721 INFO expect on server linux-kt3k: unset resource PBScraynid ... OK 2017-10-21 05:13:37,721 INFOCLI status on linux-kt3k: server license_count 2017-10-21 05:13:37,721 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-21 05:13:37,732 INFO server: linux-kt3k licensed 2017-10-21 05:13:37,744 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/server_priv/comm.lock 2017-10-21 05:13:37,781 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-10-21 05:13:37,809 INFO scheduler linux-kt3k: reverting configuration to defaults 2017-10-21 05:13:37,809 INFO manager on linux-kt3k: delete sched @default 2017-10-21 05:13:37,809 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c delete sched @default 2017-10-21 05:13:37,819 INFO manager on linux-kt3k: list sched 2017-10-21 05:13:37,819 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list sched 2017-10-21 05:13:37,827 INFO manager on linux-kt3k: unset sched ['sched_preempt_enforce_resumption', 'sched_cycle_length'] 2017-10-21 05:13:37,827 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c unset sched sched_preempt_enforce_resumption,sched_cycle_length 2017-10-21 05:13:37,841 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/dedicated_time 2017-10-21 05:13:37,861 INFOCLI2 linux-kt3k: cmp /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-10-21 05:13:37,863 INFO scheduler linux-kt3k: reverting holidays file to default 2017-10-21 05:13:37,864 INFOCLI2 linux-kt3k: cmp /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-10-21 05:13:37,866 INFOCLI2 linux-kt3k: cmp /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-10-21 05:13:37,868 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-10-21 05:13:37,888 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-10-21 05:13:37,908 INFO scheduler linux-kt3k: sent signal -HUP 2017-10-21 05:13:37,908 INFOCLI2 linux-kt3k: sudo -H kill -HUP 96312 2017-10-21 05:13:37,931 INFOCLI2 linux-kt3k: sudo -H /opt/pbs/sbin/pbsfs -e 2017-10-21 05:13:37,982 INFO scheduler linux-kt3k: config {'resources': '"ncpus, mem, arch, host, vnode, aoe, eoe, vntype"'} 2017-10-21 05:13:37,983 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /var/spool/pbs/sched_priv/sched_config /var/spool/pbs/sched_priv/sched_config.bak 2017-10-21 05:13:38,003 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /tmp/PtlPbs_Ei4df /var/spool/pbs/sched_priv/sched_config 2017-10-21 05:13:38,022 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-10-21 05:13:38,041 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chown root /var/spool/pbs/sched_priv/sched_config 2017-10-21 05:13:38,059 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chgrp root /var/spool/pbs/sched_priv/sched_config 2017-10-21 05:13:38,081 INFO scheduler linux-kt3k: sent signal -HUP 2017-10-21 05:13:38,081 INFOCLI2 linux-kt3k: sudo -H kill -HUP 96312 2017-10-21 05:13:38,103 INFO scheduler linux-kt3k log match: searching for "Error reading line" - No match 2017-10-21 05:13:39,113 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-10-21 05:13:39,144 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-10-21 05:13:39,163 INFO mom linux-kt3k: reverting configuration to defaults 2017-10-21 05:13:39,164 INFOCLI2 linux-kt3k: sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2017-10-21 05:13:39,182 INFOCLI2 linux-kt3k: sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2017-10-21 05:13:39,201 INFOCLI linux-kt3k: sudo -H /opt/pbs/sbin/pbs_mom -s list 2017-10-21 05:13:39,238 INFO mom linux-kt3k: sent signal -HUP 2017-10-21 05:13:39,238 INFOCLI2 linux-kt3k: sudo -H kill -HUP 96290 2017-10-21 05:13:39,277 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-10-21 05:13:39,305 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v -a 2017-10-21 05:13:39,317 INFO manager on linux-kt3k: delete node 2017-10-21 05:13:39,317 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c delete node @default 2017-10-21 05:13:39,329 INFO manager on linux-kt3k: create node linux-kt3k 2017-10-21 05:13:39,329 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c create node linux-kt3k 2017-10-21 05:13:39,349 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k 2017-10-21 05:13:39,364 INFO expect on server linux-kt3k: state = free node linux-kt3k got: state = state-unknown,down 2017-10-21 05:13:40,366 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k 2017-10-21 05:13:40,376 INFO expect on server linux-kt3k: state = free node linux-kt3k attempt: 2 got: state = state-unknown,down 2017-10-21 05:13:41,381 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k 2017-10-21 05:13:41,422 INFO expect on server linux-kt3k: state = free node linux-kt3k attempt: 3 ... OK 2017-10-21 05:13:41,423 INFO ======================================= 2017-10-21 05:13:41,423 INFO Completed TestSuspendResumeOnCray setUp 2017-10-21 05:13:41,423 INFO ======================================= 2017-10-21 05:13:41,423 INFO scheduler linux-kt3k: config {'preempt_prio': '"express_queue, normal_jobs, server_softlimits, queue_softlimits"'} 2017-10-21 05:13:41,423 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /var/spool/pbs/sched_priv/sched_config /var/spool/pbs/sched_priv/sched_config.bak 2017-10-21 05:13:41,447 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /tmp/PtlPbsQztBUc /var/spool/pbs/sched_priv/sched_config 2017-10-21 05:13:41,466 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-10-21 05:13:41,489 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chown root /var/spool/pbs/sched_priv/sched_config 2017-10-21 05:13:41,508 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chgrp root /var/spool/pbs/sched_priv/sched_config 2017-10-21 05:13:41,529 INFO scheduler linux-kt3k: sent signal -HUP 2017-10-21 05:13:41,530 INFOCLI2 linux-kt3k: sudo -H kill -HUP 96312 2017-10-21 05:13:41,550 INFO scheduler linux-kt3k log match: searching for "Error reading line" - No match 2017-10-21 05:13:42,552 INFO manager on linux-kt3k: create queue expressq {'started': 'True', 'Priority': 150, 'queue_type': 'e', 'enabled': 'True'} 2017-10-21 05:13:42,553 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c create queue expressq started=True,Priority=150,queue_type=e,enabled=True 2017-10-21 05:13:42,566 INFO manager on linux-kt3k: create queue expressq2 {'started': 'True', 'Priority': 160, 'queue_type': 'e', 'enabled': 'True'} 2017-10-21 05:13:42,566 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c create queue expressq2 started=True,Priority=160,queue_type=e,enabled=True 2017-10-21 05:13:42,576 INFO manager on linux-kt3k: create queue expressq3 {'started': 'True', 'Priority': 170, 'queue_type': 'e', 'enabled': 'True'} 2017-10-21 05:13:42,576 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c create queue expressq3 started=True,Priority=170,queue_type=e,enabled=True 2017-10-21 05:13:42,587 INFO filter: node {'resources_available.vntype': 'cray_compute'} 2017-10-21 05:13:42,587 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v -a 2017-10-21 05:13:42,595 INFO job: executable set to /bin/sleep with arguments: 100 2017-10-21 05:13:42,596 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0755 /tmp/PtlPbsJobScript2UoOUs 2017-10-21 05:13:42,615 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chown pbsuser /tmp/PtlPbsJobScript2UoOUs 2017-10-21 05:13:42,636 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chgrp tstgrp00 /tmp/PtlPbsJobScript2UoOUs 2017-10-21 05:13:42,656 INFOCLI linux-kt3k: sudo -H -u pbsuser /opt/pbs/bin/qsub -l place=scatter -l walltime=3600 -l select=2:ncpus=1:vntype=cray_compute /tmp/PtlPbsJobScript2UoOUs 2017-10-21 05:13:42,680 INFO submit to linux-kt3k as pbsuser: job 100.linux-kt3k OrderedDict([('Resource_List.place', 'scatter'), ('Resource_List.walltime', 3600), ('Resource_List.select', '2:ncpus=1:vntype=cray_compute')]) 2017-10-21 05:13:42,681 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 100.linux-kt3k 2017-10-21 05:13:42,691 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 100.linux-kt3k got: job_state = Q 2017-10-21 05:13:43,193 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:13:43,206 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 100.linux-kt3k 2017-10-21 05:13:43,214 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 100.linux-kt3k attempt: 2 ... OK 2017-10-21 05:13:43,215 INFO job: executable set to /bin/sleep with arguments: 100 2017-10-21 05:13:43,215 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0755 /tmp/PtlPbsJobScriptvrfHjp 2017-10-21 05:13:43,238 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chown pbsuser /tmp/PtlPbsJobScriptvrfHjp 2017-10-21 05:13:43,259 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chgrp tstgrp00 /tmp/PtlPbsJobScriptvrfHjp 2017-10-21 05:13:43,279 INFOCLI linux-kt3k: sudo -H -u pbsuser /opt/pbs/bin/qsub -q expressq -l place=scatter -l walltime=3600 -l select=2:ncpus=1:vntype=cray_compute /tmp/PtlPbsJobScriptvrfHjp 2017-10-21 05:13:43,303 INFO submit to linux-kt3k as pbsuser: job 101.linux-kt3k OrderedDict([('destination', 'expressq'), ('Resource_List.place', 'scatter'), ('Resource_List.walltime', 3600), ('Resource_List.select', '2:ncpus=1:vntype=cray_compute')]) 2017-10-21 05:13:43,303 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 100.linux-kt3k 2017-10-21 05:13:43,312 INFO expect on server linux-kt3k: job_state = S job 100.linux-kt3k got: job_state = R 2017-10-21 05:13:43,813 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:13:43,824 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 100.linux-kt3k 2017-10-21 05:13:43,833 INFO expect on server linux-kt3k: job_state = S job 100.linux-kt3k attempt: 2 got: job_state = R 2017-10-21 05:13:44,335 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:13:44,348 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 100.linux-kt3k 2017-10-21 05:13:44,361 INFO expect on server linux-kt3k: job_state = S job 100.linux-kt3k attempt: 3 got: job_state = R 2017-10-21 05:13:44,863 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:13:44,874 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 100.linux-kt3k 2017-10-21 05:13:44,883 INFO expect on server linux-kt3k: job_state = S job 100.linux-kt3k attempt: 4 got: job_state = R 2017-10-21 05:13:45,384 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:13:45,397 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 100.linux-kt3k 2017-10-21 05:13:45,406 INFO expect on server linux-kt3k: job_state = S job 100.linux-kt3k attempt: 5 got: job_state = R 2017-10-21 05:13:45,907 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:13:45,918 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 100.linux-kt3k 2017-10-21 05:13:45,927 INFO expect on server linux-kt3k: job_state = S job 100.linux-kt3k attempt: 6 got: job_state = R 2017-10-21 05:13:46,429 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:13:46,444 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 100.linux-kt3k 2017-10-21 05:13:46,456 INFO expect on server linux-kt3k: job_state = S job 100.linux-kt3k attempt: 7 got: job_state = R 2017-10-21 05:13:46,958 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:13:46,970 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 100.linux-kt3k 2017-10-21 05:13:46,978 INFO expect on server linux-kt3k: job_state = S job 100.linux-kt3k attempt: 8 got: job_state = R 2017-10-21 05:13:47,479 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:13:47,490 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 100.linux-kt3k 2017-10-21 05:13:47,499 INFO expect on server linux-kt3k: job_state = S job 100.linux-kt3k attempt: 9 got: job_state = R 2017-10-21 05:13:48,000 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:13:48,016 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 100.linux-kt3k 2017-10-21 05:13:48,026 INFO expect on server linux-kt3k: job_state = S job 100.linux-kt3k attempt: 10 got: job_state = R 2017-10-21 05:13:48,527 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:13:48,539 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 100.linux-kt3k 2017-10-21 05:13:48,548 INFO expect on server linux-kt3k: job_state = S job 100.linux-kt3k attempt: 11 got: job_state = R 2017-10-21 05:13:49,049 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:13:49,061 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 100.linux-kt3k 2017-10-21 05:13:49,070 INFO expect on server linux-kt3k: job_state = S job 100.linux-kt3k attempt: 12 got: job_state = R 2017-10-21 05:13:49,571 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:13:49,585 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 100.linux-kt3k 2017-10-21 05:13:49,594 INFO expect on server linux-kt3k: job_state = S job 100.linux-kt3k attempt: 13 got: job_state = R 2017-10-21 05:13:50,095 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:13:50,107 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 100.linux-kt3k 2017-10-21 05:13:50,117 INFO expect on server linux-kt3k: job_state = S job 100.linux-kt3k attempt: 14 got: job_state = R 2017-10-21 05:13:50,619 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:13:50,634 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 100.linux-kt3k 2017-10-21 05:13:50,648 INFO expect on server linux-kt3k: job_state = S job 100.linux-kt3k attempt: 15 got: job_state = R 2017-10-21 05:13:51,150 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:13:51,160 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 100.linux-kt3k 2017-10-21 05:13:51,169 INFO expect on server linux-kt3k: job_state = S job 100.linux-kt3k attempt: 16 got: job_state = R 2017-10-21 05:13:51,671 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:13:51,684 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 100.linux-kt3k 2017-10-21 05:13:51,692 INFO expect on server linux-kt3k: job_state = S job 100.linux-kt3k attempt: 17 got: job_state = R 2017-10-21 05:13:52,193 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:13:52,207 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 100.linux-kt3k 2017-10-21 05:13:52,220 INFO expect on server linux-kt3k: job_state = S job 100.linux-kt3k attempt: 18 got: job_state = R 2017-10-21 05:13:52,722 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:13:52,734 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 100.linux-kt3k 2017-10-21 05:13:52,745 INFO expect on server linux-kt3k: job_state = S job 100.linux-kt3k attempt: 19 got: job_state = R 2017-10-21 05:13:53,246 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:13:53,261 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 100.linux-kt3k 2017-10-21 05:13:53,278 INFO expect on server linux-kt3k: job_state = S job 100.linux-kt3k attempt: 20 ... OK 2017-10-21 05:13:53,279 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 101.linux-kt3k 2017-10-21 05:13:53,289 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 101.linux-kt3k ... OK 2017-10-21 05:13:53,289 INFO job: executable set to /bin/sleep with arguments: 100 2017-10-21 05:13:53,290 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0755 /tmp/PtlPbsJobScriptMpTFPR 2017-10-21 05:13:53,321 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chown pbsuser /tmp/PtlPbsJobScriptMpTFPR 2017-10-21 05:13:53,342 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chgrp tstgrp00 /tmp/PtlPbsJobScriptMpTFPR 2017-10-21 05:13:53,365 INFOCLI linux-kt3k: sudo -H -u pbsuser /opt/pbs/bin/qsub -q expressq2 -l place=scatter -l walltime=3600 -l select=2:ncpus=1:vntype=cray_compute /tmp/PtlPbsJobScriptMpTFPR 2017-10-21 05:13:53,395 INFO submit to linux-kt3k as pbsuser: job 102.linux-kt3k OrderedDict([('destination', 'expressq2'), ('Resource_List.place', 'scatter'), ('Resource_List.walltime', 3600), ('Resource_List.select', '2:ncpus=1:vntype=cray_compute')]) 2017-10-21 05:13:53,395 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 101.linux-kt3k 2017-10-21 05:13:53,409 INFO expect on server linux-kt3k: job_state = S job 101.linux-kt3k got: job_state = R 2017-10-21 05:13:53,911 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:13:53,924 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 101.linux-kt3k 2017-10-21 05:13:53,933 INFO expect on server linux-kt3k: job_state = S job 101.linux-kt3k attempt: 2 got: job_state = R 2017-10-21 05:13:54,435 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:13:54,449 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 101.linux-kt3k 2017-10-21 05:13:54,462 INFO expect on server linux-kt3k: job_state = S job 101.linux-kt3k attempt: 3 got: job_state = R 2017-10-21 05:13:54,964 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:13:54,975 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 101.linux-kt3k 2017-10-21 05:13:54,984 INFO expect on server linux-kt3k: job_state = S job 101.linux-kt3k attempt: 4 got: job_state = R 2017-10-21 05:13:55,487 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:13:55,498 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 101.linux-kt3k 2017-10-21 05:13:55,507 INFO expect on server linux-kt3k: job_state = S job 101.linux-kt3k attempt: 5 got: job_state = R 2017-10-21 05:13:56,008 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:13:56,021 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 101.linux-kt3k 2017-10-21 05:13:56,032 INFO expect on server linux-kt3k: job_state = S job 101.linux-kt3k attempt: 6 got: job_state = R 2017-10-21 05:13:56,533 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:13:56,547 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 101.linux-kt3k 2017-10-21 05:13:56,555 INFO expect on server linux-kt3k: job_state = S job 101.linux-kt3k attempt: 7 got: job_state = R 2017-10-21 05:13:57,057 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:13:57,069 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 101.linux-kt3k 2017-10-21 05:13:57,078 INFO expect on server linux-kt3k: job_state = S job 101.linux-kt3k attempt: 8 got: job_state = R 2017-10-21 05:13:57,580 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:13:57,592 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 101.linux-kt3k 2017-10-21 05:13:57,600 INFO expect on server linux-kt3k: job_state = S job 101.linux-kt3k attempt: 9 got: job_state = R 2017-10-21 05:13:58,102 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:13:58,118 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 101.linux-kt3k 2017-10-21 05:13:58,129 INFO expect on server linux-kt3k: job_state = S job 101.linux-kt3k attempt: 10 got: job_state = R 2017-10-21 05:13:58,630 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:13:58,642 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 101.linux-kt3k 2017-10-21 05:13:58,651 INFO expect on server linux-kt3k: job_state = S job 101.linux-kt3k attempt: 11 got: job_state = R 2017-10-21 05:13:59,152 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:13:59,167 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 101.linux-kt3k 2017-10-21 05:13:59,176 INFO expect on server linux-kt3k: job_state = S job 101.linux-kt3k attempt: 12 got: job_state = R 2017-10-21 05:13:59,677 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:13:59,689 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 101.linux-kt3k 2017-10-21 05:13:59,698 INFO expect on server linux-kt3k: job_state = S job 101.linux-kt3k attempt: 13 got: job_state = R 2017-10-21 05:14:00,199 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:14:00,215 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 101.linux-kt3k 2017-10-21 05:14:00,225 INFO expect on server linux-kt3k: job_state = S job 101.linux-kt3k attempt: 14 got: job_state = R 2017-10-21 05:14:00,728 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:14:00,740 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 101.linux-kt3k 2017-10-21 05:14:00,750 INFO expect on server linux-kt3k: job_state = S job 101.linux-kt3k attempt: 15 got: job_state = R 2017-10-21 05:14:01,252 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:14:01,264 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 101.linux-kt3k 2017-10-21 05:14:01,273 INFO expect on server linux-kt3k: job_state = S job 101.linux-kt3k attempt: 16 got: job_state = R 2017-10-21 05:14:01,775 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:14:01,786 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 101.linux-kt3k 2017-10-21 05:14:01,795 INFO expect on server linux-kt3k: job_state = S job 101.linux-kt3k attempt: 17 got: job_state = R 2017-10-21 05:14:02,296 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:14:02,308 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 101.linux-kt3k 2017-10-21 05:14:02,321 INFO expect on server linux-kt3k: job_state = S job 101.linux-kt3k attempt: 18 got: job_state = R 2017-10-21 05:14:02,822 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:14:02,833 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 101.linux-kt3k 2017-10-21 05:14:02,843 INFO expect on server linux-kt3k: job_state = S job 101.linux-kt3k attempt: 19 got: job_state = R 2017-10-21 05:14:03,345 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:14:03,359 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 101.linux-kt3k 2017-10-21 05:14:03,368 INFO expect on server linux-kt3k: job_state = S job 101.linux-kt3k attempt: 20 ... OK 2017-10-21 05:14:03,369 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 102.linux-kt3k 2017-10-21 05:14:03,377 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 102.linux-kt3k ... OK 2017-10-21 05:14:03,378 INFO job: executable set to /bin/sleep with arguments: 100 2017-10-21 05:14:03,378 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0755 /tmp/PtlPbsJobScriptGrT1Cj 2017-10-21 05:14:03,397 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chown pbsuser /tmp/PtlPbsJobScriptGrT1Cj 2017-10-21 05:14:03,418 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chgrp tstgrp00 /tmp/PtlPbsJobScriptGrT1Cj 2017-10-21 05:14:03,436 INFOCLI linux-kt3k: sudo -H -u pbsuser /opt/pbs/bin/qsub -q expressq3 -l place=scatter -l walltime=3600 -l select=2:ncpus=1:vntype=cray_compute /tmp/PtlPbsJobScriptGrT1Cj 2017-10-21 05:14:03,459 INFO submit to linux-kt3k as pbsuser: job 103.linux-kt3k OrderedDict([('destination', 'expressq3'), ('Resource_List.place', 'scatter'), ('Resource_List.walltime', 3600), ('Resource_List.select', '2:ncpus=1:vntype=cray_compute')]) 2017-10-21 05:14:03,460 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 102.linux-kt3k 2017-10-21 05:14:03,469 INFO expect on server linux-kt3k: job_state = S job 102.linux-kt3k got: job_state = R 2017-10-21 05:14:03,971 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:14:03,982 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 102.linux-kt3k 2017-10-21 05:14:03,989 INFO expect on server linux-kt3k: job_state = S job 102.linux-kt3k attempt: 2 got: job_state = R 2017-10-21 05:14:04,491 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:14:04,502 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 102.linux-kt3k 2017-10-21 05:14:04,510 INFO expect on server linux-kt3k: job_state = S job 102.linux-kt3k attempt: 3 got: job_state = R 2017-10-21 05:14:05,012 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:14:05,026 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 102.linux-kt3k 2017-10-21 05:14:05,034 INFO expect on server linux-kt3k: job_state = S job 102.linux-kt3k attempt: 4 got: job_state = R 2017-10-21 05:14:05,535 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:14:05,546 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 102.linux-kt3k 2017-10-21 05:14:05,556 INFO expect on server linux-kt3k: job_state = S job 102.linux-kt3k attempt: 5 got: job_state = R 2017-10-21 05:14:06,058 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:14:06,072 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 102.linux-kt3k 2017-10-21 05:14:06,083 INFO expect on server linux-kt3k: job_state = S job 102.linux-kt3k attempt: 6 got: job_state = R 2017-10-21 05:14:06,585 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:14:06,596 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 102.linux-kt3k 2017-10-21 05:14:06,604 INFO expect on server linux-kt3k: job_state = S job 102.linux-kt3k attempt: 7 got: job_state = R 2017-10-21 05:14:07,106 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:14:07,118 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 102.linux-kt3k 2017-10-21 05:14:07,126 INFO expect on server linux-kt3k: job_state = S job 102.linux-kt3k attempt: 8 got: job_state = R 2017-10-21 05:14:07,627 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:14:07,641 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 102.linux-kt3k 2017-10-21 05:14:07,651 INFO expect on server linux-kt3k: job_state = S job 102.linux-kt3k attempt: 9 got: job_state = R 2017-10-21 05:14:08,153 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:14:08,166 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 102.linux-kt3k 2017-10-21 05:14:08,177 INFO expect on server linux-kt3k: job_state = S job 102.linux-kt3k attempt: 10 got: job_state = R 2017-10-21 05:14:08,680 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:14:08,691 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 102.linux-kt3k 2017-10-21 05:14:08,700 INFO expect on server linux-kt3k: job_state = S job 102.linux-kt3k attempt: 11 got: job_state = R 2017-10-21 05:14:09,201 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:14:09,214 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 102.linux-kt3k 2017-10-21 05:14:09,224 INFO expect on server linux-kt3k: job_state = S job 102.linux-kt3k attempt: 12 got: job_state = R 2017-10-21 05:14:09,725 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:14:09,735 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 102.linux-kt3k 2017-10-21 05:14:09,745 INFO expect on server linux-kt3k: job_state = S job 102.linux-kt3k attempt: 13 got: job_state = R 2017-10-21 05:14:10,246 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:14:10,257 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 102.linux-kt3k 2017-10-21 05:14:10,266 INFO expect on server linux-kt3k: job_state = S job 102.linux-kt3k attempt: 14 got: job_state = R 2017-10-21 05:14:10,767 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:14:10,779 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 102.linux-kt3k 2017-10-21 05:14:10,788 INFO expect on server linux-kt3k: job_state = S job 102.linux-kt3k attempt: 15 got: job_state = R 2017-10-21 05:14:11,289 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:14:11,301 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 102.linux-kt3k 2017-10-21 05:14:11,309 INFO expect on server linux-kt3k: job_state = S job 102.linux-kt3k attempt: 16 got: job_state = R 2017-10-21 05:14:11,810 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:14:11,822 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 102.linux-kt3k 2017-10-21 05:14:11,830 INFO expect on server linux-kt3k: job_state = S job 102.linux-kt3k attempt: 17 got: job_state = R 2017-10-21 05:14:12,332 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:14:12,344 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 102.linux-kt3k 2017-10-21 05:14:12,353 INFO expect on server linux-kt3k: job_state = S job 102.linux-kt3k attempt: 18 got: job_state = R 2017-10-21 05:14:12,855 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:14:12,867 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 102.linux-kt3k 2017-10-21 05:14:12,877 INFO expect on server linux-kt3k: job_state = S job 102.linux-kt3k attempt: 19 got: job_state = R 2017-10-21 05:14:13,380 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:14:13,393 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 102.linux-kt3k 2017-10-21 05:14:13,402 INFO expect on server linux-kt3k: job_state = S job 102.linux-kt3k attempt: 20 ... OK 2017-10-21 05:14:13,402 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 103.linux-kt3k 2017-10-21 05:14:13,412 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 103.linux-kt3k ... OK 2017-10-21 05:14:13,412 INFO delete on linux-kt3k: 103.linux-kt3k 2017-10-21 05:14:13,412 INFO delete job on linux-kt3k: 103.linux-kt3k 2017-10-21 05:14:13,412 INFOCLI linux-kt3k: /opt/pbs/bin/qdel 103.linux-kt3k 2017-10-21 05:14:13,429 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 102.linux-kt3k 2017-10-21 05:14:13,452 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 102.linux-kt3k got: job_state = S 2017-10-21 05:14:13,954 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:14:13,967 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 102.linux-kt3k 2017-10-21 05:14:13,979 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 102.linux-kt3k attempt: 2 ... OK 2017-10-21 05:14:13,980 INFO ========================================= 2017-10-21 05:14:13,980 INFO Entered TestSuspendResumeOnCray tearDown 2017-10-21 05:14:13,980 INFO ========================================= 2017-10-21 05:14:13,980 INFO ========================================== 2017-10-21 05:14:13,980 INFO Completed TestSuspendResumeOnCray tearDown 2017-10-21 05:14:13,980 INFO ========================================== 2017-10-21 05:14:13,980 INFO ok 2017-10-21 05:14:13,980 INFO test name: test_preempt_STF (tests.functional.pbs_cray_suspend_resume.TestSuspendResumeOnCray)... 2017-10-21 05:14:13,980 INFO test start time: Sat Oct 21 05:14:13 2017 2017-10-21 05:14:13,981 INFO test docstring: Test shrink to fit by creating a reservation for all compute nodes starting in 100 sec. with a duration of two hours. A preempted STF job with min_walltime of 1 min. and max_walltime of 2 hours will stay suspended after higher priority job goes away if its min_walltime can't be satisfied. 2017-10-21 05:14:13,981 INFO ====================================== 2017-10-21 05:14:13,981 INFO Entered TestSuspendResumeOnCray setUp 2017-10-21 05:14:13,981 INFO ====================================== 2017-10-21 05:14:13,981 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-21 05:14:13,989 INFO status on linux-kt3k: server 2017-10-21 05:14:13,990 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-21 05:14:14,001 INFO manager on linux-kt3k: set server {'managers': (3, 'root@*')} 2017-10-21 05:14:14,001 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c set server managers-=root@* 2017-10-21 05:14:14,036 INFO manager on linux-kt3k: set server {'managers': (2, 'root@*')} 2017-10-21 05:14:14,037 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c set server managers+=root@* 2017-10-21 05:14:14,072 INFO server linux-kt3k: reverting configuration to defaults 2017-10-21 05:14:14,072 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-21 05:14:14,081 INFO select on linux-kt3k: __ALL__ 2017-10-21 05:14:14,081 INFOCLI linux-kt3k: /opt/pbs/bin/qselect 2017-10-21 05:14:14,089 INFO delete job on linux-kt3k: 100.linux-kt3k, 101.linux-kt3k, 102.linux-kt3k 2017-10-21 05:14:14,090 INFOCLI linux-kt3k: /opt/pbs/bin/qdel -W force 100.linux-kt3k 101.linux-kt3k 102.linux-kt3k 2017-10-21 05:14:14,159 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 100.linux-kt3k 2017-10-21 05:14:14,188 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 101.linux-kt3k 2017-10-21 05:14:14,200 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 102.linux-kt3k 2017-10-21 05:14:14,222 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f @linux-kt3k 2017-10-21 05:14:14,234 INFO expect on server linux-kt3k: job_state set 0 job ... OK 2017-10-21 05:14:14,234 INFOCLI linux-kt3k: /opt/pbs/bin/pbs_rstat -f 2017-10-21 05:14:14,260 INFO manager on linux-kt3k: unset server ['restrict_res_to_release_on_suspend'] 2017-10-21 05:14:14,260 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c unset server restrict_res_to_release_on_suspend 2017-10-21 05:14:14,284 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-10-21 05:14:14,325 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Qf @linux-kt3k 2017-10-21 05:14:14,341 INFO status on linux-kt3k: node 2017-10-21 05:14:14,342 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v -a 2017-10-21 05:14:14,353 INFO manager on linux-kt3k: delete queue workq,expressq,expressq2,expressq3 2017-10-21 05:14:14,353 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c delete queue workq,expressq,expressq2,expressq3 2017-10-21 05:14:14,369 INFO server linux-kt3k: expect offset set to 0.5 2017-10-21 05:14:14,870 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Qf workq@linux-kt3k 2017-10-21 05:14:14,882 INFO expect on server linux-kt3k: unset queue workq ... OK 2017-10-21 05:14:14,882 INFO server linux-kt3k: expect offset set to 0.5 2017-10-21 05:14:15,383 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Qf expressq@linux-kt3k 2017-10-21 05:14:15,393 INFO expect on server linux-kt3k: unset queue expressq ... OK 2017-10-21 05:14:15,394 INFO server linux-kt3k: expect offset set to 0.5 2017-10-21 05:14:15,895 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Qf expressq2@linux-kt3k 2017-10-21 05:14:15,905 INFO expect on server linux-kt3k: unset queue expressq2 ... OK 2017-10-21 05:14:15,905 INFO server linux-kt3k: expect offset set to 0.5 2017-10-21 05:14:16,407 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Qf expressq3@linux-kt3k 2017-10-21 05:14:16,418 INFO expect on server linux-kt3k: unset queue expressq3 ... OK 2017-10-21 05:14:16,418 INFO manager on linux-kt3k: create queue workq {'started': 'True', 'queue_type': 'Execution', 'enabled': 'True'} 2017-10-21 05:14:16,418 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c create queue workq started=True,queue_type=Execution,enabled=True 2017-10-21 05:14:16,428 INFO server linux-kt3k: expect offset set to 0.5 2017-10-21 05:14:16,930 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Qf workq@linux-kt3k 2017-10-21 05:14:16,941 INFO expect on server linux-kt3k: started set True || queue_type set Execution || enabled set True queue workq ... OK 2017-10-21 05:14:16,941 INFOCLI2 linux-kt3k: sudo -H cmp /opt/pbs/lib/python/altair/pbs_hooks/PBS_translate_mpp.HK /var/spool/pbs/server_priv/hooks/PBS_translate_mpp.HK 2017-10-21 05:14:16,968 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /opt/pbs/lib/python/altair/pbs_hooks/PBS_translate_mpp.HK /var/spool/pbs/server_priv/hooks/PBS_translate_mpp.HK 2017-10-21 05:14:16,993 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/server_priv/hooks/PBS_translate_mpp.HK 2017-10-21 05:14:17,016 INFO server linux-kt3k: sent signal -HUP 2017-10-21 05:14:17,016 INFOCLI2 linux-kt3k: sudo -H kill -HUP 31472 2017-10-21 05:14:17,037 INFO manager on linux-kt3k: set pbshook PBS_translate_mpp {'enabled': 'true'} 2017-10-21 05:14:17,038 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c set pbshook PBS_translate_mpp enabled=true 2017-10-21 05:14:17,065 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-10-21 05:14:17,091 INFO manager on linux-kt3k: set server {'default_queue': 'workq', 'restrict_res_to_release_on_suspend': 'ncpus'} 2017-10-21 05:14:17,091 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server default_queue=workq,restrict_res_to_release_on_suspend=ncpus 2017-10-21 05:14:17,101 INFO status on linux-kt3k: resource 2017-10-21 05:14:17,101 INFO manager on linux-kt3k: list resource 2017-10-21 05:14:17,102 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource 2017-10-21 05:14:17,110 INFO manager on linux-kt3k: delete resource PBScraylabel_G34_IL,PBScrayorder,PBScrayhost,PBScraynid 2017-10-21 05:14:17,110 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c delete resource PBScraylabel_G34_IL,PBScrayorder,PBScrayhost,PBScraynid 2017-10-21 05:14:17,192 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource PBScraylabel_G34_IL 2017-10-21 05:14:17,201 INFO expect on server linux-kt3k: unset resource PBScraylabel_G34_IL ... OK 2017-10-21 05:14:17,201 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource PBScrayorder 2017-10-21 05:14:17,212 INFO expect on server linux-kt3k: unset resource PBScrayorder ... OK 2017-10-21 05:14:17,213 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource PBScrayhost 2017-10-21 05:14:17,226 INFO expect on server linux-kt3k: unset resource PBScrayhost ... OK 2017-10-21 05:14:17,227 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource PBScraynid 2017-10-21 05:14:17,235 INFO expect on server linux-kt3k: unset resource PBScraynid ... OK 2017-10-21 05:14:17,235 INFOCLI status on linux-kt3k: server license_count 2017-10-21 05:14:17,235 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-21 05:14:17,244 INFO server: linux-kt3k licensed 2017-10-21 05:14:17,252 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/server_priv/comm.lock 2017-10-21 05:14:17,278 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-10-21 05:14:17,297 INFO scheduler linux-kt3k: reverting configuration to defaults 2017-10-21 05:14:17,297 INFO manager on linux-kt3k: delete sched @default 2017-10-21 05:14:17,298 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c delete sched @default 2017-10-21 05:14:17,308 INFO manager on linux-kt3k: list sched 2017-10-21 05:14:17,308 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list sched 2017-10-21 05:14:17,316 INFO manager on linux-kt3k: unset sched ['sched_preempt_enforce_resumption', 'sched_cycle_length'] 2017-10-21 05:14:17,316 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c unset sched sched_preempt_enforce_resumption,sched_cycle_length 2017-10-21 05:14:17,325 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/dedicated_time 2017-10-21 05:14:17,345 INFOCLI2 linux-kt3k: cmp /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-10-21 05:14:17,347 INFO scheduler linux-kt3k: reverting holidays file to default 2017-10-21 05:14:17,347 INFOCLI2 linux-kt3k: cmp /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-10-21 05:14:17,349 INFOCLI2 linux-kt3k: cmp /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-10-21 05:14:17,351 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-10-21 05:14:17,371 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-10-21 05:14:17,389 INFO scheduler linux-kt3k: sent signal -HUP 2017-10-21 05:14:17,389 INFOCLI2 linux-kt3k: sudo -H kill -HUP 96312 2017-10-21 05:14:17,409 INFOCLI2 linux-kt3k: sudo -H /opt/pbs/sbin/pbsfs -e 2017-10-21 05:14:17,460 INFO scheduler linux-kt3k: config {'resources': '"ncpus, mem, arch, host, vnode, aoe, eoe, vntype"'} 2017-10-21 05:14:17,461 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /var/spool/pbs/sched_priv/sched_config /var/spool/pbs/sched_priv/sched_config.bak 2017-10-21 05:14:17,480 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /tmp/PtlPbshvDp3c /var/spool/pbs/sched_priv/sched_config 2017-10-21 05:14:17,509 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-10-21 05:14:17,535 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chown root /var/spool/pbs/sched_priv/sched_config 2017-10-21 05:14:17,556 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chgrp root /var/spool/pbs/sched_priv/sched_config 2017-10-21 05:14:17,580 INFO scheduler linux-kt3k: sent signal -HUP 2017-10-21 05:14:17,580 INFOCLI2 linux-kt3k: sudo -H kill -HUP 96312 2017-10-21 05:14:17,611 INFO scheduler linux-kt3k log match: searching for "Error reading line" - No match 2017-10-21 05:14:18,622 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-10-21 05:14:18,650 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-10-21 05:14:18,669 INFO mom linux-kt3k: reverting configuration to defaults 2017-10-21 05:14:18,670 INFOCLI2 linux-kt3k: sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2017-10-21 05:14:18,691 INFOCLI2 linux-kt3k: sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2017-10-21 05:14:18,716 INFOCLI linux-kt3k: sudo -H /opt/pbs/sbin/pbs_mom -s list 2017-10-21 05:14:18,762 INFO mom linux-kt3k: sent signal -HUP 2017-10-21 05:14:18,762 INFOCLI2 linux-kt3k: sudo -H kill -HUP 96290 2017-10-21 05:14:18,795 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-10-21 05:14:18,829 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v -a 2017-10-21 05:14:18,842 INFO manager on linux-kt3k: delete node 2017-10-21 05:14:18,842 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c delete node @default 2017-10-21 05:14:18,858 INFO manager on linux-kt3k: create node linux-kt3k 2017-10-21 05:14:18,858 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c create node linux-kt3k 2017-10-21 05:14:18,886 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k 2017-10-21 05:14:18,916 INFO expect on server linux-kt3k: state = free node linux-kt3k got: state = state-unknown,down 2017-10-21 05:14:19,917 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k 2017-10-21 05:14:19,929 INFO expect on server linux-kt3k: state = free node linux-kt3k attempt: 2 got: state = state-unknown,down 2017-10-21 05:14:20,936 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k 2017-10-21 05:14:20,972 INFO expect on server linux-kt3k: state = free node linux-kt3k attempt: 3 ... OK 2017-10-21 05:14:20,972 INFO ======================================= 2017-10-21 05:14:20,972 INFO Completed TestSuspendResumeOnCray setUp 2017-10-21 05:14:20,972 INFO ======================================= 2017-10-21 05:14:20,972 INFO manager on linux-kt3k: create queue highp {'queue_type': 'execution'} 2017-10-21 05:14:20,972 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c create queue highp queue_type=execution 2017-10-21 05:14:20,982 INFO manager on linux-kt3k: set queue highp {'started': 'True', 'priority': '150', 'enabled': 'True'} 2017-10-21 05:14:20,983 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set queue highp started=True,priority=150,enabled=True 2017-10-21 05:14:20,992 INFO filter: node {'resources_available.vntype': 'cray_compute'} 2017-10-21 05:14:20,992 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v -a 2017-10-21 05:14:21,001 INFOCLI linux-kt3k: sudo -H -u pbsuser /opt/pbs/bin/pbs_rsub -R 201710210516.01 -l place=scatter -D 7200 -l select=2:ncpus=1:vntype=cray_compute 2017-10-21 05:14:21,036 INFO submit to linux-kt3k as pbsuser: reservation R104.linux-kt3k UNCONFIRMED OrderedDict([('reserve_start', 1508543161), ('Resource_List.place', 'scatter'), ('reserve_duration', 7200), ('Resource_List.select', '2:ncpus=1:vntype=cray_compute')]) 2017-10-21 05:14:21,036 INFOCLI linux-kt3k: /opt/pbs/bin/pbs_rstat -f R104.linux-kt3k 2017-10-21 05:14:21,046 INFO expect on server linux-kt3k: reserve_state ~ RESV_CONFIRMED|2 reservation R104.linux-kt3k got: reserve_state = RESV_UNCONFIRMED 2017-10-21 05:14:21,547 INFOCLI linux-kt3k: /opt/pbs/bin/pbs_rstat -f R104.linux-kt3k 2017-10-21 05:14:21,559 INFO expect on server linux-kt3k: reserve_state ~ RESV_CONFIRMED|2 reservation R104.linux-kt3k attempt: 2 ... OK 2017-10-21 05:14:21,559 INFO job: executable set to /bin/sleep with arguments: 100 2017-10-21 05:14:21,559 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0755 /tmp/PtlPbsJobScriptbyMPzD 2017-10-21 05:14:21,583 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chown pbsuser /tmp/PtlPbsJobScriptbyMPzD 2017-10-21 05:14:21,604 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chgrp tstgrp00 /tmp/PtlPbsJobScriptbyMPzD 2017-10-21 05:14:21,625 INFOCLI linux-kt3k: sudo -H -u pbsuser /opt/pbs/bin/qsub -l min_walltime=00:01:00 -l place=scatter -l max_walltime=02:00:00 -l select=2:ncpus=1:vntype=cray_compute /tmp/PtlPbsJobScriptbyMPzD 2017-10-21 05:14:21,650 INFO submit to linux-kt3k as pbsuser: job 105.linux-kt3k OrderedDict([('Resource_List.min_walltime', '00:01:00'), ('Resource_List.place', 'scatter'), ('Resource_List.max_walltime', '02:00:00'), ('Resource_List.select', '2:ncpus=1:vntype=cray_compute')]) 2017-10-21 05:14:21,651 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 105.linux-kt3k 2017-10-21 05:14:21,662 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 105.linux-kt3k got: job_state = Q 2017-10-21 05:14:22,164 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:14:22,178 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 105.linux-kt3k 2017-10-21 05:14:22,187 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 105.linux-kt3k attempt: 2 ... OK 2017-10-21 05:14:22,187 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 105.linux-kt3k 2017-10-21 05:14:22,196 INFO expect on server linux-kt3k: Resource_List.walltime <= 00:01:40 job 105.linux-kt3k ... OK 2017-10-21 05:14:22,196 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 105.linux-kt3k 2017-10-21 05:14:22,205 INFO expect on server linux-kt3k: Resource_List.walltime >= 00:01:00 job 105.linux-kt3k ... OK 2017-10-21 05:14:57,242 INFO job: executable set to /bin/sleep with arguments: 100 2017-10-21 05:14:57,243 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0755 /tmp/PtlPbsJobScriptCmUKYO 2017-10-21 05:14:57,265 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chown pbsuser /tmp/PtlPbsJobScriptCmUKYO 2017-10-21 05:14:57,285 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chgrp tstgrp00 /tmp/PtlPbsJobScriptCmUKYO 2017-10-21 05:14:57,307 INFOCLI linux-kt3k: sudo -H -u pbsuser /opt/pbs/bin/qsub -q highp -l place=scatter -l walltime=00:01:00 -l select=2:ncpus=1:vntype=cray_compute /tmp/PtlPbsJobScriptCmUKYO 2017-10-21 05:14:57,331 INFO submit to linux-kt3k as pbsuser: job 106.linux-kt3k OrderedDict([('destination', 'highp'), ('Resource_List.place', 'scatter'), ('Resource_List.walltime', '00:01:00'), ('Resource_List.select', '2:ncpus=1:vntype=cray_compute')]) 2017-10-21 05:14:57,332 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 106.linux-kt3k 2017-10-21 05:14:57,340 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 106.linux-kt3k got: job_state = Q 2017-10-21 05:14:57,842 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:14:57,854 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 106.linux-kt3k 2017-10-21 05:14:57,863 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 106.linux-kt3k attempt: 2 got: job_state = Q 2017-10-21 05:14:58,364 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:14:58,377 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 106.linux-kt3k 2017-10-21 05:14:58,388 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 106.linux-kt3k attempt: 3 got: job_state = Q 2017-10-21 05:14:58,889 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:14:58,903 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 106.linux-kt3k 2017-10-21 05:14:58,913 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 106.linux-kt3k attempt: 4 got: job_state = Q 2017-10-21 05:14:59,414 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:14:59,425 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 106.linux-kt3k 2017-10-21 05:14:59,433 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 106.linux-kt3k attempt: 5 got: job_state = Q 2017-10-21 05:14:59,934 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:14:59,946 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 106.linux-kt3k 2017-10-21 05:14:59,955 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 106.linux-kt3k attempt: 6 got: job_state = Q 2017-10-21 05:15:00,456 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:15:00,469 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 106.linux-kt3k 2017-10-21 05:15:00,479 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 106.linux-kt3k attempt: 7 got: job_state = Q 2017-10-21 05:15:00,980 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:15:00,994 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 106.linux-kt3k 2017-10-21 05:15:01,003 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 106.linux-kt3k attempt: 8 got: job_state = Q 2017-10-21 05:15:01,505 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:15:01,518 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 106.linux-kt3k 2017-10-21 05:15:01,527 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 106.linux-kt3k attempt: 9 got: job_state = Q 2017-10-21 05:15:02,028 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:15:02,041 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 106.linux-kt3k 2017-10-21 05:15:02,050 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 106.linux-kt3k attempt: 10 got: job_state = Q 2017-10-21 05:15:02,551 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:15:02,563 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 106.linux-kt3k 2017-10-21 05:15:02,573 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 106.linux-kt3k attempt: 11 got: job_state = Q 2017-10-21 05:15:03,075 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:15:03,086 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 106.linux-kt3k 2017-10-21 05:15:03,095 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 106.linux-kt3k attempt: 12 got: job_state = Q 2017-10-21 05:15:03,597 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:15:03,608 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 106.linux-kt3k 2017-10-21 05:15:03,617 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 106.linux-kt3k attempt: 13 got: job_state = Q 2017-10-21 05:15:04,118 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:15:04,130 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 106.linux-kt3k 2017-10-21 05:15:04,139 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 106.linux-kt3k attempt: 14 got: job_state = Q 2017-10-21 05:15:04,641 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:15:04,653 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 106.linux-kt3k 2017-10-21 05:15:04,663 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 106.linux-kt3k attempt: 15 got: job_state = Q 2017-10-21 05:15:05,164 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:15:05,176 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 106.linux-kt3k 2017-10-21 05:15:05,184 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 106.linux-kt3k attempt: 16 got: job_state = Q 2017-10-21 05:15:05,686 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:15:05,701 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 106.linux-kt3k 2017-10-21 05:15:05,711 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 106.linux-kt3k attempt: 17 got: job_state = Q 2017-10-21 05:15:06,212 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:15:06,223 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 106.linux-kt3k 2017-10-21 05:15:06,232 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 106.linux-kt3k attempt: 18 got: job_state = Q 2017-10-21 05:15:06,734 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:15:06,748 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 106.linux-kt3k 2017-10-21 05:15:06,759 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 106.linux-kt3k attempt: 19 got: job_state = Q 2017-10-21 05:15:07,260 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:15:07,275 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 106.linux-kt3k 2017-10-21 05:15:07,286 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 106.linux-kt3k attempt: 20 ... OK 2017-10-21 05:15:07,286 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 105.linux-kt3k 2017-10-21 05:15:07,296 INFO expect on server linux-kt3k: job_state = S job 105.linux-kt3k ... OK 2017-10-21 05:15:57,329 INFO delete on linux-kt3k: 106.linux-kt3k 2017-10-21 05:15:57,330 INFO delete job on linux-kt3k: 106.linux-kt3k 2017-10-21 05:15:57,330 INFOCLI linux-kt3k: /opt/pbs/bin/qdel 106.linux-kt3k 2017-10-21 05:15:57,353 INFO manager on linux-kt3k: set server {'scheduling': 'True'} 2017-10-21 05:15:57,353 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:15:57,387 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-21 05:15:57,410 INFO expect on server linux-kt3k: server_state = Active server linux-kt3k got: server_state = Scheduling 2017-10-21 05:15:57,911 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-21 05:15:57,921 INFO expect on server linux-kt3k: server_state = Active server linux-kt3k attempt: 2 ... OK 2017-10-21 05:15:57,922 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 105.linux-kt3k 2017-10-21 05:15:57,932 INFO expect on server linux-kt3k: job_state = S job 105.linux-kt3k ... OK 2017-10-21 05:15:57,932 INFO ========================================= 2017-10-21 05:15:57,933 INFO Entered TestSuspendResumeOnCray tearDown 2017-10-21 05:15:57,933 INFO ========================================= 2017-10-21 05:15:57,933 INFO ========================================== 2017-10-21 05:15:57,933 INFO Completed TestSuspendResumeOnCray tearDown 2017-10-21 05:15:57,933 INFO ========================================== 2017-10-21 05:15:57,934 INFO ok 2017-10-21 05:15:57,934 INFO test name: test_preempted_topjob_calendared (tests.functional.pbs_cray_suspend_resume.TestSuspendResumeOnCray)... 2017-10-21 05:15:57,934 INFO test start time: Sat Oct 21 05:15:57 2017 2017-10-21 05:15:57,934 INFO test docstring: That even if topjob_ineligible is set for a preempted job and sched_preempt_enforce_resumption is set true, the preempted job will be calendared 2017-10-21 05:15:57,934 INFO ====================================== 2017-10-21 05:15:57,934 INFO Entered TestSuspendResumeOnCray setUp 2017-10-21 05:15:57,935 INFO ====================================== 2017-10-21 05:15:57,935 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-21 05:15:57,944 INFO status on linux-kt3k: server 2017-10-21 05:15:57,944 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-21 05:15:57,954 INFO manager on linux-kt3k: set server {'managers': (3, 'root@*')} 2017-10-21 05:15:57,955 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c set server managers-=root@* 2017-10-21 05:15:57,997 INFO manager on linux-kt3k: set server {'managers': (2, 'root@*')} 2017-10-21 05:15:57,997 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c set server managers+=root@* 2017-10-21 05:15:58,025 INFO server linux-kt3k: reverting configuration to defaults 2017-10-21 05:15:58,025 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-21 05:15:58,034 INFO select on linux-kt3k: __ALL__ 2017-10-21 05:15:58,034 INFOCLI linux-kt3k: /opt/pbs/bin/qselect 2017-10-21 05:15:58,043 INFO delete job on linux-kt3k: 105.linux-kt3k 2017-10-21 05:15:58,044 INFOCLI linux-kt3k: /opt/pbs/bin/qdel -W force 105.linux-kt3k 2017-10-21 05:15:58,068 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 105.linux-kt3k 2017-10-21 05:15:58,093 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f @linux-kt3k 2017-10-21 05:15:58,111 INFO expect on server linux-kt3k: job_state set 0 job ... OK 2017-10-21 05:15:58,111 INFOCLI linux-kt3k: /opt/pbs/bin/pbs_rstat -f 2017-10-21 05:15:58,130 INFO delete resv on linux-kt3k: R104.linux-kt3k 2017-10-21 05:15:58,130 INFOCLI linux-kt3k: /opt/pbs/bin/pbs_rdel R104.linux-kt3k 2017-10-21 05:15:58,159 INFOCLI linux-kt3k: /opt/pbs/bin/pbs_rstat -f 2017-10-21 05:15:58,189 INFO manager on linux-kt3k: unset server ['restrict_res_to_release_on_suspend'] 2017-10-21 05:15:58,189 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c unset server restrict_res_to_release_on_suspend 2017-10-21 05:15:58,223 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-10-21 05:15:58,259 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Qf @linux-kt3k 2017-10-21 05:15:58,274 INFO status on linux-kt3k: node 2017-10-21 05:15:58,274 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v -a 2017-10-21 05:15:58,285 INFO manager on linux-kt3k: delete queue workq,highp 2017-10-21 05:15:58,286 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c delete queue workq,highp 2017-10-21 05:15:58,300 INFO server linux-kt3k: expect offset set to 0.5 2017-10-21 05:15:58,802 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Qf workq@linux-kt3k 2017-10-21 05:15:58,813 INFO expect on server linux-kt3k: unset queue workq ... OK 2017-10-21 05:15:58,814 INFO server linux-kt3k: expect offset set to 0.5 2017-10-21 05:15:59,316 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Qf highp@linux-kt3k 2017-10-21 05:15:59,326 INFO expect on server linux-kt3k: unset queue highp ... OK 2017-10-21 05:15:59,326 INFO manager on linux-kt3k: create queue workq {'started': 'True', 'queue_type': 'Execution', 'enabled': 'True'} 2017-10-21 05:15:59,327 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c create queue workq started=True,queue_type=Execution,enabled=True 2017-10-21 05:15:59,337 INFO server linux-kt3k: expect offset set to 0.5 2017-10-21 05:15:59,838 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Qf workq@linux-kt3k 2017-10-21 05:15:59,850 INFO expect on server linux-kt3k: started set True || queue_type set Execution || enabled set True queue workq ... OK 2017-10-21 05:15:59,851 INFOCLI2 linux-kt3k: sudo -H cmp /opt/pbs/lib/python/altair/pbs_hooks/PBS_translate_mpp.HK /var/spool/pbs/server_priv/hooks/PBS_translate_mpp.HK 2017-10-21 05:15:59,870 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /opt/pbs/lib/python/altair/pbs_hooks/PBS_translate_mpp.HK /var/spool/pbs/server_priv/hooks/PBS_translate_mpp.HK 2017-10-21 05:15:59,891 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/server_priv/hooks/PBS_translate_mpp.HK 2017-10-21 05:15:59,912 INFO server linux-kt3k: sent signal -HUP 2017-10-21 05:15:59,912 INFOCLI2 linux-kt3k: sudo -H kill -HUP 31472 2017-10-21 05:15:59,931 INFO manager on linux-kt3k: set pbshook PBS_translate_mpp {'enabled': 'true'} 2017-10-21 05:15:59,931 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c set pbshook PBS_translate_mpp enabled=true 2017-10-21 05:15:59,958 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-10-21 05:15:59,987 INFO manager on linux-kt3k: set server {'default_queue': 'workq', 'restrict_res_to_release_on_suspend': 'ncpus'} 2017-10-21 05:15:59,987 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server default_queue=workq,restrict_res_to_release_on_suspend=ncpus 2017-10-21 05:16:00,005 INFO status on linux-kt3k: resource 2017-10-21 05:16:00,006 INFO manager on linux-kt3k: list resource 2017-10-21 05:16:00,006 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource 2017-10-21 05:16:00,015 INFO manager on linux-kt3k: delete resource PBScraylabel_G34_IL,PBScrayorder,PBScrayhost,PBScraynid 2017-10-21 05:16:00,015 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c delete resource PBScraylabel_G34_IL,PBScrayorder,PBScrayhost,PBScraynid 2017-10-21 05:16:00,105 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource PBScraylabel_G34_IL 2017-10-21 05:16:00,115 INFO expect on server linux-kt3k: unset resource PBScraylabel_G34_IL ... OK 2017-10-21 05:16:00,116 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource PBScrayorder 2017-10-21 05:16:00,124 INFO expect on server linux-kt3k: unset resource PBScrayorder ... OK 2017-10-21 05:16:00,124 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource PBScrayhost 2017-10-21 05:16:00,133 INFO expect on server linux-kt3k: unset resource PBScrayhost ... OK 2017-10-21 05:16:00,133 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource PBScraynid 2017-10-21 05:16:00,141 INFO expect on server linux-kt3k: unset resource PBScraynid ... OK 2017-10-21 05:16:00,141 INFOCLI status on linux-kt3k: server license_count 2017-10-21 05:16:00,141 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-10-21 05:16:00,150 INFO server: linux-kt3k licensed 2017-10-21 05:16:00,159 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/server_priv/comm.lock 2017-10-21 05:16:00,186 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-10-21 05:16:00,206 INFO scheduler linux-kt3k: reverting configuration to defaults 2017-10-21 05:16:00,206 INFO manager on linux-kt3k: delete sched @default 2017-10-21 05:16:00,206 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c delete sched @default 2017-10-21 05:16:00,214 INFO manager on linux-kt3k: list sched 2017-10-21 05:16:00,215 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list sched 2017-10-21 05:16:00,223 INFO manager on linux-kt3k: unset sched ['sched_preempt_enforce_resumption', 'sched_cycle_length'] 2017-10-21 05:16:00,223 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c unset sched sched_preempt_enforce_resumption,sched_cycle_length 2017-10-21 05:16:00,233 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/dedicated_time 2017-10-21 05:16:00,252 INFOCLI2 linux-kt3k: cmp /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-10-21 05:16:00,254 INFO scheduler linux-kt3k: reverting holidays file to default 2017-10-21 05:16:00,254 INFOCLI2 linux-kt3k: cmp /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-10-21 05:16:00,256 INFOCLI2 linux-kt3k: cmp /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-10-21 05:16:00,257 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-10-21 05:16:00,276 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-10-21 05:16:00,298 INFO scheduler linux-kt3k: sent signal -HUP 2017-10-21 05:16:00,298 INFOCLI2 linux-kt3k: sudo -H kill -HUP 96312 2017-10-21 05:16:00,318 INFOCLI2 linux-kt3k: sudo -H /opt/pbs/sbin/pbsfs -e 2017-10-21 05:16:00,362 INFO scheduler linux-kt3k: config {'resources': '"ncpus, mem, arch, host, vnode, aoe, eoe, vntype"'} 2017-10-21 05:16:00,363 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /var/spool/pbs/sched_priv/sched_config /var/spool/pbs/sched_priv/sched_config.bak 2017-10-21 05:16:00,383 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /tmp/PtlPbsmiU_B8 /var/spool/pbs/sched_priv/sched_config 2017-10-21 05:16:00,402 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-10-21 05:16:00,425 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chown root /var/spool/pbs/sched_priv/sched_config 2017-10-21 05:16:00,444 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chgrp root /var/spool/pbs/sched_priv/sched_config 2017-10-21 05:16:00,466 INFO scheduler linux-kt3k: sent signal -HUP 2017-10-21 05:16:00,467 INFOCLI2 linux-kt3k: sudo -H kill -HUP 96312 2017-10-21 05:16:00,489 INFO scheduler linux-kt3k log match: searching for "Error reading line" - No match 2017-10-21 05:16:01,500 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-10-21 05:16:01,529 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-10-21 05:16:01,550 INFO mom linux-kt3k: reverting configuration to defaults 2017-10-21 05:16:01,550 INFOCLI2 linux-kt3k: sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2017-10-21 05:16:01,570 INFOCLI2 linux-kt3k: sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2017-10-21 05:16:01,589 INFOCLI linux-kt3k: sudo -H /opt/pbs/sbin/pbs_mom -s list 2017-10-21 05:16:01,633 INFO mom linux-kt3k: sent signal -HUP 2017-10-21 05:16:01,633 INFOCLI2 linux-kt3k: sudo -H kill -HUP 96290 2017-10-21 05:16:01,663 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-10-21 05:16:01,687 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v -a 2017-10-21 05:16:01,703 INFO manager on linux-kt3k: delete node 2017-10-21 05:16:01,703 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c delete node @default 2017-10-21 05:16:01,717 INFO manager on linux-kt3k: create node linux-kt3k 2017-10-21 05:16:01,718 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c create node linux-kt3k 2017-10-21 05:16:01,732 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k 2017-10-21 05:16:01,749 INFO expect on server linux-kt3k: state = free node linux-kt3k got: state = state-unknown,down 2017-10-21 05:16:02,751 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k 2017-10-21 05:16:02,760 INFO expect on server linux-kt3k: state = free node linux-kt3k attempt: 2 got: state = state-unknown,down 2017-10-21 05:16:03,765 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k 2017-10-21 05:16:03,797 INFO expect on server linux-kt3k: state = free node linux-kt3k attempt: 3 ... OK 2017-10-21 05:16:03,797 INFO ======================================= 2017-10-21 05:16:03,798 INFO Completed TestSuspendResumeOnCray setUp 2017-10-21 05:16:03,798 INFO ======================================= 2017-10-21 05:16:03,798 INFO manager on linux-kt3k: set sched {'sched_preempt_enforce_resumption': 'true'} 2017-10-21 05:16:03,798 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set sched sched_preempt_enforce_resumption=true 2017-10-21 05:16:03,808 INFO manager on linux-kt3k: set server {'backfill_depth': '2'} 2017-10-21 05:16:03,808 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server backfill_depth=2 2017-10-21 05:16:03,820 INFO filter: node {'resources_available.vntype': 'cray_compute'} 2017-10-21 05:16:03,820 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v -a 2017-10-21 05:16:03,829 INFO job: executable set to /bin/sleep with arguments: 100 2017-10-21 05:16:03,829 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0755 /tmp/PtlPbsJobScriptPSpQQ4 2017-10-21 05:16:03,853 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chown pbsuser /tmp/PtlPbsJobScriptPSpQQ4 2017-10-21 05:16:03,874 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chgrp tstgrp00 /tmp/PtlPbsJobScriptPSpQQ4 2017-10-21 05:16:03,900 INFOCLI linux-kt3k: sudo -H -u pbsuser /opt/pbs/bin/qsub -l place=scatter -l walltime=120 -l select=2:ncpus=1:vntype=cray_compute /tmp/PtlPbsJobScriptPSpQQ4 2017-10-21 05:16:03,940 INFO submit to linux-kt3k as pbsuser: job 107.linux-kt3k OrderedDict([('Resource_List.place', 'scatter'), ('Resource_List.walltime', '120'), ('Resource_List.select', '2:ncpus=1:vntype=cray_compute')]) 2017-10-21 05:16:03,940 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 107.linux-kt3k 2017-10-21 05:16:03,950 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 107.linux-kt3k got: job_state = Q 2017-10-21 05:16:04,453 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:16:04,466 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 107.linux-kt3k 2017-10-21 05:16:04,476 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 107.linux-kt3k attempt: 2 ... OK 2017-10-21 05:16:04,476 INFO alter on linux-kt3k as root: 107.linux-kt3k {'Additional_Attributes': 'topjob_ineligible = true'} 2017-10-21 05:16:04,476 INFOCLI linux-kt3k: /opt/pbs/bin/qalter -W topjob_ineligible = true 107.linux-kt3k 2017-10-21 05:16:04,487 INFO manager on linux-kt3k: create queue highp {'started': 't', 'priority': '150', 'queue_type': 'e', 'enabled': 'True'} 2017-10-21 05:16:04,488 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c create queue highp started=t,priority=150,queue_type=e,enabled=True 2017-10-21 05:16:04,500 INFO job: executable set to /bin/sleep with arguments: 100 2017-10-21 05:16:04,501 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0755 /tmp/PtlPbsJobScript8kXdux 2017-10-21 05:16:04,520 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chown pbsuser /tmp/PtlPbsJobScript8kXdux 2017-10-21 05:16:04,539 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chgrp tstgrp00 /tmp/PtlPbsJobScript8kXdux 2017-10-21 05:16:04,558 INFOCLI linux-kt3k: sudo -H -u pbsuser /opt/pbs/bin/qsub -q highp -l walltime=60 -l vntype=cray_compute /tmp/PtlPbsJobScript8kXdux 2017-10-21 05:16:04,592 INFO submit to linux-kt3k as pbsuser: job 108.linux-kt3k OrderedDict([('queue', 'highp'), ('Resource_List.walltime', '60'), ('Resource_List.vntype', 'cray_compute')]) 2017-10-21 05:16:04,592 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 108.linux-kt3k 2017-10-21 05:16:04,605 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 108.linux-kt3k got: job_state = Q 2017-10-21 05:16:05,107 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:16:05,120 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 108.linux-kt3k 2017-10-21 05:16:05,129 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 108.linux-kt3k attempt: 2 got: job_state = Q 2017-10-21 05:16:05,629 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:16:05,639 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 108.linux-kt3k 2017-10-21 05:16:05,648 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 108.linux-kt3k attempt: 3 got: job_state = Q 2017-10-21 05:16:06,150 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:16:06,161 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 108.linux-kt3k 2017-10-21 05:16:06,169 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 108.linux-kt3k attempt: 4 got: job_state = Q 2017-10-21 05:16:06,671 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:16:06,683 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 108.linux-kt3k 2017-10-21 05:16:06,692 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 108.linux-kt3k attempt: 5 got: job_state = Q 2017-10-21 05:16:07,194 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:16:07,206 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 108.linux-kt3k 2017-10-21 05:16:07,215 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 108.linux-kt3k attempt: 6 got: job_state = Q 2017-10-21 05:16:07,716 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:16:07,730 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 108.linux-kt3k 2017-10-21 05:16:07,739 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 108.linux-kt3k attempt: 7 got: job_state = Q 2017-10-21 05:16:08,241 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:16:08,253 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 108.linux-kt3k 2017-10-21 05:16:08,261 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 108.linux-kt3k attempt: 8 got: job_state = Q 2017-10-21 05:16:08,763 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:16:08,774 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 108.linux-kt3k 2017-10-21 05:16:08,783 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 108.linux-kt3k attempt: 9 got: job_state = Q 2017-10-21 05:16:09,284 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:16:09,296 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 108.linux-kt3k 2017-10-21 05:16:09,304 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 108.linux-kt3k attempt: 10 got: job_state = Q 2017-10-21 05:16:09,805 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:16:09,819 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 108.linux-kt3k 2017-10-21 05:16:09,828 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 108.linux-kt3k attempt: 11 got: job_state = Q 2017-10-21 05:16:10,330 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:16:10,341 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 108.linux-kt3k 2017-10-21 05:16:10,350 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 108.linux-kt3k attempt: 12 got: job_state = Q 2017-10-21 05:16:10,851 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:16:10,862 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 108.linux-kt3k 2017-10-21 05:16:10,871 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 108.linux-kt3k attempt: 13 got: job_state = Q 2017-10-21 05:16:11,372 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:16:11,384 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 108.linux-kt3k 2017-10-21 05:16:11,392 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 108.linux-kt3k attempt: 14 got: job_state = Q 2017-10-21 05:16:11,893 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:16:11,907 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 108.linux-kt3k 2017-10-21 05:16:11,923 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 108.linux-kt3k attempt: 15 got: job_state = Q 2017-10-21 05:16:12,426 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:16:12,439 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 108.linux-kt3k 2017-10-21 05:16:12,448 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 108.linux-kt3k attempt: 16 got: job_state = Q 2017-10-21 05:16:12,950 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:16:12,963 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 108.linux-kt3k 2017-10-21 05:16:12,971 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 108.linux-kt3k attempt: 17 got: job_state = Q 2017-10-21 05:16:13,473 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:16:13,487 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 108.linux-kt3k 2017-10-21 05:16:13,496 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 108.linux-kt3k attempt: 18 got: job_state = Q 2017-10-21 05:16:13,998 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:16:14,019 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 108.linux-kt3k 2017-10-21 05:16:14,032 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 108.linux-kt3k attempt: 19 got: substate = 41 2017-10-21 05:16:14,534 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-10-21 05:16:14,548 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 108.linux-kt3k 2017-10-21 05:16:14,557 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 108.linux-kt3k attempt: 20 ... OK 2017-10-21 05:16:14,558 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 107.linux-kt3k 2017-10-21 05:16:14,567 INFO expect on server linux-kt3k: estimated.start_time set job 107.linux-kt3k ... OK 2017-10-21 05:16:14,568 INFO status on linux-kt3k: job 107.linux-kt3k estimated.start_time 2017-10-21 05:16:14,568 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 107.linux-kt3k 2017-10-21 05:16:14,577 INFO scheduler linux-kt3k log match: searching for "107.linux-kt3k;Job is a top job"... OK 2017-10-21 05:16:14,577 INFO ========================================= 2017-10-21 05:16:14,577 INFO Entered TestSuspendResumeOnCray tearDown 2017-10-21 05:16:14,577 INFO ========================================= 2017-10-21 05:16:14,577 INFO ========================================== 2017-10-21 05:16:14,577 INFO Completed TestSuspendResumeOnCray tearDown 2017-10-21 05:16:14,577 INFO ========================================== 2017-10-21 05:16:14,578 INFO ok 2017-10-21 05:16:14,578 INFO 2017-10-21 05:16:14,578 INFO ====================================================================== 2017-10-21 05:16:14,578 INFO FAILED: test_basic_admin_suspend_restart (tests.functional.pbs_cray_suspend_resume.TestSuspendResumeOnCray) 2017-10-21 05:16:14,578 INFO ___m_oo_m___ 2017-10-21 05:16:14,578 INFO Traceback (most recent call last): File "/tmp/PBSPro_18.2.0/tests/functional/pbs_cray_suspend_resume.py", line 121, in test_basic_admin_suspend_restart self.server.expect(NODE, {'state': 'maintenance'}, id=vname) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7983, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/ptl/lib/pbs_testlib.py", line 7792, in expect raise PtlExpectError(rc=1, rv=False, msg=_msg) PtlExpectError: rc=1, rv=False, msg=expected on server linux-kt3k: state = maintenance node linux-kt3k_2 attempt: 60 got: state = job-exclusive,maintenance 2017-10-21 05:16:14,578 INFO ================================================================================ failed: test_basic_admin_suspend_restart (tests.functional.pbs_cray_suspend_resume.TestSuspendResumeOnCray) Test cases with failures: TestSuspendResumeOnCray.test_basic_admin_suspend_restart Test suites with failures: TestSuspendResumeOnCray run: 8, succeeded: 7, failed: 1, errors: 0, skipped: 0, timedout: 0 Tests run in 0:04:27.467306 2017-10-21 05:16:14,579 INFO Cleaning up temporary files