2017-09-30 05:13:09,700 INFO input command: pbs_benchpress -f tests/functional/pbs_cray_suspend_resume.py -o simulator_test_results.txt 2017-09-30 05:13:09,700 INFO param: None 2017-09-30 05:13:09,700 INFO ptl version: 17.1.0 2017-09-30 05:13:09,700 INFO platform: Linux linux-kt3k 3.12.28-4-default #1 SMP Thu Sep 25 17:02:34 UTC 2014 (9879bd4) x86_64 x86_64 2017-09-30 05:13:09,701 INFO python version: 2.7.7 2017-09-30 05:13:09,701 INFO user: root 2017-09-30 05:13:09,701 INFO -------------------------------------------------------------------------------- 2017-09-30 05:13:09,701 INFO Cleaning up temporary files 2017-09-30 05:13:09,707 INFO ====================================================================== 2017-09-30 05:13:09,707 INFO suite name: TestSuspendResumeOnCray 2017-09-30 05:13:09,707 INFO suite docstring: Test special cases where suspend/resume functionality differs on cray as compared to other platforms. This test suite expects the platform to be 'cray' and assumes that suspend/resume feature is enabled on it. 2017-09-30 05:13:09,708 INFO ====================================================================== 2017-09-30 05:13:09,708 INFO =========================================== 2017-09-30 05:13:09,708 INFO Entered TestSuspendResumeOnCray setUpClass 2017-09-30 05:13:09,708 INFO =========================================== 2017-09-30 05:13:09,709 INFOCLI2 linux-kt3k: id pbsuser 2017-09-30 05:13:09,712 INFOCLI2 linux-kt3k: id pbsuser1 2017-09-30 05:13:09,715 INFOCLI2 linux-kt3k: id pbsuser2 2017-09-30 05:13:09,719 INFOCLI2 linux-kt3k: id pbsuser3 2017-09-30 05:13:09,729 INFO server linux-kt3k: server operating mode set to cli 2017-09-30 05:13:09,729 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-09-30 05:13:09,737 INFO server linux-kt3k: version 18.2.0.20170929182825 2017-09-30 05:13:09,737 INFO expect action: created new action kicksched 2017-09-30 05:13:09,737 INFO expect action: added action kicksched to server linux-kt3k 2017-09-30 05:13:09,763 INFOCLI2 linux-kt3k: sudo -H /opt/pbs/sbin/pbsfs 2017-09-30 05:13:09,787 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2017-09-30 05:13:09,806 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2017-09-30 05:13:09,826 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list sched default 2017-09-30 05:13:09,862 INFOCLI2 linux-kt3k: sudo -H /opt/pbs/sbin/pbsfs 2017-09-30 05:13:09,890 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2017-09-30 05:13:09,910 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2017-09-30 05:13:09,931 INFO ============================================ 2017-09-30 05:13:09,931 INFO Completed TestSuspendResumeOnCray setUpClass 2017-09-30 05:13:09,931 INFO ============================================ 2017-09-30 05:13:09,932 INFO test name: test_default_restrict_res_to_release_on_suspend_setting (tests.functional.pbs_cray_suspend_resume.TestSuspendResumeOnCray)... 2017-09-30 05:13:09,932 INFO test start time: Sat Sep 30 05:13:09 2017 2017-09-30 05:13:09,932 INFO test docstring: Check that on Cray restrict_res_to_release_on_suspend is always set to 'ncpus' by default 2017-09-30 05:13:09,932 INFO ====================================== 2017-09-30 05:13:09,932 INFO Entered TestSuspendResumeOnCray setUp 2017-09-30 05:13:09,932 INFO ====================================== 2017-09-30 05:13:09,932 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-09-30 05:13:09,940 INFO status on linux-kt3k: server 2017-09-30 05:13:09,941 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-09-30 05:13:09,950 INFO manager on linux-kt3k: set server {'managers': (3, 'root@*')} 2017-09-30 05:13:09,950 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c set server managers-=root@* 2017-09-30 05:13:09,976 INFO manager on linux-kt3k: set server {'managers': (2, 'root@*')} 2017-09-30 05:13:09,976 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c set server managers+=root@* 2017-09-30 05:13:10,001 INFO server linux-kt3k: reverting configuration to defaults 2017-09-30 05:13:10,001 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-09-30 05:13:10,010 INFO select on linux-kt3k: __ALL__ 2017-09-30 05:13:10,010 INFOCLI linux-kt3k: /opt/pbs/bin/qselect 2017-09-30 05:13:10,018 INFO delete job on linux-kt3k: 0.linux-kt3k 2017-09-30 05:13:10,019 INFOCLI linux-kt3k: /opt/pbs/bin/qdel -W force 0.linux-kt3k 2017-09-30 05:13:10,041 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 0.linux-kt3k 2017-09-30 05:13:10,057 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f @linux-kt3k 2017-09-30 05:13:10,073 INFO expect on server linux-kt3k: job_state set 0 job ... OK 2017-09-30 05:13:10,073 INFOCLI linux-kt3k: /opt/pbs/bin/pbs_rstat -f 2017-09-30 05:13:10,103 INFO manager on linux-kt3k: unset server ['restrict_res_to_release_on_suspend'] 2017-09-30 05:13:10,103 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c unset server restrict_res_to_release_on_suspend 2017-09-30 05:13:10,128 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-09-30 05:13:10,167 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Qf @linux-kt3k 2017-09-30 05:13:10,176 INFO status on linux-kt3k: node 2017-09-30 05:13:10,176 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v -a 2017-09-30 05:13:10,260 INFO manager on linux-kt3k: delete queue workq 2017-09-30 05:13:10,260 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c delete queue workq 2017-09-30 05:13:10,270 INFO server linux-kt3k: expect offset set to 0.5 2017-09-30 05:13:10,770 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Qf workq@linux-kt3k 2017-09-30 05:13:10,780 INFO expect on server linux-kt3k: unset queue workq ... OK 2017-09-30 05:13:10,780 INFO manager on linux-kt3k: create queue workq {'started': 'True', 'queue_type': 'Execution', 'enabled': 'True'} 2017-09-30 05:13:10,780 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c create queue workq started=True,queue_type=Execution,enabled=True 2017-09-30 05:13:10,790 INFO server linux-kt3k: expect offset set to 0.5 2017-09-30 05:13:11,292 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Qf workq@linux-kt3k 2017-09-30 05:13:11,303 INFO expect on server linux-kt3k: started set True || queue_type set Execution || enabled set True queue workq ... OK 2017-09-30 05:13:11,303 INFOCLI2 linux-kt3k: sudo -H cmp /opt/pbs/lib/python/altair/pbs_hooks/PBS_translate_mpp.HK /var/spool/pbs/server_priv/hooks/PBS_translate_mpp.HK 2017-09-30 05:13:11,322 INFOCLI2 linux-kt3k: which cp 2017-09-30 05:13:11,324 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /opt/pbs/lib/python/altair/pbs_hooks/PBS_translate_mpp.HK /var/spool/pbs/server_priv/hooks/PBS_translate_mpp.HK 2017-09-30 05:13:11,344 INFOCLI2 linux-kt3k: which chmod 2017-09-30 05:13:11,347 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/server_priv/hooks/PBS_translate_mpp.HK 2017-09-30 05:13:11,367 INFO server linux-kt3k: sent signal -HUP 2017-09-30 05:13:11,367 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/server_priv/server.lock 2017-09-30 05:13:11,386 INFOCLI2 linux-kt3k: sudo -H kill -HUP 40356 2017-09-30 05:13:11,406 INFO manager on linux-kt3k: set pbshook PBS_translate_mpp {'enabled': 'true'} 2017-09-30 05:13:11,406 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c set pbshook PBS_translate_mpp enabled=true 2017-09-30 05:13:11,431 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-09-30 05:13:11,457 INFO manager on linux-kt3k: set server {'default_queue': 'workq', 'restrict_res_to_release_on_suspend': 'ncpus'} 2017-09-30 05:13:11,457 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server default_queue=workq,restrict_res_to_release_on_suspend=ncpus 2017-09-30 05:13:11,470 INFO status on linux-kt3k: resource 2017-09-30 05:13:11,470 INFO manager on linux-kt3k: list resource 2017-09-30 05:13:11,470 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource 2017-09-30 05:13:11,481 INFO manager on linux-kt3k: delete resource PBScraylabel_G34_IL,PBScraylabel_BAX_IL,PBScrayorder,PBScrayhost,PBScraynid 2017-09-30 05:13:11,481 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c delete resource PBScraylabel_G34_IL,PBScraylabel_BAX_IL,PBScrayorder,PBScrayhost,PBScraynid 2017-09-30 05:13:12,366 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource PBScraylabel_G34_IL 2017-09-30 05:13:12,376 INFO expect on server linux-kt3k: unset resource PBScraylabel_G34_IL ... OK 2017-09-30 05:13:12,376 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource PBScraylabel_BAX_IL 2017-09-30 05:13:12,384 INFO expect on server linux-kt3k: unset resource PBScraylabel_BAX_IL ... OK 2017-09-30 05:13:12,385 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource PBScrayorder 2017-09-30 05:13:12,393 INFO expect on server linux-kt3k: unset resource PBScrayorder ... OK 2017-09-30 05:13:12,394 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource PBScrayhost 2017-09-30 05:13:12,403 INFO expect on server linux-kt3k: unset resource PBScrayhost ... OK 2017-09-30 05:13:12,403 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource PBScraynid 2017-09-30 05:13:12,411 INFO expect on server linux-kt3k: unset resource PBScraynid ... OK 2017-09-30 05:13:12,411 INFOCLI status on linux-kt3k: server license_count 2017-09-30 05:13:12,411 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-09-30 05:13:12,419 INFO server: linux-kt3k licensed 2017-09-30 05:13:12,428 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/server_priv/comm.lock 2017-09-30 05:13:12,463 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-09-30 05:13:12,482 INFO scheduler linux-kt3k: reverting configuration to defaults 2017-09-30 05:13:12,483 INFO manager on linux-kt3k: delete sched @default 2017-09-30 05:13:12,483 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c delete sched @default 2017-09-30 05:13:12,492 INFO manager on linux-kt3k: list sched 2017-09-30 05:13:12,492 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list sched 2017-09-30 05:13:12,501 INFO manager on linux-kt3k: unset sched ['sched_cycle_length'] 2017-09-30 05:13:12,502 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c unset sched sched_cycle_length 2017-09-30 05:13:12,511 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/dedicated_time 2017-09-30 05:13:12,529 INFOCLI2 linux-kt3k: cmp /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-09-30 05:13:12,531 INFO scheduler linux-kt3k: reverting holidays file to default 2017-09-30 05:13:12,531 INFOCLI2 linux-kt3k: cmp /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-09-30 05:13:12,533 INFOCLI2 linux-kt3k: cmp /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-09-30 05:13:12,536 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-09-30 05:13:12,554 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-09-30 05:13:12,575 INFO scheduler linux-kt3k: sent signal -HUP 2017-09-30 05:13:12,575 INFOCLI2 linux-kt3k: sudo -H kill -HUP 40110 2017-09-30 05:13:12,599 INFOCLI2 linux-kt3k: sudo -H /opt/pbs/sbin/pbsfs -e 2017-09-30 05:13:12,644 INFO scheduler linux-kt3k: config {'resources': '"ncpus, mem, arch, host, vnode, aoe, eoe, vntype"'} 2017-09-30 05:13:12,646 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /var/spool/pbs/sched_priv/sched_config /var/spool/pbs/sched_priv/sched_config.bak 2017-09-30 05:13:12,668 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /tmp/PtlPbs5d1nJL /var/spool/pbs/sched_priv/sched_config 2017-09-30 05:13:12,690 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-09-30 05:13:12,708 INFOCLI2 linux-kt3k: which chown 2017-09-30 05:13:12,711 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chown root /var/spool/pbs/sched_priv/sched_config 2017-09-30 05:13:12,729 INFOCLI2 linux-kt3k: which chgrp 2017-09-30 05:13:12,731 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chgrp root /var/spool/pbs/sched_priv/sched_config 2017-09-30 05:13:12,753 INFO scheduler linux-kt3k: sent signal -HUP 2017-09-30 05:13:12,753 INFOCLI2 linux-kt3k: sudo -H kill -HUP 40110 2017-09-30 05:13:12,773 INFO scheduler linux-kt3k log match: searching for "Error reading line" - No match 2017-09-30 05:13:13,784 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-09-30 05:13:13,812 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-09-30 05:13:13,833 INFOCLI2 linux-kt3k: sudo -H /opt/pbs/sbin/pbs_mom --version 2017-09-30 05:13:13,871 INFO mom linux-kt3k: reverting configuration to defaults 2017-09-30 05:13:13,871 INFOCLI2 linux-kt3k: which rm 2017-09-30 05:13:13,874 INFOCLI2 linux-kt3k: sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2017-09-30 05:13:13,892 INFOCLI2 linux-kt3k: sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2017-09-30 05:13:13,912 INFOCLI linux-kt3k: sudo -H /opt/pbs/sbin/pbs_mom -s list 2017-09-30 05:13:13,952 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /tmp/PtlPbsJYDlYb /var/spool/pbs/mom_priv/config 2017-09-30 05:13:13,976 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/mom_priv/config 2017-09-30 05:13:13,997 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chown root /var/spool/pbs/mom_priv/config 2017-09-30 05:13:14,019 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chgrp root /var/spool/pbs/mom_priv/config 2017-09-30 05:13:14,039 INFO mom linux-kt3k: sent signal -HUP 2017-09-30 05:13:14,040 INFOCLI2 linux-kt3k: sudo -H kill -HUP 40107 2017-09-30 05:13:14,073 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-09-30 05:13:14,097 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v -a 2017-09-30 05:13:14,143 INFO manager on linux-kt3k: delete node 2017-09-30 05:13:14,143 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c delete node @default 2017-09-30 05:13:14,252 INFO manager on linux-kt3k: create node linux-kt3k 2017-09-30 05:13:14,253 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c create node linux-kt3k 2017-09-30 05:13:14,282 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k 2017-09-30 05:13:14,303 INFO expect on server linux-kt3k: state = free node linux-kt3k got: state = state-unknown,down 2017-09-30 05:13:15,305 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k 2017-09-30 05:13:15,317 INFO expect on server linux-kt3k: state = free node linux-kt3k attempt: 2 got: state = state-unknown,down 2017-09-30 05:13:16,318 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k 2017-09-30 05:13:16,424 INFO expect on server linux-kt3k: state = free node linux-kt3k attempt: 3 ... OK 2017-09-30 05:13:16,425 INFO ======================================= 2017-09-30 05:13:16,425 INFO Completed TestSuspendResumeOnCray setUp 2017-09-30 05:13:16,425 INFO ======================================= 2017-09-30 05:13:16,425 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-09-30 05:13:16,433 INFO expect on server linux-kt3k: restrict_res_to_release_on_suspend = ncpus server linux-kt3k ... OK 2017-09-30 05:13:16,434 INFO ========================================= 2017-09-30 05:13:16,434 INFO Entered TestSuspendResumeOnCray tearDown 2017-09-30 05:13:16,434 INFO ========================================= 2017-09-30 05:13:16,434 INFO ========================================== 2017-09-30 05:13:16,434 INFO Completed TestSuspendResumeOnCray tearDown 2017-09-30 05:13:16,434 INFO ========================================== 2017-09-30 05:13:16,434 INFO ok 2017-09-30 05:13:16,435 INFO test name: test_exclusive_job_not_suspended (tests.functional.pbs_cray_suspend_resume.TestSuspendResumeOnCray)... 2017-09-30 05:13:16,435 INFO test start time: Sat Sep 30 05:13:16 2017 2017-09-30 05:13:16,435 INFO test docstring: If a running job is a job with exclusive placement then this job can not be suspended. This test is checking for a log message which is an unstable interface and may need change in future when interface changes. 2017-09-30 05:13:16,435 INFO ====================================== 2017-09-30 05:13:16,435 INFO Entered TestSuspendResumeOnCray setUp 2017-09-30 05:13:16,435 INFO ====================================== 2017-09-30 05:13:16,435 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-09-30 05:13:16,445 INFO status on linux-kt3k: server 2017-09-30 05:13:16,445 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-09-30 05:13:16,454 INFO manager on linux-kt3k: set server {'managers': (3, 'root@*')} 2017-09-30 05:13:16,454 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c set server managers-=root@* 2017-09-30 05:13:16,481 INFO manager on linux-kt3k: set server {'managers': (2, 'root@*')} 2017-09-30 05:13:16,481 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c set server managers+=root@* 2017-09-30 05:13:16,509 INFO server linux-kt3k: reverting configuration to defaults 2017-09-30 05:13:16,510 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-09-30 05:13:16,520 INFO select on linux-kt3k: __ALL__ 2017-09-30 05:13:16,520 INFOCLI linux-kt3k: /opt/pbs/bin/qselect 2017-09-30 05:13:16,528 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f @linux-kt3k 2017-09-30 05:13:16,536 INFO expect on server linux-kt3k: job_state set 0 job ... OK 2017-09-30 05:13:16,537 INFOCLI linux-kt3k: /opt/pbs/bin/pbs_rstat -f 2017-09-30 05:13:16,544 INFO manager on linux-kt3k: unset server ['restrict_res_to_release_on_suspend'] 2017-09-30 05:13:16,545 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c unset server restrict_res_to_release_on_suspend 2017-09-30 05:13:16,556 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-09-30 05:13:16,588 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Qf @linux-kt3k 2017-09-30 05:13:16,596 INFO status on linux-kt3k: node 2017-09-30 05:13:16,596 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v -a 2017-09-30 05:13:16,649 INFO manager on linux-kt3k: delete queue workq 2017-09-30 05:13:16,649 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c delete queue workq 2017-09-30 05:13:16,660 INFO server linux-kt3k: expect offset set to 0.5 2017-09-30 05:13:17,161 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Qf workq@linux-kt3k 2017-09-30 05:13:17,171 INFO expect on server linux-kt3k: unset queue workq ... OK 2017-09-30 05:13:17,171 INFO manager on linux-kt3k: create queue workq {'started': 'True', 'queue_type': 'Execution', 'enabled': 'True'} 2017-09-30 05:13:17,171 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c create queue workq started=True,queue_type=Execution,enabled=True 2017-09-30 05:13:17,181 INFO server linux-kt3k: expect offset set to 0.5 2017-09-30 05:13:17,683 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Qf workq@linux-kt3k 2017-09-30 05:13:17,693 INFO expect on server linux-kt3k: started set True || queue_type set Execution || enabled set True queue workq ... OK 2017-09-30 05:13:17,693 INFOCLI2 linux-kt3k: sudo -H cmp /opt/pbs/lib/python/altair/pbs_hooks/PBS_translate_mpp.HK /var/spool/pbs/server_priv/hooks/PBS_translate_mpp.HK 2017-09-30 05:13:17,712 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /opt/pbs/lib/python/altair/pbs_hooks/PBS_translate_mpp.HK /var/spool/pbs/server_priv/hooks/PBS_translate_mpp.HK 2017-09-30 05:13:17,734 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/server_priv/hooks/PBS_translate_mpp.HK 2017-09-30 05:13:17,757 INFO server linux-kt3k: sent signal -HUP 2017-09-30 05:13:17,757 INFOCLI2 linux-kt3k: sudo -H kill -HUP 40356 2017-09-30 05:13:17,780 INFO manager on linux-kt3k: set pbshook PBS_translate_mpp {'enabled': 'true'} 2017-09-30 05:13:17,780 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c set pbshook PBS_translate_mpp enabled=true 2017-09-30 05:13:17,810 INFOCLI linux-kt3k: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-09-30 05:13:17,840 INFO manager on linux-kt3k: set server {'default_queue': 'workq', 'restrict_res_to_release_on_suspend': 'ncpus'} 2017-09-30 05:13:17,841 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server default_queue=workq,restrict_res_to_release_on_suspend=ncpus 2017-09-30 05:13:17,854 INFO status on linux-kt3k: resource 2017-09-30 05:13:17,854 INFO manager on linux-kt3k: list resource 2017-09-30 05:13:17,854 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource 2017-09-30 05:13:17,863 INFO manager on linux-kt3k: delete resource PBScraylabel_G34_IL,PBScraylabel_BAX_IL,PBScrayorder,PBScrayhost,PBScraynid 2017-09-30 05:13:17,863 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c delete resource PBScraylabel_G34_IL,PBScraylabel_BAX_IL,PBScrayorder,PBScrayhost,PBScraynid 2017-09-30 05:13:18,733 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource PBScraylabel_G34_IL 2017-09-30 05:13:18,748 INFO expect on server linux-kt3k: unset resource PBScraylabel_G34_IL ... OK 2017-09-30 05:13:18,748 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource PBScraylabel_BAX_IL 2017-09-30 05:13:18,757 INFO expect on server linux-kt3k: unset resource PBScraylabel_BAX_IL ... OK 2017-09-30 05:13:18,757 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource PBScrayorder 2017-09-30 05:13:18,765 INFO expect on server linux-kt3k: unset resource PBScrayorder ... OK 2017-09-30 05:13:18,766 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource PBScrayhost 2017-09-30 05:13:18,776 INFO expect on server linux-kt3k: unset resource PBScrayhost ... OK 2017-09-30 05:13:18,777 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list resource PBScraynid 2017-09-30 05:13:18,785 INFO expect on server linux-kt3k: unset resource PBScraynid ... OK 2017-09-30 05:13:18,786 INFOCLI status on linux-kt3k: server license_count 2017-09-30 05:13:18,786 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -Bf linux-kt3k 2017-09-30 05:13:18,795 INFO server: linux-kt3k licensed 2017-09-30 05:13:18,805 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/server_priv/comm.lock 2017-09-30 05:13:18,838 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-09-30 05:13:18,857 INFO scheduler linux-kt3k: reverting configuration to defaults 2017-09-30 05:13:18,857 INFO manager on linux-kt3k: delete sched @default 2017-09-30 05:13:18,857 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c delete sched @default 2017-09-30 05:13:18,866 INFO manager on linux-kt3k: list sched 2017-09-30 05:13:18,866 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c list sched 2017-09-30 05:13:18,875 INFO manager on linux-kt3k: unset sched ['sched_cycle_length'] 2017-09-30 05:13:18,875 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c unset sched sched_cycle_length 2017-09-30 05:13:18,885 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/dedicated_time 2017-09-30 05:13:18,904 INFOCLI2 linux-kt3k: cmp /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-09-30 05:13:18,907 INFO scheduler linux-kt3k: reverting holidays file to default 2017-09-30 05:13:18,907 INFOCLI2 linux-kt3k: cmp /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-09-30 05:13:18,909 INFOCLI2 linux-kt3k: cmp /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-09-30 05:13:18,911 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-09-30 05:13:18,931 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-09-30 05:13:18,952 INFO scheduler linux-kt3k: sent signal -HUP 2017-09-30 05:13:18,953 INFOCLI2 linux-kt3k: sudo -H kill -HUP 40110 2017-09-30 05:13:18,976 INFOCLI2 linux-kt3k: sudo -H /opt/pbs/sbin/pbsfs -e 2017-09-30 05:13:19,029 INFO scheduler linux-kt3k: config {'resources': '"ncpus, mem, arch, host, vnode, aoe, eoe, vntype"'} 2017-09-30 05:13:19,030 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /var/spool/pbs/sched_priv/sched_config /var/spool/pbs/sched_priv/sched_config.bak 2017-09-30 05:13:19,049 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cp -p /tmp/PtlPbsMbornU /var/spool/pbs/sched_priv/sched_config 2017-09-30 05:13:19,069 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-09-30 05:13:19,088 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chown root /var/spool/pbs/sched_priv/sched_config 2017-09-30 05:13:19,111 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chgrp root /var/spool/pbs/sched_priv/sched_config 2017-09-30 05:13:19,135 INFO scheduler linux-kt3k: sent signal -HUP 2017-09-30 05:13:19,135 INFOCLI2 linux-kt3k: sudo -H kill -HUP 40110 2017-09-30 05:13:19,158 INFO scheduler linux-kt3k log match: searching for "Error reading line" - No match 2017-09-30 05:13:20,170 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-09-30 05:13:20,197 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-09-30 05:13:20,217 INFO mom linux-kt3k: reverting configuration to defaults 2017-09-30 05:13:20,217 INFOCLI2 linux-kt3k: sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2017-09-30 05:13:20,236 INFOCLI2 linux-kt3k: sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2017-09-30 05:13:20,254 INFOCLI linux-kt3k: sudo -H /opt/pbs/sbin/pbs_mom -s list 2017-09-30 05:13:20,295 INFO mom linux-kt3k: sent signal -HUP 2017-09-30 05:13:20,295 INFOCLI2 linux-kt3k: sudo -H kill -HUP 40107 2017-09-30 05:13:20,325 INFOCLI2 linux-kt3k: sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-09-30 05:13:20,349 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v -a 2017-09-30 05:13:20,394 INFO manager on linux-kt3k: delete node 2017-09-30 05:13:20,395 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c delete node @default 2017-09-30 05:13:20,536 INFO manager on linux-kt3k: create node linux-kt3k 2017-09-30 05:13:20,536 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c create node linux-kt3k 2017-09-30 05:13:20,547 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k 2017-09-30 05:13:20,557 INFO expect on server linux-kt3k: state = free node linux-kt3k got: state = state-unknown,down 2017-09-30 05:13:21,558 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k 2017-09-30 05:13:21,568 INFO expect on server linux-kt3k: state = free node linux-kt3k attempt: 2 got: state = state-unknown,down 2017-09-30 05:13:22,569 INFOCLI linux-kt3k: /opt/pbs/bin/pbsnodes -s linux-kt3k -v linux-kt3k 2017-09-30 05:13:22,694 INFO expect on server linux-kt3k: state = free node linux-kt3k attempt: 3 ... OK 2017-09-30 05:13:22,695 INFO ======================================= 2017-09-30 05:13:22,695 INFO Completed TestSuspendResumeOnCray setUp 2017-09-30 05:13:22,695 INFO ======================================= 2017-09-30 05:13:22,695 INFO job: executable set to /bin/sleep with arguments: 100 2017-09-30 05:13:22,695 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chmod 0755 /tmp/PtlPbsJobScriptMAd6B3 2017-09-30 05:13:22,719 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chown pbsuser /tmp/PtlPbsJobScriptMAd6B3 2017-09-30 05:13:22,742 INFOCLI2 linux-kt3k: sudo -H /usr/bin/chgrp tstgrp00 /tmp/PtlPbsJobScriptMAd6B3 2017-09-30 05:13:22,763 INFOCLI linux-kt3k: sudo -H -u pbsuser /opt/pbs/bin/qsub -l place=excl -l select=1:ncpus=1:vntype=cray_compute /tmp/PtlPbsJobScriptMAd6B3 2017-09-30 05:13:22,793 INFO submit to linux-kt3k as pbsuser: job 1.linux-kt3k OrderedDict([('Resource_List.place', 'excl'), ('Resource_List.select', '1:ncpus=1:vntype=cray_compute')]) 2017-09-30 05:13:22,793 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 1.linux-kt3k 2017-09-30 05:13:22,803 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 1.linux-kt3k got: job_state = Q 2017-09-30 05:13:23,304 INFOCLI linux-kt3k: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-09-30 05:13:23,318 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 1.linux-kt3k 2017-09-30 05:13:23,327 INFO expect on server linux-kt3k: job_state = R && substate = 42 job 1.linux-kt3k attempt: 2 ... OK 2017-09-30 05:13:23,327 INFO signal on linux-kt3k: 1.linux-kt3k with signal = suspend 2017-09-30 05:13:23,327 INFOCLI linux-kt3k: /opt/pbs/bin/qsig -s suspend 1.linux-kt3k 2017-09-30 05:13:23,395 ERROR err: ['qsig: Switching ALPS reservation failed 1.linux-kt3k'] 2017-09-30 05:13:23,395 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 1.linux-kt3k 2017-09-30 05:13:23,406 INFO expect on server linux-kt3k: exec_host set job 1.linux-kt3k ... OK 2017-09-30 05:13:23,406 INFO status on linux-kt3k: job 1.linux-kt3k 2017-09-30 05:13:23,406 INFOCLI linux-kt3k: /opt/pbs/bin/qstat -f 1.linux-kt3k 2017-09-30 05:13:23,416 INFO mom linux-kt3k log match: searching for "BASIL;ERROR: ALPS error: apsched: at least resid .* is exclusive" - using regular expression ... OK 2017-09-30 05:13:23,417 INFO ========================================= 2017-09-30 05:13:23,417 INFO Entered TestSuspendResumeOnCray tearDown 2017-09-30 05:13:23,417 INFO ========================================= 2017-09-30 05:13:23,417 INFO ========================================== 2017-09-30 05:13:23,417 INFO Completed TestSuspendResumeOnCray tearDown 2017-09-30 05:13:23,417 INFO ========================================== 2017-09-30 05:13:23,417 INFO ok 2017-09-30 05:13:23,418 INFO ================================================================================ run: 2, succeeded: 2, failed: 0, errors: 0, skipped: 0, timedout: 0 Tests run in 0:00:13.710760 2017-09-30 05:13:23,418 INFO Cleaning up temporary files