2017-06-09 01:34:33,822 INFO input command: pbs_benchpress -f pbs_fairshare.py -o /tmp/ptl-fairshare.out 2017-06-09 01:34:33,822 INFO param: None 2017-06-09 01:34:33,822 INFO ptl version: 1.0.0 2017-06-09 01:34:33,822 INFO platform: Linux nimitz 3.10.0-514.10.2.el7.x86_64 #1 SMP Fri Mar 3 00:04:05 UTC 2017 x86_64 x86_64 2017-06-09 01:34:33,822 INFO python version: 2.7.5 2017-06-09 01:34:33,822 INFO user: bmann 2017-06-09 01:34:33,822 INFO -------------------------------------------------------------------------------- 2017-06-09 01:34:33,822 INFO Cleaning up temporary files 2017-06-09 01:34:33,835 INFO ====================================================================== 2017-06-09 01:34:33,835 INFO suite name: TestFairshare 2017-06-09 01:34:33,835 INFO suite docstring: Test the pbs_sched fairshare functionality. Note there are two fairshare tests in the standard smoke test that are not included here. 2017-06-09 01:34:33,835 INFO ====================================================================== 2017-06-09 01:34:33,835 INFO ================================= 2017-06-09 01:34:33,835 INFO Entered TestFairshare setUpClass 2017-06-09 01:34:33,835 INFO ================================= 2017-06-09 01:34:33,835 INFOCLI2 nimitz: id pbsuser 2017-06-09 01:34:33,838 INFOCLI2 nimitz: id pbsuser1 2017-06-09 01:34:33,841 INFOCLI2 nimitz: id pbsuser2 2017-06-09 01:34:33,844 INFOCLI2 nimitz: id pbsuser3 2017-06-09 01:34:33,856 INFO server nimitz: server operating mode set to cli 2017-06-09 01:34:33,856 INFOCLI nimitz: /opt/pbs/bin/qstat -Bf nimitz 2017-06-09 01:34:33,865 INFO server nimitz: version 14.0.1 2017-06-09 01:34:33,865 INFO expect action: created new action kicksched 2017-06-09 01:34:33,865 INFO expect action: added action kicksched to server nimitz 2017-06-09 01:34:33,885 INFOCLI2 nimitz: sudo -H /opt/pbs/sbin/pbsfs 2017-06-09 01:34:33,899 INFOCLI2 nimitz: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:33,909 INFOCLI nimitz: /opt/pbs/bin/qmgr -c list sched 2017-06-09 01:34:33,919 INFOCLI2 nimitz: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2017-06-09 01:34:33,934 INFO ================================== 2017-06-09 01:34:33,934 INFO Completed TestFairshare setUpClass 2017-06-09 01:34:33,934 INFO ================================== 2017-06-09 01:34:33,934 INFO test name: test_fairshare_formula (tests.functional.pbs_fairshare.TestFairshare)... 2017-06-09 01:34:33,934 INFO test start time: Fri Jun 9 01:34:33 2017 2017-06-09 01:34:33,934 INFO test docstring: Test fairshare in the formula. Make sure the fairshare_tree_usage is correct 2017-06-09 01:34:33,935 INFO ============================ 2017-06-09 01:34:33,935 INFO Entered TestFairshare setUp 2017-06-09 01:34:33,935 INFO ============================ 2017-06-09 01:34:33,935 INFOCLI nimitz: /opt/pbs/bin/qstat -Bf nimitz 2017-06-09 01:34:33,944 INFO status on nimitz: server 2017-06-09 01:34:33,945 INFOCLI nimitz: /opt/pbs/bin/qstat -Bf nimitz 2017-06-09 01:34:33,958 INFO manager on nimitz: set server {'managers': (3, 'bmann@*')} 2017-06-09 01:34:33,958 INFOCLI nimitz: sudo -H /opt/pbs/bin/qmgr -c set server managers-=bmann@* 2017-06-09 01:34:33,982 INFO manager on nimitz: set server {'managers': (2, 'bmann@*')} 2017-06-09 01:34:33,982 INFOCLI nimitz: sudo -H /opt/pbs/bin/qmgr -c set server managers+=bmann@* 2017-06-09 01:34:34,011 INFO server nimitz: reverting configuration to defaults 2017-06-09 01:34:34,012 INFOCLI nimitz: /opt/pbs/bin/qstat -Bf nimitz 2017-06-09 01:34:34,024 INFO select on nimitz: __ALL__ 2017-06-09 01:34:34,024 INFOCLI nimitz: /opt/pbs/bin/qselect 2017-06-09 01:34:34,033 INFOCLI nimitz: /opt/pbs/bin/qstat -f @nimitz 2017-06-09 01:34:34,043 INFO expect on server nimitz: job_state set 0 job ... OK 2017-06-09 01:34:34,044 INFOCLI nimitz: /opt/pbs/bin/pbs_rstat -f 2017-06-09 01:34:34,053 INFOCLI nimitz: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-06-09 01:34:34,071 INFOCLI nimitz: /opt/pbs/bin/qstat -Qf @nimitz 2017-06-09 01:34:34,081 INFO manager on nimitz: delete queue workq 2017-06-09 01:34:34,081 INFOCLI nimitz: /opt/pbs/bin/qmgr -c delete queue workq 2017-06-09 01:34:34,092 INFO server nimitz: expect offset set to 0.5 2017-06-09 01:34:34,597 INFOCLI nimitz: /opt/pbs/bin/qstat -Qf workq@nimitz 2017-06-09 01:34:34,609 INFO expect on server nimitz: unset queue workq ... OK 2017-06-09 01:34:34,610 INFO manager on nimitz: create queue workq {'started': 'True', 'queue_type': 'Execution', 'enabled': 'True'} 2017-06-09 01:34:34,610 INFOCLI nimitz: /opt/pbs/bin/qmgr -c create queue workq started=True,queue_type=Execution,enabled=True 2017-06-09 01:34:34,621 INFO server nimitz: expect offset set to 0.5 2017-06-09 01:34:35,128 INFOCLI nimitz: /opt/pbs/bin/qstat -Qf workq@nimitz 2017-06-09 01:34:35,142 INFO expect on server nimitz: started set True || queue_type set Execution || enabled set True queue workq ... OK 2017-06-09 01:34:35,142 INFO manager on nimitz: set server {'default_queue': 'workq'} 2017-06-09 01:34:35,142 INFOCLI nimitz: /opt/pbs/bin/qmgr -c set server default_queue=workq 2017-06-09 01:34:35,162 INFO status on nimitz: resource 2017-06-09 01:34:35,162 INFO manager on nimitz: list resource 2017-06-09 01:34:35,162 INFOCLI nimitz: /opt/pbs/bin/qmgr -c list resource 2017-06-09 01:34:35,173 INFOCLI status on nimitz: server license_count 2017-06-09 01:34:35,173 INFOCLI nimitz: /opt/pbs/bin/qstat -Bf nimitz 2017-06-09 01:34:35,185 INFO server: nimitz licensed 2017-06-09 01:34:35,206 INFOCLI2 nimitz: sudo -H /usr/bin/cat /var/spool/pbs/server_priv/comm.lock 2017-06-09 01:34:35,234 INFOCLI2 nimitz: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-06-09 01:34:35,244 INFO scheduler nimitz: reverting configuration to defaults 2017-06-09 01:34:35,244 INFO manager on nimitz: list sched 2017-06-09 01:34:35,244 INFOCLI nimitz: /opt/pbs/bin/qmgr -c list sched 2017-06-09 01:34:35,254 INFO manager on nimitz: unset sched ['job_sort_formula_threshold', 'sched_cycle_length'] 2017-06-09 01:34:35,254 INFOCLI nimitz: /opt/pbs/bin/qmgr -c unset sched job_sort_formula_threshold,sched_cycle_length 2017-06-09 01:34:35,276 INFOCLI2 nimitz: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/dedicated_time 2017-06-09 01:34:35,286 INFOCLI2 nimitz: cmp /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:35,295 ERROR err: ['cmp: /var/spool/pbs/sched_priv/resource_group: Permission denied'] 2017-06-09 01:34:35,296 INFOCLI2 nimitz: which cp 2017-06-09 01:34:35,298 INFOCLI2 nimitz: sudo -H /usr/bin/cp -p /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:35,309 INFOCLI2 nimitz: which chmod 2017-06-09 01:34:35,312 INFOCLI2 nimitz: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:35,322 INFO scheduler nimitz: reverting holidays file to default 2017-06-09 01:34:35,322 INFOCLI2 nimitz: cmp /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-06-09 01:34:35,324 ERROR err: ['cmp: /var/spool/pbs/sched_priv/holidays: Permission denied'] 2017-06-09 01:34:35,324 INFOCLI2 nimitz: sudo -H /usr/bin/cp -p /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-06-09 01:34:35,342 INFOCLI2 nimitz: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/holidays 2017-06-09 01:34:35,353 INFO scheduler nimitz: Parsing holidays file 2017-06-09 01:34:35,353 INFOCLI2 nimitz: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2017-06-09 01:34:35,363 INFOCLI2 nimitz: cmp /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-06-09 01:34:35,366 ERROR err: ['cmp: /var/spool/pbs/sched_priv/sched_config: Permission denied'] 2017-06-09 01:34:35,366 INFOCLI2 nimitz: sudo -H /usr/bin/cp -p /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-06-09 01:34:35,387 INFOCLI2 nimitz: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-06-09 01:34:35,397 INFO scheduler nimitz: sent signal -HUP 2017-06-09 01:34:35,397 INFOCLI2 nimitz: sudo -H kill -HUP 14109 2017-06-09 01:34:35,407 INFOCLI2 nimitz: sudo -H /opt/pbs/sbin/pbsfs -e 2017-06-09 01:34:35,446 INFOCLI2 nimitz: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-06-09 01:34:35,469 INFOCLI2 nimitz: sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-06-09 01:34:35,479 INFOCLI2 nimitz: sudo -H /opt/pbs/sbin/pbs_mom --version 2017-06-09 01:34:35,512 INFO mom nimitz: reverting configuration to defaults 2017-06-09 01:34:35,512 INFOCLI2 nimitz: which rm 2017-06-09 01:34:35,515 INFOCLI2 nimitz: sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2017-06-09 01:34:35,525 INFOCLI2 nimitz: sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2017-06-09 01:34:35,534 INFOCLI nimitz: sudo -H /opt/pbs/sbin/pbs_mom -s list 2017-06-09 01:34:35,631 INFOCLI2 nimitz: sudo -H /usr/bin/cp -p /tmp/PtlPbsXZpZwK /var/spool/pbs/mom_priv/config 2017-06-09 01:34:35,641 INFOCLI2 nimitz: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/mom_priv/config 2017-06-09 01:34:35,652 INFOCLI2 nimitz: which chown 2017-06-09 01:34:35,654 INFOCLI2 nimitz: sudo -H /usr/bin/chown root /var/spool/pbs/mom_priv/config 2017-06-09 01:34:35,664 INFOCLI2 nimitz: which chgrp 2017-06-09 01:34:35,666 INFOCLI2 nimitz: sudo -H /usr/bin/chgrp root /var/spool/pbs/mom_priv/config 2017-06-09 01:34:35,690 INFO mom nimitz: sent signal -HUP 2017-06-09 01:34:35,690 INFOCLI2 nimitz: sudo -H kill -HUP 9035 2017-06-09 01:34:35,717 INFOCLI2 nimitz: sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-06-09 01:34:35,728 INFOCLI nimitz: /opt/pbs/bin/pbsnodes -s nimitz -v -a 2017-06-09 01:34:35,778 INFO status on nimitz: node nimitz 2017-06-09 01:34:35,779 INFOCLI nimitz: /opt/pbs/bin/pbsnodes -s nimitz -v nimitz 2017-06-09 01:34:35,789 INFOCLI nimitz: /opt/pbs/bin/pbsnodes -s nimitz -v nimitz 2017-06-09 01:34:35,800 INFO expect on server nimitz: state = free node nimitz ... OK 2017-06-09 01:34:35,800 INFO ============================= 2017-06-09 01:34:35,800 INFO Completed TestFairshare setUp 2017-06-09 01:34:35,800 INFO ============================= 2017-06-09 01:34:35,800 INFOCLI2 nimitz: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:35,810 INFO creating tree node: group1 2017-06-09 01:34:35,811 INFOCLI2 nimitz: sudo -H /usr/bin/cp -p /tmp/PtlPbsnrkIRn /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:35,821 INFOCLI2 nimitz: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:35,834 INFOCLI2 nimitz: sudo -H /usr/bin/chown root /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:35,852 INFOCLI2 nimitz: sudo -H /usr/bin/chgrp root /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:35,863 INFO creating tree node: group2 2017-06-09 01:34:35,864 INFOCLI2 nimitz: sudo -H /usr/bin/cp -p /tmp/PtlPbsJGGfud /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:35,881 INFOCLI2 nimitz: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:35,893 INFOCLI2 nimitz: sudo -H /usr/bin/chown root /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:35,902 INFOCLI2 nimitz: sudo -H /usr/bin/chgrp root /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:35,911 INFO creating tree node: pbsuser 2017-06-09 01:34:35,912 INFOCLI2 nimitz: sudo -H /usr/bin/cp -p /tmp/PtlPbswemcTV /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:35,922 INFOCLI2 nimitz: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:35,931 INFOCLI2 nimitz: sudo -H /usr/bin/chown root /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:35,940 INFOCLI2 nimitz: sudo -H /usr/bin/chgrp root /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:35,949 INFO creating tree node: pbsuser1 2017-06-09 01:34:35,949 INFOCLI2 nimitz: sudo -H /usr/bin/cp -p /tmp/PtlPbs7Dtpcm /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:35,959 INFOCLI2 nimitz: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:35,968 INFOCLI2 nimitz: sudo -H /usr/bin/chown root /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:35,977 INFOCLI2 nimitz: sudo -H /usr/bin/chgrp root /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:35,988 INFO creating tree node: pbsuser2 2017-06-09 01:34:35,988 INFOCLI2 nimitz: sudo -H /usr/bin/cp -p /tmp/PtlPbsC8dE1n /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:35,998 INFOCLI2 nimitz: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:36,008 INFOCLI2 nimitz: sudo -H /usr/bin/chown root /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:36,017 INFOCLI2 nimitz: sudo -H /usr/bin/chgrp root /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:36,029 INFO creating tree node: pbsuser3 2017-06-09 01:34:36,029 INFOCLI2 nimitz: sudo -H /usr/bin/cp -p /tmp/PtlPbshnLuIJ /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:36,039 INFOCLI2 nimitz: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:36,048 INFOCLI2 nimitz: sudo -H /usr/bin/chown root /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:36,059 INFOCLI2 nimitz: sudo -H /usr/bin/chgrp root /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:36,070 INFOCLI2 nimitz: sudo -H /opt/pbs/sbin/pbsfs -s pbsuser 100 2017-06-09 01:34:36,084 INFOCLI2 nimitz: sudo -H /opt/pbs/sbin/pbsfs -s pbsuser1 100 2017-06-09 01:34:36,100 INFOCLI2 nimitz: sudo -H /opt/pbs/sbin/pbsfs -s pbsuser3 1000 2017-06-09 01:34:36,114 INFO scheduler nimitz: config {'log_filter': 2048} 2017-06-09 01:34:36,115 INFOCLI2 nimitz: sudo -H /usr/bin/cp -p /var/spool/pbs/sched_priv/sched_config /var/spool/pbs/sched_priv/sched_config.bak 2017-06-09 01:34:36,125 INFOCLI2 nimitz: sudo -H /usr/bin/cp -p /tmp/PtlPbs63gAxp /var/spool/pbs/sched_priv/sched_config 2017-06-09 01:34:36,135 INFOCLI2 nimitz: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-06-09 01:34:36,145 INFOCLI2 nimitz: sudo -H /usr/bin/chown root /var/spool/pbs/sched_priv/sched_config 2017-06-09 01:34:36,154 INFOCLI2 nimitz: sudo -H /usr/bin/chgrp root /var/spool/pbs/sched_priv/sched_config 2017-06-09 01:34:36,165 INFO scheduler nimitz: sent signal -HUP 2017-06-09 01:34:36,165 INFOCLI2 nimitz: sudo -H kill -HUP 14109 2017-06-09 01:34:36,175 INFO scheduler nimitz log match: searching for "Error reading line" - No match 2017-06-09 01:34:37,178 INFO manager on nimitz: set server {'scheduling': 'False'} 2017-06-09 01:34:37,179 INFOCLI nimitz: /opt/pbs/bin/qmgr -c set server scheduling=False 2017-06-09 01:34:37,206 INFO manager on nimitz: set server {'job_sort_formula': 'fairshare_tree_usage'} 2017-06-09 01:34:37,206 INFOCLI nimitz: sudo -H /opt/pbs/bin/qmgr -c set server job_sort_formula=fairshare_tree_usage 2017-06-09 01:34:37,361 INFO job: executable set to /bin/sleep with arguments: 100 2017-06-09 01:34:37,361 INFOCLI nimitz: sudo -H -u pbsuser /opt/pbs/bin/qsub -- /bin/sleep 100 2017-06-09 01:34:37,457 INFO submit to nimitz as pbsuser: job 82.nimitz 2017-06-09 01:34:37,457 INFO job: executable set to /bin/sleep with arguments: 100 2017-06-09 01:34:37,458 INFOCLI nimitz: sudo -H -u pbsuser1 /opt/pbs/bin/qsub -- /bin/sleep 100 2017-06-09 01:34:37,486 INFO submit to nimitz as pbsuser1: job 83.nimitz 2017-06-09 01:34:37,487 INFO job: executable set to /bin/sleep with arguments: 100 2017-06-09 01:34:37,487 INFOCLI nimitz: sudo -H -u pbsuser2 /opt/pbs/bin/qsub -- /bin/sleep 100 2017-06-09 01:34:37,515 INFO submit to nimitz as pbsuser2: job 84.nimitz 2017-06-09 01:34:37,515 INFO job: executable set to /bin/sleep with arguments: 100 2017-06-09 01:34:37,515 INFOCLI nimitz: sudo -H -u pbsuser3 /opt/pbs/bin/qsub -- /bin/sleep 100 2017-06-09 01:34:37,545 INFO submit to nimitz as pbsuser3: job 85.nimitz 2017-06-09 01:34:37,545 INFO manager on nimitz: set server {'scheduling': 'True'} 2017-06-09 01:34:37,546 INFOCLI nimitz: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-06-09 01:34:37,568 INFO scheduler nimitz log match: searching for "82.nimitz;Formula Evaluation = 0.1253" - No match 2017-06-09 01:34:38,570 INFO scheduler nimitz log match: searching for "83.nimitz;Formula Evaluation = 0.1253"... OK 2017-06-09 01:34:38,571 INFO scheduler nimitz log match: searching for "84.nimitz;Formula Evaluation = 0.5004"... OK 2017-06-09 01:34:38,571 INFO scheduler nimitz log match: searching for "85.nimitz;Formula Evaluation = 0.8330"... OK 2017-06-09 01:34:38,571 INFO =============================== 2017-06-09 01:34:38,571 INFO Entered TestFairshare tearDown 2017-06-09 01:34:38,571 INFO =============================== 2017-06-09 01:34:38,571 INFO ================================ 2017-06-09 01:34:38,571 INFO Completed TestFairshare tearDown 2017-06-09 01:34:38,571 INFO ================================ 2017-06-09 01:34:38,573 INFO ok 2017-06-09 01:34:38,573 INFO test name: test_fairshare_formula2 (tests.functional.pbs_fairshare.TestFairshare)... 2017-06-09 01:34:38,573 INFO test start time: Fri Jun 9 01:34:38 2017 2017-06-09 01:34:38,573 INFO test docstring: Test fairshare in the formula. Make sure the fairshare_perc is correct 2017-06-09 01:34:38,573 INFO ============================ 2017-06-09 01:34:38,574 INFO Entered TestFairshare setUp 2017-06-09 01:34:38,574 INFO ============================ 2017-06-09 01:34:38,574 INFOCLI nimitz: /opt/pbs/bin/qstat -Bf nimitz 2017-06-09 01:34:38,584 INFO status on nimitz: server 2017-06-09 01:34:38,584 INFOCLI nimitz: /opt/pbs/bin/qstat -Bf nimitz 2017-06-09 01:34:38,594 INFO manager on nimitz: set server {'managers': (3, 'bmann@*')} 2017-06-09 01:34:38,595 INFOCLI nimitz: sudo -H /opt/pbs/bin/qmgr -c set server managers-=bmann@* 2017-06-09 01:34:38,629 INFO manager on nimitz: set server {'managers': (2, 'bmann@*')} 2017-06-09 01:34:38,629 INFOCLI nimitz: sudo -H /opt/pbs/bin/qmgr -c set server managers+=bmann@* 2017-06-09 01:34:38,649 INFO server nimitz: reverting configuration to defaults 2017-06-09 01:34:38,650 INFOCLI nimitz: /opt/pbs/bin/qstat -Bf nimitz 2017-06-09 01:34:38,659 INFO select on nimitz: __ALL__ 2017-06-09 01:34:38,659 INFOCLI nimitz: /opt/pbs/bin/qselect 2017-06-09 01:34:38,668 INFO delete job on nimitz: 82.nimitz, 83.nimitz, 84.nimitz, 85.nimitz 2017-06-09 01:34:38,668 INFOCLI nimitz: /opt/pbs/bin/qdel -W force 82.nimitz 83.nimitz 84.nimitz 85.nimitz 2017-06-09 01:34:38,707 INFOCLI nimitz: /opt/pbs/bin/qstat -f 82.nimitz 2017-06-09 01:34:38,724 INFOCLI nimitz: /opt/pbs/bin/qstat -f 83.nimitz 2017-06-09 01:34:38,737 INFOCLI nimitz: /opt/pbs/bin/qstat -f 84.nimitz 2017-06-09 01:34:38,752 INFOCLI nimitz: /opt/pbs/bin/qstat -f 85.nimitz 2017-06-09 01:34:38,767 INFOCLI nimitz: /opt/pbs/bin/qstat -f @nimitz 2017-06-09 01:34:38,776 INFO expect on server nimitz: job_state set 0 job ... OK 2017-06-09 01:34:38,776 INFOCLI nimitz: /opt/pbs/bin/pbs_rstat -f 2017-06-09 01:34:38,786 INFO manager on nimitz: unset server ['job_sort_formula'] 2017-06-09 01:34:38,786 INFOCLI nimitz: sudo -H /opt/pbs/bin/qmgr -c unset server job_sort_formula 2017-06-09 01:34:38,806 INFOCLI nimitz: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-06-09 01:34:38,827 INFOCLI nimitz: /opt/pbs/bin/qstat -Qf @nimitz 2017-06-09 01:34:38,838 INFO manager on nimitz: delete queue workq 2017-06-09 01:34:38,838 INFOCLI nimitz: /opt/pbs/bin/qmgr -c delete queue workq 2017-06-09 01:34:38,849 INFO server nimitz: expect offset set to 0.5 2017-06-09 01:34:39,365 INFOCLI nimitz: /opt/pbs/bin/qstat -Qf workq@nimitz 2017-06-09 01:34:39,378 INFO expect on server nimitz: unset queue workq ... OK 2017-06-09 01:34:39,378 INFO manager on nimitz: create queue workq {'started': 'True', 'queue_type': 'Execution', 'enabled': 'True'} 2017-06-09 01:34:39,379 INFOCLI nimitz: /opt/pbs/bin/qmgr -c create queue workq started=True,queue_type=Execution,enabled=True 2017-06-09 01:34:39,399 INFO server nimitz: expect offset set to 0.5 2017-06-09 01:34:39,909 INFOCLI nimitz: /opt/pbs/bin/qstat -Qf workq@nimitz 2017-06-09 01:34:39,923 INFO expect on server nimitz: started set True || queue_type set Execution || enabled set True queue workq ... OK 2017-06-09 01:34:39,923 INFO manager on nimitz: set server {'default_queue': 'workq'} 2017-06-09 01:34:39,924 INFOCLI nimitz: /opt/pbs/bin/qmgr -c set server default_queue=workq 2017-06-09 01:34:39,936 INFO status on nimitz: resource 2017-06-09 01:34:39,936 INFO manager on nimitz: list resource 2017-06-09 01:34:39,936 INFOCLI nimitz: /opt/pbs/bin/qmgr -c list resource 2017-06-09 01:34:39,947 INFOCLI status on nimitz: server license_count 2017-06-09 01:34:39,947 INFOCLI nimitz: /opt/pbs/bin/qstat -Bf nimitz 2017-06-09 01:34:39,957 INFO server: nimitz licensed 2017-06-09 01:34:39,970 INFOCLI2 nimitz: sudo -H /usr/bin/cat /var/spool/pbs/server_priv/comm.lock 2017-06-09 01:34:39,999 INFOCLI2 nimitz: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-06-09 01:34:40,009 INFO scheduler nimitz: reverting configuration to defaults 2017-06-09 01:34:40,010 INFO manager on nimitz: list sched 2017-06-09 01:34:40,010 INFOCLI nimitz: /opt/pbs/bin/qmgr -c list sched 2017-06-09 01:34:40,020 INFO manager on nimitz: unset sched ['job_sort_formula_threshold', 'sched_cycle_length'] 2017-06-09 01:34:40,020 INFOCLI nimitz: /opt/pbs/bin/qmgr -c unset sched job_sort_formula_threshold,sched_cycle_length 2017-06-09 01:34:40,031 INFOCLI2 nimitz: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/dedicated_time 2017-06-09 01:34:40,049 INFOCLI2 nimitz: cmp /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:40,052 ERROR err: ['cmp: /var/spool/pbs/sched_priv/resource_group: Permission denied'] 2017-06-09 01:34:40,052 INFOCLI2 nimitz: sudo -H /usr/bin/cp -p /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:40,062 INFOCLI2 nimitz: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:40,071 INFO scheduler nimitz: reverting holidays file to default 2017-06-09 01:34:40,071 INFOCLI2 nimitz: cmp /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-06-09 01:34:40,073 ERROR err: ['cmp: /var/spool/pbs/sched_priv/holidays: Permission denied'] 2017-06-09 01:34:40,073 INFOCLI2 nimitz: sudo -H /usr/bin/cp -p /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-06-09 01:34:40,084 INFOCLI2 nimitz: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/holidays 2017-06-09 01:34:40,093 INFO scheduler nimitz: Parsing holidays file 2017-06-09 01:34:40,093 INFOCLI2 nimitz: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2017-06-09 01:34:40,103 INFOCLI2 nimitz: cmp /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-06-09 01:34:40,105 ERROR err: ['cmp: /var/spool/pbs/sched_priv/sched_config: Permission denied'] 2017-06-09 01:34:40,105 INFOCLI2 nimitz: sudo -H /usr/bin/cp -p /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-06-09 01:34:40,115 INFOCLI2 nimitz: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-06-09 01:34:40,127 INFO scheduler nimitz: sent signal -HUP 2017-06-09 01:34:40,127 INFOCLI2 nimitz: sudo -H kill -HUP 14109 2017-06-09 01:34:40,137 INFOCLI2 nimitz: sudo -H /opt/pbs/sbin/pbsfs -e 2017-06-09 01:34:40,176 INFOCLI2 nimitz: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-06-09 01:34:40,206 INFOCLI2 nimitz: sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-06-09 01:34:40,216 INFO mom nimitz: reverting configuration to defaults 2017-06-09 01:34:40,217 INFOCLI2 nimitz: sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2017-06-09 01:34:40,227 INFOCLI2 nimitz: sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2017-06-09 01:34:40,243 INFOCLI nimitz: sudo -H /opt/pbs/sbin/pbs_mom -s list 2017-06-09 01:34:40,314 INFO mom nimitz: sent signal -HUP 2017-06-09 01:34:40,314 INFOCLI2 nimitz: sudo -H kill -HUP 9035 2017-06-09 01:34:40,343 INFOCLI2 nimitz: sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-06-09 01:34:40,353 INFOCLI nimitz: /opt/pbs/bin/pbsnodes -s nimitz -v -a 2017-06-09 01:34:40,362 INFO status on nimitz: node nimitz 2017-06-09 01:34:40,362 INFOCLI nimitz: /opt/pbs/bin/pbsnodes -s nimitz -v nimitz 2017-06-09 01:34:40,372 INFOCLI nimitz: /opt/pbs/bin/pbsnodes -s nimitz -v nimitz 2017-06-09 01:34:40,385 INFO expect on server nimitz: state = free node nimitz ... OK 2017-06-09 01:34:40,386 INFO ============================= 2017-06-09 01:34:40,386 INFO Completed TestFairshare setUp 2017-06-09 01:34:40,386 INFO ============================= 2017-06-09 01:34:40,386 INFOCLI2 nimitz: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:40,396 INFO creating tree node: group1 2017-06-09 01:34:40,396 INFOCLI2 nimitz: sudo -H /usr/bin/cp -p /tmp/PtlPbs8FXGqg /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:40,406 INFOCLI2 nimitz: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:40,416 INFOCLI2 nimitz: sudo -H /usr/bin/chown root /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:40,426 INFOCLI2 nimitz: sudo -H /usr/bin/chgrp root /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:40,436 INFO creating tree node: group2 2017-06-09 01:34:40,436 INFOCLI2 nimitz: sudo -H /usr/bin/cp -p /tmp/PtlPbsYD8U0Q /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:40,446 INFOCLI2 nimitz: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:40,457 INFOCLI2 nimitz: sudo -H /usr/bin/chown root /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:40,466 INFOCLI2 nimitz: sudo -H /usr/bin/chgrp root /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:40,475 INFO creating tree node: pbsuser 2017-06-09 01:34:40,475 INFOCLI2 nimitz: sudo -H /usr/bin/cp -p /tmp/PtlPbsRXu9BW /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:40,498 INFOCLI2 nimitz: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:40,508 INFOCLI2 nimitz: sudo -H /usr/bin/chown root /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:40,518 INFOCLI2 nimitz: sudo -H /usr/bin/chgrp root /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:40,529 INFO creating tree node: pbsuser1 2017-06-09 01:34:40,530 INFOCLI2 nimitz: sudo -H /usr/bin/cp -p /tmp/PtlPbsvJK5x6 /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:40,539 INFOCLI2 nimitz: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:40,549 INFOCLI2 nimitz: sudo -H /usr/bin/chown root /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:40,558 INFOCLI2 nimitz: sudo -H /usr/bin/chgrp root /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:40,568 INFO creating tree node: pbsuser2 2017-06-09 01:34:40,569 INFOCLI2 nimitz: sudo -H /usr/bin/cp -p /tmp/PtlPbs1COSDu /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:40,578 INFOCLI2 nimitz: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:40,588 INFOCLI2 nimitz: sudo -H /usr/bin/chown root /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:40,598 INFOCLI2 nimitz: sudo -H /usr/bin/chgrp root /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:40,609 INFO creating tree node: pbsuser3 2017-06-09 01:34:40,610 INFOCLI2 nimitz: sudo -H /usr/bin/cp -p /tmp/PtlPbs41db62 /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:40,619 INFOCLI2 nimitz: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:40,630 INFOCLI2 nimitz: sudo -H /usr/bin/chown root /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:40,642 INFOCLI2 nimitz: sudo -H /usr/bin/chgrp root /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:40,657 INFOCLI2 nimitz: sudo -H /opt/pbs/sbin/pbsfs -s pbsuser 100 2017-06-09 01:34:40,672 INFOCLI2 nimitz: sudo -H /opt/pbs/sbin/pbsfs -s pbsuser1 100 2017-06-09 01:34:40,686 INFOCLI2 nimitz: sudo -H /opt/pbs/sbin/pbsfs -s pbsuser3 1000 2017-06-09 01:34:40,702 INFO scheduler nimitz: config {'log_filter': 2048} 2017-06-09 01:34:40,703 INFOCLI2 nimitz: sudo -H /usr/bin/cp -p /var/spool/pbs/sched_priv/sched_config /var/spool/pbs/sched_priv/sched_config.bak 2017-06-09 01:34:40,713 INFOCLI2 nimitz: sudo -H /usr/bin/cp -p /tmp/PtlPbsG5W3pT /var/spool/pbs/sched_priv/sched_config 2017-06-09 01:34:40,732 INFOCLI2 nimitz: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-06-09 01:34:40,743 INFOCLI2 nimitz: sudo -H /usr/bin/chown root /var/spool/pbs/sched_priv/sched_config 2017-06-09 01:34:40,753 INFOCLI2 nimitz: sudo -H /usr/bin/chgrp root /var/spool/pbs/sched_priv/sched_config 2017-06-09 01:34:40,764 INFO scheduler nimitz: sent signal -HUP 2017-06-09 01:34:40,764 INFOCLI2 nimitz: sudo -H kill -HUP 14109 2017-06-09 01:34:40,774 INFO scheduler nimitz log match: searching for "Error reading line" - No match 2017-06-09 01:34:41,778 INFO manager on nimitz: set server {'scheduling': 'False'} 2017-06-09 01:34:41,779 INFOCLI nimitz: /opt/pbs/bin/qmgr -c set server scheduling=False 2017-06-09 01:34:41,796 INFO manager on nimitz: set server {'job_sort_formula': 'fairshare_perc'} 2017-06-09 01:34:41,796 INFOCLI nimitz: sudo -H /opt/pbs/bin/qmgr -c set server job_sort_formula=fairshare_perc 2017-06-09 01:34:41,824 INFO job: executable set to /bin/sleep with arguments: 100 2017-06-09 01:34:41,824 INFOCLI nimitz: sudo -H -u pbsuser /opt/pbs/bin/qsub -- /bin/sleep 100 2017-06-09 01:34:41,842 INFO submit to nimitz as pbsuser: job 86.nimitz 2017-06-09 01:34:41,842 INFO job: executable set to /bin/sleep with arguments: 100 2017-06-09 01:34:41,842 INFOCLI nimitz: sudo -H -u pbsuser1 /opt/pbs/bin/qsub -- /bin/sleep 100 2017-06-09 01:34:41,867 INFO submit to nimitz as pbsuser1: job 87.nimitz 2017-06-09 01:34:41,868 INFO job: executable set to /bin/sleep with arguments: 100 2017-06-09 01:34:41,868 INFOCLI nimitz: sudo -H -u pbsuser2 /opt/pbs/bin/qsub -- /bin/sleep 100 2017-06-09 01:34:41,892 INFO submit to nimitz as pbsuser2: job 88.nimitz 2017-06-09 01:34:41,892 INFO job: executable set to /bin/sleep with arguments: 100 2017-06-09 01:34:41,893 INFOCLI nimitz: sudo -H -u pbsuser3 /opt/pbs/bin/qsub -- /bin/sleep 100 2017-06-09 01:34:41,912 INFO submit to nimitz as pbsuser3: job 89.nimitz 2017-06-09 01:34:41,912 INFO manager on nimitz: set server {'scheduling': 'True'} 2017-06-09 01:34:41,912 INFOCLI nimitz: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-06-09 01:34:41,930 INFO scheduler nimitz log match: searching for "86.nimitz;Formula Evaluation = 0.2" - No match 2017-06-09 01:34:42,936 INFO scheduler nimitz log match: searching for "87.nimitz;Formula Evaluation = 0.2"... OK 2017-06-09 01:34:42,936 INFO scheduler nimitz log match: searching for "88.nimitz;Formula Evaluation = 0.36"... OK 2017-06-09 01:34:42,937 INFO scheduler nimitz log match: searching for "89.nimitz;Formula Evaluation = 0.24"... OK 2017-06-09 01:34:42,937 INFO =============================== 2017-06-09 01:34:42,937 INFO Entered TestFairshare tearDown 2017-06-09 01:34:42,937 INFO =============================== 2017-06-09 01:34:42,937 INFO ================================ 2017-06-09 01:34:42,937 INFO Completed TestFairshare tearDown 2017-06-09 01:34:42,938 INFO ================================ 2017-06-09 01:34:42,938 INFO ok 2017-06-09 01:34:42,939 INFO test name: test_fairshare_formula3 (tests.functional.pbs_fairshare.TestFairshare)... 2017-06-09 01:34:42,939 INFO test start time: Fri Jun 9 01:34:42 2017 2017-06-09 01:34:42,939 INFO test docstring: Test fairshare in the formula. Make sure entities with small usage are negatively affected by their high usage siblings. Make sure that jobs run in the correct order. 2017-06-09 01:34:42,939 INFO ============================ 2017-06-09 01:34:42,939 INFO Entered TestFairshare setUp 2017-06-09 01:34:42,939 INFO ============================ 2017-06-09 01:34:42,940 INFOCLI nimitz: /opt/pbs/bin/qstat -Bf nimitz 2017-06-09 01:34:42,953 INFO status on nimitz: server 2017-06-09 01:34:42,953 INFOCLI nimitz: /opt/pbs/bin/qstat -Bf nimitz 2017-06-09 01:34:42,964 INFO manager on nimitz: set server {'managers': (3, 'bmann@*')} 2017-06-09 01:34:42,964 INFOCLI nimitz: sudo -H /opt/pbs/bin/qmgr -c set server managers-=bmann@* 2017-06-09 01:34:42,991 INFO manager on nimitz: set server {'managers': (2, 'bmann@*')} 2017-06-09 01:34:42,991 INFOCLI nimitz: sudo -H /opt/pbs/bin/qmgr -c set server managers+=bmann@* 2017-06-09 01:34:43,011 INFO server nimitz: reverting configuration to defaults 2017-06-09 01:34:43,011 INFOCLI nimitz: /opt/pbs/bin/qstat -Bf nimitz 2017-06-09 01:34:43,022 INFO select on nimitz: __ALL__ 2017-06-09 01:34:43,022 INFOCLI nimitz: /opt/pbs/bin/qselect 2017-06-09 01:34:43,032 INFO delete job on nimitz: 86.nimitz, 87.nimitz, 88.nimitz, 89.nimitz 2017-06-09 01:34:43,032 INFOCLI nimitz: /opt/pbs/bin/qdel -W force 86.nimitz 87.nimitz 88.nimitz 89.nimitz 2017-06-09 01:34:43,079 INFOCLI nimitz: /opt/pbs/bin/qstat -f 86.nimitz 2017-06-09 01:34:43,092 INFOCLI nimitz: /opt/pbs/bin/qstat -f 87.nimitz 2017-06-09 01:34:43,123 INFOCLI nimitz: /opt/pbs/bin/qstat -f 88.nimitz 2017-06-09 01:34:43,138 INFOCLI nimitz: /opt/pbs/bin/qstat -f 89.nimitz 2017-06-09 01:34:43,152 INFOCLI nimitz: /opt/pbs/bin/qstat -f @nimitz 2017-06-09 01:34:43,162 INFO expect on server nimitz: job_state set 0 job ... OK 2017-06-09 01:34:43,162 INFOCLI nimitz: /opt/pbs/bin/pbs_rstat -f 2017-06-09 01:34:43,172 INFO manager on nimitz: unset server ['job_sort_formula'] 2017-06-09 01:34:43,172 INFOCLI nimitz: sudo -H /opt/pbs/bin/qmgr -c unset server job_sort_formula 2017-06-09 01:34:43,196 INFOCLI nimitz: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-06-09 01:34:43,222 INFOCLI nimitz: /opt/pbs/bin/qstat -Qf @nimitz 2017-06-09 01:34:43,231 INFO manager on nimitz: delete queue workq 2017-06-09 01:34:43,232 INFOCLI nimitz: /opt/pbs/bin/qmgr -c delete queue workq 2017-06-09 01:34:43,242 INFO server nimitz: expect offset set to 0.5 2017-06-09 01:34:43,744 INFOCLI nimitz: /opt/pbs/bin/qstat -Qf workq@nimitz 2017-06-09 01:34:43,758 INFO expect on server nimitz: unset queue workq ... OK 2017-06-09 01:34:43,759 INFO manager on nimitz: create queue workq {'started': 'True', 'queue_type': 'Execution', 'enabled': 'True'} 2017-06-09 01:34:43,759 INFOCLI nimitz: /opt/pbs/bin/qmgr -c create queue workq started=True,queue_type=Execution,enabled=True 2017-06-09 01:34:43,771 INFO server nimitz: expect offset set to 0.5 2017-06-09 01:34:44,276 INFOCLI nimitz: /opt/pbs/bin/qstat -Qf workq@nimitz 2017-06-09 01:34:44,290 INFO expect on server nimitz: started set True || queue_type set Execution || enabled set True queue workq ... OK 2017-06-09 01:34:44,291 INFO manager on nimitz: set server {'default_queue': 'workq'} 2017-06-09 01:34:44,291 INFOCLI nimitz: /opt/pbs/bin/qmgr -c set server default_queue=workq 2017-06-09 01:34:44,319 INFO status on nimitz: resource 2017-06-09 01:34:44,319 INFO manager on nimitz: list resource 2017-06-09 01:34:44,319 INFOCLI nimitz: /opt/pbs/bin/qmgr -c list resource 2017-06-09 01:34:44,330 INFOCLI status on nimitz: server license_count 2017-06-09 01:34:44,331 INFOCLI nimitz: /opt/pbs/bin/qstat -Bf nimitz 2017-06-09 01:34:44,340 INFO server: nimitz licensed 2017-06-09 01:34:44,354 INFOCLI2 nimitz: sudo -H /usr/bin/cat /var/spool/pbs/server_priv/comm.lock 2017-06-09 01:34:44,384 INFOCLI2 nimitz: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-06-09 01:34:44,401 INFO scheduler nimitz: reverting configuration to defaults 2017-06-09 01:34:44,401 INFO manager on nimitz: list sched 2017-06-09 01:34:44,401 INFOCLI nimitz: /opt/pbs/bin/qmgr -c list sched 2017-06-09 01:34:44,411 INFO manager on nimitz: unset sched ['job_sort_formula_threshold', 'sched_cycle_length'] 2017-06-09 01:34:44,412 INFOCLI nimitz: /opt/pbs/bin/qmgr -c unset sched job_sort_formula_threshold,sched_cycle_length 2017-06-09 01:34:44,423 INFOCLI2 nimitz: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/dedicated_time 2017-06-09 01:34:44,433 INFOCLI2 nimitz: cmp /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:44,435 ERROR err: ['cmp: /var/spool/pbs/sched_priv/resource_group: Permission denied'] 2017-06-09 01:34:44,435 INFOCLI2 nimitz: sudo -H /usr/bin/cp -p /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:44,449 INFOCLI2 nimitz: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:44,459 INFO scheduler nimitz: reverting holidays file to default 2017-06-09 01:34:44,459 INFOCLI2 nimitz: cmp /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-06-09 01:34:44,461 ERROR err: ['cmp: /var/spool/pbs/sched_priv/holidays: Permission denied'] 2017-06-09 01:34:44,461 INFOCLI2 nimitz: sudo -H /usr/bin/cp -p /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-06-09 01:34:44,471 INFOCLI2 nimitz: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/holidays 2017-06-09 01:34:44,480 INFO scheduler nimitz: Parsing holidays file 2017-06-09 01:34:44,480 INFOCLI2 nimitz: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2017-06-09 01:34:44,497 INFOCLI2 nimitz: cmp /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-06-09 01:34:44,500 ERROR err: ['cmp: /var/spool/pbs/sched_priv/sched_config: Permission denied'] 2017-06-09 01:34:44,500 INFOCLI2 nimitz: sudo -H /usr/bin/cp -p /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-06-09 01:34:44,509 INFOCLI2 nimitz: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-06-09 01:34:44,520 INFO scheduler nimitz: sent signal -HUP 2017-06-09 01:34:44,520 INFOCLI2 nimitz: sudo -H kill -HUP 14109 2017-06-09 01:34:44,530 INFOCLI2 nimitz: sudo -H /opt/pbs/sbin/pbsfs -e 2017-06-09 01:34:44,568 INFOCLI2 nimitz: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-06-09 01:34:44,592 INFOCLI2 nimitz: sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-06-09 01:34:44,602 INFO mom nimitz: reverting configuration to defaults 2017-06-09 01:34:44,602 INFOCLI2 nimitz: sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2017-06-09 01:34:44,612 INFOCLI2 nimitz: sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2017-06-09 01:34:44,621 INFOCLI nimitz: sudo -H /opt/pbs/sbin/pbs_mom -s list 2017-06-09 01:34:44,691 INFO mom nimitz: sent signal -HUP 2017-06-09 01:34:44,692 INFOCLI2 nimitz: sudo -H kill -HUP 9035 2017-06-09 01:34:44,717 INFOCLI2 nimitz: sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-06-09 01:34:44,727 INFOCLI nimitz: /opt/pbs/bin/pbsnodes -s nimitz -v -a 2017-06-09 01:34:44,737 INFO status on nimitz: node nimitz 2017-06-09 01:34:44,738 INFOCLI nimitz: /opt/pbs/bin/pbsnodes -s nimitz -v nimitz 2017-06-09 01:34:44,747 INFOCLI nimitz: /opt/pbs/bin/pbsnodes -s nimitz -v nimitz 2017-06-09 01:34:44,757 INFO expect on server nimitz: state = free node nimitz ... OK 2017-06-09 01:34:44,757 INFO ============================= 2017-06-09 01:34:44,757 INFO Completed TestFairshare setUp 2017-06-09 01:34:44,758 INFO ============================= 2017-06-09 01:34:44,758 INFOCLI2 nimitz: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:44,768 INFO creating tree node: group1 2017-06-09 01:34:44,769 INFOCLI2 nimitz: sudo -H /usr/bin/cp -p /tmp/PtlPbs7Rcrit /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:44,779 INFOCLI2 nimitz: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:44,789 INFOCLI2 nimitz: sudo -H /usr/bin/chown root /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:44,798 INFOCLI2 nimitz: sudo -H /usr/bin/chgrp root /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:44,811 INFO creating tree node: group2 2017-06-09 01:34:44,811 INFOCLI2 nimitz: sudo -H /usr/bin/cp -p /tmp/PtlPbs55IFI1 /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:44,821 INFOCLI2 nimitz: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:44,830 INFOCLI2 nimitz: sudo -H /usr/bin/chown root /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:44,841 INFOCLI2 nimitz: sudo -H /usr/bin/chgrp root /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:44,857 INFO creating tree node: pbsuser 2017-06-09 01:34:44,858 INFOCLI2 nimitz: sudo -H /usr/bin/cp -p /tmp/PtlPbsFnizWs /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:44,868 INFOCLI2 nimitz: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:44,878 INFOCLI2 nimitz: sudo -H /usr/bin/chown root /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:44,888 INFOCLI2 nimitz: sudo -H /usr/bin/chgrp root /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:44,897 INFO creating tree node: pbsuser1 2017-06-09 01:34:44,898 INFOCLI2 nimitz: sudo -H /usr/bin/cp -p /tmp/PtlPbsktm8C2 /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:44,909 INFOCLI2 nimitz: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:44,921 INFOCLI2 nimitz: sudo -H /usr/bin/chown root /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:44,930 INFOCLI2 nimitz: sudo -H /usr/bin/chgrp root /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:44,939 INFO creating tree node: pbsuser2 2017-06-09 01:34:44,940 INFOCLI2 nimitz: sudo -H /usr/bin/cp -p /tmp/PtlPbsw5NJ3r /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:44,951 INFOCLI2 nimitz: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:44,960 INFOCLI2 nimitz: sudo -H /usr/bin/chown root /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:44,970 INFOCLI2 nimitz: sudo -H /usr/bin/chgrp root /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:44,980 INFO creating tree node: pbsuser3 2017-06-09 01:34:44,981 INFOCLI2 nimitz: sudo -H /usr/bin/cp -p /tmp/PtlPbsYyiCJM /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:44,991 INFOCLI2 nimitz: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:45,000 INFOCLI2 nimitz: sudo -H /usr/bin/chown root /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:45,009 INFOCLI2 nimitz: sudo -H /usr/bin/chgrp root /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:45,020 INFOCLI2 nimitz: sudo -H /opt/pbs/sbin/pbsfs -s pbsuser 100 2017-06-09 01:34:45,034 INFOCLI2 nimitz: sudo -H /opt/pbs/sbin/pbsfs -s pbsuser1 100 2017-06-09 01:34:45,048 INFOCLI2 nimitz: sudo -H /opt/pbs/sbin/pbsfs -s pbsuser3 1000 2017-06-09 01:34:45,063 INFO scheduler nimitz: config {'log_filter': 2048} 2017-06-09 01:34:45,064 INFOCLI2 nimitz: sudo -H /usr/bin/cp -p /var/spool/pbs/sched_priv/sched_config /var/spool/pbs/sched_priv/sched_config.bak 2017-06-09 01:34:45,073 INFOCLI2 nimitz: sudo -H /usr/bin/cp -p /tmp/PtlPbsqTKMjB /var/spool/pbs/sched_priv/sched_config 2017-06-09 01:34:45,084 INFOCLI2 nimitz: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-06-09 01:34:45,098 INFOCLI2 nimitz: sudo -H /usr/bin/chown root /var/spool/pbs/sched_priv/sched_config 2017-06-09 01:34:45,107 INFOCLI2 nimitz: sudo -H /usr/bin/chgrp root /var/spool/pbs/sched_priv/sched_config 2017-06-09 01:34:45,117 INFO scheduler nimitz: sent signal -HUP 2017-06-09 01:34:45,117 INFOCLI2 nimitz: sudo -H kill -HUP 14109 2017-06-09 01:34:45,128 INFO scheduler nimitz log match: searching for "Error reading line" - No match 2017-06-09 01:34:46,135 INFO manager on nimitz: set server {'scheduling': 'False'} 2017-06-09 01:34:46,136 INFOCLI nimitz: /opt/pbs/bin/qmgr -c set server scheduling=False 2017-06-09 01:34:46,156 INFO manager on nimitz: set server {'job_sort_formula': '"pow(2,-(fairshare_tree_usage/fairshare_perc))"'} 2017-06-09 01:34:46,156 INFOCLI nimitz: sudo -H /opt/pbs/bin/qmgr -c set server job_sort_formula="pow(2,-(fairshare_tree_usage/fairshare_perc))" 2017-06-09 01:34:46,188 INFO job: executable set to /bin/sleep with arguments: 100 2017-06-09 01:34:46,188 INFOCLI nimitz: sudo -H -u pbsuser2 /opt/pbs/bin/qsub -- /bin/sleep 100 2017-06-09 01:34:46,200 INFO submit to nimitz as pbsuser2: job 90.nimitz 2017-06-09 01:34:46,200 INFO job: executable set to /bin/sleep with arguments: 100 2017-06-09 01:34:46,200 INFOCLI nimitz: sudo -H -u pbsuser3 /opt/pbs/bin/qsub -- /bin/sleep 100 2017-06-09 01:34:46,214 INFO submit to nimitz as pbsuser3: job 91.nimitz 2017-06-09 01:34:46,215 INFO job: executable set to /bin/sleep with arguments: 100 2017-06-09 01:34:46,215 INFOCLI nimitz: sudo -H -u pbsuser /opt/pbs/bin/qsub -- /bin/sleep 100 2017-06-09 01:34:46,229 INFO submit to nimitz as pbsuser: job 92.nimitz 2017-06-09 01:34:46,230 INFO job: executable set to /bin/sleep with arguments: 100 2017-06-09 01:34:46,230 INFOCLI nimitz: sudo -H -u pbsuser1 /opt/pbs/bin/qsub -- /bin/sleep 100 2017-06-09 01:34:46,249 INFO submit to nimitz as pbsuser1: job 93.nimitz 2017-06-09 01:34:46,249 INFO manager on nimitz: set server {'scheduling': 'True'} 2017-06-09 01:34:46,250 INFOCLI nimitz: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-06-09 01:34:46,264 INFO scheduler nimitz log match: searching for "90.nimitz;Formula Evaluation = 0.3816" - No match 2017-06-09 01:34:47,273 INFO scheduler nimitz log match: searching for "91.nimitz;Formula Evaluation = 0.0902"... OK 2017-06-09 01:34:47,274 INFO scheduler nimitz log match: searching for "92.nimitz;Formula Evaluation = 0.6477"... OK 2017-06-09 01:34:47,275 INFO scheduler nimitz log match: searching for "93.nimitz;Formula Evaluation = 0.6477"... OK 2017-06-09 01:34:47,275 INFO scheduler nimitz log match: searching for "Leaving Scheduling Cycle"... OK 2017-06-09 01:34:47,286 INFO =============================== 2017-06-09 01:34:47,286 INFO Entered TestFairshare tearDown 2017-06-09 01:34:47,286 INFO =============================== 2017-06-09 01:34:47,286 INFO ================================ 2017-06-09 01:34:47,286 INFO Completed TestFairshare tearDown 2017-06-09 01:34:47,286 INFO ================================ 2017-06-09 01:34:47,287 INFO ok 2017-06-09 01:34:47,287 INFO test name: test_formula_keyword (tests.functional.pbs_fairshare.TestFairshare)... 2017-06-09 01:34:47,287 INFO test start time: Fri Jun 9 01:34:47 2017 2017-06-09 01:34:47,287 INFO test docstring: Test to see if 'fairshare_tree_usage' and 'fairshare_perc' are allowed to be set in the job_sort_formula 2017-06-09 01:34:47,287 INFO ============================ 2017-06-09 01:34:47,287 INFO Entered TestFairshare setUp 2017-06-09 01:34:47,287 INFO ============================ 2017-06-09 01:34:47,288 INFOCLI nimitz: /opt/pbs/bin/qstat -Bf nimitz 2017-06-09 01:34:47,299 INFO status on nimitz: server 2017-06-09 01:34:47,299 INFOCLI nimitz: /opt/pbs/bin/qstat -Bf nimitz 2017-06-09 01:34:47,314 INFO manager on nimitz: set server {'managers': (3, 'bmann@*')} 2017-06-09 01:34:47,314 INFOCLI nimitz: sudo -H /opt/pbs/bin/qmgr -c set server managers-=bmann@* 2017-06-09 01:34:47,340 INFO manager on nimitz: set server {'managers': (2, 'bmann@*')} 2017-06-09 01:34:47,340 INFOCLI nimitz: sudo -H /opt/pbs/bin/qmgr -c set server managers+=bmann@* 2017-06-09 01:34:47,369 INFO server nimitz: reverting configuration to defaults 2017-06-09 01:34:47,369 INFOCLI nimitz: /opt/pbs/bin/qstat -Bf nimitz 2017-06-09 01:34:47,380 INFO select on nimitz: __ALL__ 2017-06-09 01:34:47,380 INFOCLI nimitz: /opt/pbs/bin/qselect 2017-06-09 01:34:47,390 INFO delete job on nimitz: 90.nimitz, 91.nimitz, 92.nimitz, 93.nimitz 2017-06-09 01:34:47,390 INFOCLI nimitz: /opt/pbs/bin/qdel -W force 90.nimitz 91.nimitz 92.nimitz 93.nimitz 2017-06-09 01:34:47,430 INFOCLI nimitz: /opt/pbs/bin/qstat -f 90.nimitz 2017-06-09 01:34:47,444 INFOCLI nimitz: /opt/pbs/bin/qstat -f 91.nimitz 2017-06-09 01:34:47,462 INFOCLI nimitz: /opt/pbs/bin/qstat -f 92.nimitz 2017-06-09 01:34:47,475 INFOCLI nimitz: /opt/pbs/bin/qstat -f 93.nimitz 2017-06-09 01:34:47,489 INFOCLI nimitz: /opt/pbs/bin/qstat -f @nimitz 2017-06-09 01:34:47,500 INFO expect on server nimitz: job_state set 0 job ... OK 2017-06-09 01:34:47,500 INFOCLI nimitz: /opt/pbs/bin/pbs_rstat -f 2017-06-09 01:34:47,509 INFO manager on nimitz: unset server ['job_sort_formula'] 2017-06-09 01:34:47,509 INFOCLI nimitz: sudo -H /opt/pbs/bin/qmgr -c unset server job_sort_formula 2017-06-09 01:34:47,532 INFOCLI nimitz: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-06-09 01:34:47,549 INFOCLI nimitz: /opt/pbs/bin/qstat -Qf @nimitz 2017-06-09 01:34:47,560 INFO manager on nimitz: delete queue workq 2017-06-09 01:34:47,561 INFOCLI nimitz: /opt/pbs/bin/qmgr -c delete queue workq 2017-06-09 01:34:47,574 INFO server nimitz: expect offset set to 0.5 2017-06-09 01:34:48,078 INFOCLI nimitz: /opt/pbs/bin/qstat -Qf workq@nimitz 2017-06-09 01:34:48,091 INFO expect on server nimitz: unset queue workq ... OK 2017-06-09 01:34:48,091 INFO manager on nimitz: create queue workq {'started': 'True', 'queue_type': 'Execution', 'enabled': 'True'} 2017-06-09 01:34:48,091 INFOCLI nimitz: /opt/pbs/bin/qmgr -c create queue workq started=True,queue_type=Execution,enabled=True 2017-06-09 01:34:48,104 INFO server nimitz: expect offset set to 0.5 2017-06-09 01:34:48,648 INFOCLI nimitz: /opt/pbs/bin/qstat -Qf workq@nimitz 2017-06-09 01:34:48,662 INFO expect on server nimitz: started set True || queue_type set Execution || enabled set True queue workq ... OK 2017-06-09 01:34:48,662 INFO manager on nimitz: set server {'default_queue': 'workq'} 2017-06-09 01:34:48,662 INFOCLI nimitz: /opt/pbs/bin/qmgr -c set server default_queue=workq 2017-06-09 01:34:48,684 INFO status on nimitz: resource 2017-06-09 01:34:48,684 INFO manager on nimitz: list resource 2017-06-09 01:34:48,684 INFOCLI nimitz: /opt/pbs/bin/qmgr -c list resource 2017-06-09 01:34:48,694 INFOCLI status on nimitz: server license_count 2017-06-09 01:34:48,695 INFOCLI nimitz: /opt/pbs/bin/qstat -Bf nimitz 2017-06-09 01:34:48,704 INFO server: nimitz licensed 2017-06-09 01:34:48,718 INFOCLI2 nimitz: sudo -H /usr/bin/cat /var/spool/pbs/server_priv/comm.lock 2017-06-09 01:34:48,741 INFOCLI2 nimitz: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-06-09 01:34:48,752 INFO scheduler nimitz: reverting configuration to defaults 2017-06-09 01:34:48,752 INFO manager on nimitz: list sched 2017-06-09 01:34:48,752 INFOCLI nimitz: /opt/pbs/bin/qmgr -c list sched 2017-06-09 01:34:48,764 INFO manager on nimitz: unset sched ['job_sort_formula_threshold', 'sched_cycle_length'] 2017-06-09 01:34:48,764 INFOCLI nimitz: /opt/pbs/bin/qmgr -c unset sched job_sort_formula_threshold,sched_cycle_length 2017-06-09 01:34:48,775 INFOCLI2 nimitz: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/dedicated_time 2017-06-09 01:34:48,785 INFOCLI2 nimitz: cmp /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:48,787 ERROR err: ['cmp: /var/spool/pbs/sched_priv/resource_group: Permission denied'] 2017-06-09 01:34:48,788 INFOCLI2 nimitz: sudo -H /usr/bin/cp -p /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:48,797 INFOCLI2 nimitz: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-06-09 01:34:48,820 INFO scheduler nimitz: reverting holidays file to default 2017-06-09 01:34:48,820 INFOCLI2 nimitz: cmp /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-06-09 01:34:48,823 ERROR err: ['cmp: /var/spool/pbs/sched_priv/holidays: Permission denied'] 2017-06-09 01:34:48,823 INFOCLI2 nimitz: sudo -H /usr/bin/cp -p /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-06-09 01:34:48,834 INFOCLI2 nimitz: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/holidays 2017-06-09 01:34:48,843 INFO scheduler nimitz: Parsing holidays file 2017-06-09 01:34:48,843 INFOCLI2 nimitz: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2017-06-09 01:34:48,854 INFOCLI2 nimitz: cmp /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-06-09 01:34:48,856 ERROR err: ['cmp: /var/spool/pbs/sched_priv/sched_config: Permission denied'] 2017-06-09 01:34:48,857 INFOCLI2 nimitz: sudo -H /usr/bin/cp -p /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-06-09 01:34:48,869 INFOCLI2 nimitz: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-06-09 01:34:48,878 INFO scheduler nimitz: sent signal -HUP 2017-06-09 01:34:48,878 INFOCLI2 nimitz: sudo -H kill -HUP 14109 2017-06-09 01:34:48,889 INFOCLI2 nimitz: sudo -H /opt/pbs/sbin/pbsfs -e 2017-06-09 01:34:48,934 INFOCLI2 nimitz: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-06-09 01:34:48,957 INFOCLI2 nimitz: sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-06-09 01:34:48,967 INFO mom nimitz: reverting configuration to defaults 2017-06-09 01:34:48,967 INFOCLI2 nimitz: sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2017-06-09 01:34:48,977 INFOCLI2 nimitz: sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2017-06-09 01:34:48,986 INFOCLI nimitz: sudo -H /opt/pbs/sbin/pbs_mom -s list 2017-06-09 01:34:49,037 INFO mom nimitz: sent signal -HUP 2017-06-09 01:34:49,037 INFOCLI2 nimitz: sudo -H kill -HUP 9035 2017-06-09 01:34:49,062 INFOCLI2 nimitz: sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-06-09 01:34:49,073 INFOCLI nimitz: /opt/pbs/bin/pbsnodes -s nimitz -v -a 2017-06-09 01:34:49,084 INFO status on nimitz: node nimitz 2017-06-09 01:34:49,084 INFOCLI nimitz: /opt/pbs/bin/pbsnodes -s nimitz -v nimitz 2017-06-09 01:34:49,094 INFOCLI nimitz: /opt/pbs/bin/pbsnodes -s nimitz -v nimitz 2017-06-09 01:34:49,104 INFO expect on server nimitz: state = free node nimitz ... OK 2017-06-09 01:34:49,104 INFO ============================= 2017-06-09 01:34:49,104 INFO Completed TestFairshare setUp 2017-06-09 01:34:49,104 INFO ============================= 2017-06-09 01:34:49,104 INFO manager on nimitz: set server {'job_sort_formula': 'fairshare_tree_usage'} 2017-06-09 01:34:49,104 INFOCLI nimitz: sudo -H /opt/pbs/bin/qmgr -c set server job_sort_formula=fairshare_tree_usage 2017-06-09 01:34:49,125 INFO manager on nimitz: set server {'job_sort_formula': 'fairshare_perc'} 2017-06-09 01:34:49,126 INFOCLI nimitz: sudo -H /opt/pbs/bin/qmgr -c set server job_sort_formula=fairshare_perc 2017-06-09 01:34:49,152 INFO manager on nimitz: set server {'job_sort_formula': '"pow(2,-(fairshare_tree_usage/fairshare_perc))"'} 2017-06-09 01:34:49,153 INFOCLI nimitz: sudo -H /opt/pbs/bin/qmgr -c set server job_sort_formula="pow(2,-(fairshare_tree_usage/fairshare_perc))" 2017-06-09 01:34:49,178 INFO =============================== 2017-06-09 01:34:49,178 INFO Entered TestFairshare tearDown 2017-06-09 01:34:49,178 INFO =============================== 2017-06-09 01:34:49,178 INFO ================================ 2017-06-09 01:34:49,178 INFO Completed TestFairshare tearDown 2017-06-09 01:34:49,178 INFO ================================ 2017-06-09 01:34:49,179 INFO ok 2017-06-09 01:34:49,179 INFO ================================================================================ run: 4, succeeded: 4, failed: 0, errors: 0, skipped: 0, timedout: 0 Tests run in 0:00:15.344586 2017-06-09 01:34:49,179 INFO Cleaning up temporary files