2017-08-10 09:52:19,995 INFO input command: pbs_benchpress -t TestFairshare -o /home/anamika/fairshare_17.2_try2.out 2017-08-10 09:52:19,995 INFO param: None 2017-08-10 09:52:19,995 INFO ptl version: 17.2.0.20170728010834 2017-08-10 09:52:19,996 INFO platform: Linux testdev-02-s11p2 3.0.13-0.27-xen #1 SMP Wed Feb 15 13:33:49 UTC 2012 (d73692b) x86_64 x86_64 2017-08-10 09:52:19,996 INFO python version: 2.7.11 2017-08-10 09:52:19,996 INFO user: anamika 2017-08-10 09:52:19,996 INFO -------------------------------------------------------------------------------- 2017-08-10 09:52:19,996 INFO Cleaning up temporary files 2017-08-10 09:52:26,596 INFO ====================================================================== 2017-08-10 09:52:26,596 INFO suite name: TestFairshare 2017-08-10 09:52:26,596 INFO suite docstring: Test the pbs_sched fairshare functionality. Note there are two fairshare tests in the standard smoke test that are not included here. 2017-08-10 09:52:26,597 INFO ====================================================================== 2017-08-10 09:52:26,597 INFO ================================= 2017-08-10 09:52:26,597 INFO Entered TestFairshare setUpClass 2017-08-10 09:52:26,597 INFO ================================= 2017-08-10 09:52:26,597 INFOCLI2 testdev-02-s11p2: id pbsuser 2017-08-10 09:52:26,605 INFOCLI2 testdev-02-s11p2: id pbsuser1 2017-08-10 09:52:26,611 INFOCLI2 testdev-02-s11p2: id pbsuser2 2017-08-10 09:52:26,616 INFOCLI2 testdev-02-s11p2: id pbsuser3 2017-08-10 09:52:26,622 INFO FQDN name testdev-02-s11p2.pbspro.com differs from name provided testdev-02-s11p2 2017-08-10 09:52:26,639 INFO server testdev-02-s11p2: server operating mode set to cli 2017-08-10 09:52:26,639 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -Bf testdev-02-s11p2.pbspro.com 2017-08-10 09:52:26,654 INFO server testdev-02-s11p2: version 17.2.0.20170728010834 2017-08-10 09:52:26,655 INFO expect action: created new action kicksched 2017-08-10 09:52:26,659 INFO expect action: added action kicksched to server testdev-02-s11p2 2017-08-10 09:52:26,660 INFO FQDN name testdev-02-s11p2.pbspro.com differs from name provided testdev-02-s11p2 2017-08-10 09:52:26,671 INFO FQDN name testdev-02-s11p2.pbspro.com differs from name provided testdev-02-s11p2 2017-08-10 09:52:26,698 INFOCLI2 testdev-02-s11p2: sudo -H /opt/pbs/sbin/pbsfs 2017-08-10 09:52:26,715 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:26,726 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/sched_priv/holidays 2017-08-10 09:52:26,738 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c list sched default 2017-08-10 09:52:26,774 INFOCLI2 testdev-02-s11p2: sudo -H /opt/pbs/sbin/pbsfs 2017-08-10 09:52:26,790 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:26,802 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/sched_priv/holidays 2017-08-10 09:52:26,814 INFO FQDN name testdev-02-s11p2.pbspro.com differs from name provided testdev-02-s11p2 2017-08-10 09:52:26,824 INFO ================================== 2017-08-10 09:52:26,824 INFO Completed TestFairshare setUpClass 2017-08-10 09:52:26,824 INFO ================================== 2017-08-10 09:52:26,825 INFO test name: test_fairshare_formula (tests.functional.pbs_fairshare.TestFairshare)... 2017-08-10 09:52:26,825 INFO test start time: Thu Aug 10 09:52:26 2017 2017-08-10 09:52:26,825 INFO test docstring: Test fairshare in the formula. Make sure the fairshare_tree_usage is correct 2017-08-10 09:52:26,825 INFO ============================ 2017-08-10 09:52:26,825 INFO Entered TestFairshare setUp 2017-08-10 09:52:26,825 INFO ============================ 2017-08-10 09:52:26,826 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -Bf testdev-02-s11p2.pbspro.com 2017-08-10 09:52:26,839 INFO status on testdev-02-s11p2: server 2017-08-10 09:52:26,839 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -Bf testdev-02-s11p2.pbspro.com 2017-08-10 09:52:26,853 INFO manager on testdev-02-s11p2: set server {'managers': (3, 'anamika@*')} 2017-08-10 09:52:26,853 INFOCLI testdev-02-s11p2: sudo -H /opt/pbs/bin/qmgr -c set server managers-=anamika@* 2017-08-10 09:52:27,302 INFO manager on testdev-02-s11p2: set server {'managers': (2, 'anamika@*')} 2017-08-10 09:52:27,302 INFOCLI testdev-02-s11p2: sudo -H /opt/pbs/bin/qmgr -c set server managers+=anamika@* 2017-08-10 09:52:27,391 INFO server testdev-02-s11p2: reverting configuration to defaults 2017-08-10 09:52:27,392 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -Bf testdev-02-s11p2.pbspro.com 2017-08-10 09:52:27,406 INFO select on testdev-02-s11p2: __ALL__ 2017-08-10 09:52:27,407 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qselect 2017-08-10 09:52:27,420 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -f @testdev-02-s11p2.pbspro.com 2017-08-10 09:52:27,433 INFO expect on server testdev-02-s11p2: job_state set 0 job ... OK 2017-08-10 09:52:27,434 INFOCLI testdev-02-s11p2: /opt/pbs/bin/pbs_rstat -f 2017-08-10 09:52:27,447 INFOCLI testdev-02-s11p2: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-08-10 09:52:27,467 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -Qf @testdev-02-s11p2.pbspro.com 2017-08-10 09:52:27,482 INFO status on testdev-02-s11p2: node 2017-08-10 09:52:27,482 INFOCLI testdev-02-s11p2: /opt/pbs/bin/pbsnodes -s testdev-02-s11p2.pbspro.com -v -a 2017-08-10 09:52:27,506 INFO manager on testdev-02-s11p2: delete queue workq 2017-08-10 09:52:27,507 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c delete queue workq 2017-08-10 09:52:27,661 INFO server testdev-02-s11p2: expect offset set to 0.5 2017-08-10 09:52:28,162 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -Qf workq@testdev-02-s11p2.pbspro.com 2017-08-10 09:52:28,177 INFO expect on server testdev-02-s11p2: unset queue workq ... OK 2017-08-10 09:52:28,178 INFO manager on testdev-02-s11p2: create queue workq {'started': 'True', 'queue_type': 'Execution', 'enabled': 'True'} 2017-08-10 09:52:28,178 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c create queue workq started=True,queue_type=Execution,enabled=True 2017-08-10 09:52:28,381 INFO server testdev-02-s11p2: expect offset set to 0.5 2017-08-10 09:52:28,883 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -Qf workq@testdev-02-s11p2.pbspro.com 2017-08-10 09:52:28,898 INFO expect on server testdev-02-s11p2: started set True || queue_type set Execution || enabled set True queue workq ... OK 2017-08-10 09:52:28,899 INFO manager on testdev-02-s11p2: set server {'default_queue': 'workq'} 2017-08-10 09:52:28,899 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c set server default_queue=workq 2017-08-10 09:52:28,971 INFO status on testdev-02-s11p2: resource 2017-08-10 09:52:28,971 INFO manager on testdev-02-s11p2: list resource 2017-08-10 09:52:28,971 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c list resource 2017-08-10 09:52:28,986 INFOCLI status on testdev-02-s11p2: server license_count 2017-08-10 09:52:28,986 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -Bf testdev-02-s11p2.pbspro.com 2017-08-10 09:52:29,000 INFO server: testdev-02-s11p2.pbspro.com licensed 2017-08-10 09:52:29,011 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/server_priv/comm.lock 2017-08-10 09:52:29,036 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/sched_priv/sched.lock 2017-08-10 09:52:29,049 INFO scheduler testdev-02-s11p2: reverting configuration to defaults 2017-08-10 09:52:29,049 INFO manager on testdev-02-s11p2: delete sched @default 2017-08-10 09:52:29,049 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c delete sched @default 2017-08-10 09:52:29,064 INFO manager on testdev-02-s11p2: list sched 2017-08-10 09:52:29,065 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c list sched 2017-08-10 09:52:29,080 INFO manager on testdev-02-s11p2: unset sched ['sched_cycle_length'] 2017-08-10 09:52:29,080 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c unset sched sched_cycle_length 2017-08-10 09:52:29,131 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/sched_priv/dedicated_time 2017-08-10 09:52:29,145 INFOCLI2 testdev-02-s11p2: cmp /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:29,150 ERROR err: ['cmp: /var/spool/pbs/sched_priv/resource_group: Permission denied'] 2017-08-10 09:52:29,150 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:29,163 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:29,176 INFO scheduler testdev-02-s11p2: reverting holidays file to default 2017-08-10 09:52:29,177 INFOCLI2 testdev-02-s11p2: cmp /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-08-10 09:52:29,183 ERROR err: ['cmp: /var/spool/pbs/sched_priv/holidays: Permission denied'] 2017-08-10 09:52:29,183 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-08-10 09:52:29,196 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/holidays 2017-08-10 09:52:29,209 INFO scheduler testdev-02-s11p2: Parsing holidays file 2017-08-10 09:52:29,210 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/sched_priv/holidays 2017-08-10 09:52:29,223 INFOCLI2 testdev-02-s11p2: cmp /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-08-10 09:52:29,229 ERROR err: ['cmp: /var/spool/pbs/sched_priv/sched_config: Permission denied'] 2017-08-10 09:52:29,230 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-08-10 09:52:29,242 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-08-10 09:52:29,254 INFO scheduler testdev-02-s11p2: sent signal -HUP 2017-08-10 09:52:29,255 INFOCLI2 testdev-02-s11p2: sudo -H kill -HUP 12179 2017-08-10 09:52:29,266 INFOCLI2 testdev-02-s11p2: sudo -H /opt/pbs/sbin/pbsfs -e 2017-08-10 09:52:29,308 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/sched_priv/sched.lock 2017-08-10 09:52:29,332 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/mom_priv/mom.lock 2017-08-10 09:52:29,345 INFOCLI2 testdev-02-s11p2: sudo -H /opt/pbs/sbin/pbs_mom --version 2017-08-10 09:52:29,358 INFO mom testdev-02-s11p2: reverting configuration to defaults 2017-08-10 09:52:29,358 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/rm -f /var/spool/pbs/mom_priv/epilogue 2017-08-10 09:52:29,372 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/rm -f /var/spool/pbs/mom_priv/prologue 2017-08-10 09:52:29,385 INFOCLI testdev-02-s11p2: sudo -H /opt/pbs/sbin/pbs_mom -s list 2017-08-10 09:52:29,406 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /tmp/PtlPbsEg7p1g /var/spool/pbs/mom_priv/config 2017-08-10 09:52:29,418 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/mom_priv/config 2017-08-10 09:52:29,430 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chown root /var/spool/pbs/mom_priv/config 2017-08-10 09:52:29,442 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chgrp root /var/spool/pbs/mom_priv/config 2017-08-10 09:52:29,455 INFO mom testdev-02-s11p2: sent signal -HUP 2017-08-10 09:52:29,456 INFOCLI2 testdev-02-s11p2: sudo -H kill -HUP 12176 2017-08-10 09:52:29,480 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/mom_priv/mom.lock 2017-08-10 09:52:29,493 INFOCLI testdev-02-s11p2: /opt/pbs/bin/pbsnodes -s testdev-02-s11p2.pbspro.com -v -a 2017-08-10 09:52:29,508 INFO status on testdev-02-s11p2: node testdev-02-s11p2 2017-08-10 09:52:29,509 INFOCLI testdev-02-s11p2: /opt/pbs/bin/pbsnodes -s testdev-02-s11p2.pbspro.com -v testdev-02-s11p2 2017-08-10 09:52:29,523 INFOCLI testdev-02-s11p2: /opt/pbs/bin/pbsnodes -s testdev-02-s11p2.pbspro.com -v testdev-02-s11p2 2017-08-10 09:52:29,536 INFO expect on server testdev-02-s11p2: state = free node testdev-02-s11p2 ... OK 2017-08-10 09:52:29,537 INFO ============================= 2017-08-10 09:52:29,537 INFO Completed TestFairshare setUp 2017-08-10 09:52:29,537 INFO ============================= 2017-08-10 09:52:29,537 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:29,549 INFO creating tree node: group1 2017-08-10 09:52:29,549 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /tmp/PtlPbsK16qie /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:29,562 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:29,573 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chown root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:29,585 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chgrp root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:29,597 INFO creating tree node: group2 2017-08-10 09:52:29,598 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /tmp/PtlPbsX5Vj5t /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:29,610 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:29,621 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chown root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:29,633 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chgrp root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:29,644 INFO creating tree node: pbsuser 2017-08-10 09:52:29,645 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /tmp/PtlPbskT0Mo8 /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:29,657 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:29,668 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chown root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:29,680 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chgrp root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:29,691 INFO creating tree node: pbsuser1 2017-08-10 09:52:29,692 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /tmp/PtlPbsCxT7xk /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:29,704 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:29,716 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chown root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:29,727 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chgrp root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:29,739 INFO creating tree node: pbsuser2 2017-08-10 09:52:29,740 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /tmp/PtlPbsnHbjOj /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:29,753 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:29,765 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chown root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:29,777 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chgrp root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:29,788 INFO creating tree node: pbsuser3 2017-08-10 09:52:29,789 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /tmp/PtlPbsxDybSe /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:29,801 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:29,812 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chown root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:29,824 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chgrp root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:29,836 INFOCLI2 testdev-02-s11p2: sudo -H /opt/pbs/sbin/pbsfs -s pbsuser 100 2017-08-10 09:52:29,852 INFOCLI2 testdev-02-s11p2: sudo -H /opt/pbs/sbin/pbsfs -s pbsuser1 100 2017-08-10 09:52:29,868 INFOCLI2 testdev-02-s11p2: sudo -H /opt/pbs/sbin/pbsfs -s pbsuser3 1000 2017-08-10 09:52:29,885 INFO scheduler testdev-02-s11p2: config {'log_filter': 2048} 2017-08-10 09:52:29,886 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /var/spool/pbs/sched_priv/sched_config /var/spool/pbs/sched_priv/sched_config.bak 2017-08-10 09:52:29,898 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /tmp/PtlPbsl2fX1s /var/spool/pbs/sched_priv/sched_config 2017-08-10 09:52:29,910 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-08-10 09:52:29,922 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chown root /var/spool/pbs/sched_priv/sched_config 2017-08-10 09:52:29,934 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chgrp root /var/spool/pbs/sched_priv/sched_config 2017-08-10 09:52:29,945 INFO scheduler testdev-02-s11p2: sent signal -HUP 2017-08-10 09:52:29,946 INFOCLI2 testdev-02-s11p2: sudo -H kill -HUP 12179 2017-08-10 09:52:29,958 INFO scheduler testdev-02-s11p2 log match: searching for "Error reading line" - No match 2017-08-10 09:52:30,960 INFO manager on testdev-02-s11p2: set server {'scheduling': 'False'} 2017-08-10 09:52:30,960 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c set server scheduling=False 2017-08-10 09:52:31,141 INFO manager on testdev-02-s11p2: set server {'job_sort_formula': 'fairshare_tree_usage'} 2017-08-10 09:52:31,142 INFOCLI testdev-02-s11p2: sudo -H /opt/pbs/bin/qmgr -c set server job_sort_formula=fairshare_tree_usage 2017-08-10 09:52:31,262 INFO job: executable set to /bin/sleep with arguments: 100 2017-08-10 09:52:31,263 INFOCLI testdev-02-s11p2: sudo -H -u pbsuser /opt/pbs/bin/qsub -- /bin/sleep 100 2017-08-10 09:52:31,332 INFO submit to testdev-02-s11p2 as pbsuser: job 185.testdev-02-s11p2 2017-08-10 09:52:31,332 INFO job: executable set to /bin/sleep with arguments: 100 2017-08-10 09:52:31,333 INFOCLI testdev-02-s11p2: sudo -H -u pbsuser1 /opt/pbs/bin/qsub -- /bin/sleep 100 2017-08-10 09:52:31,402 INFO submit to testdev-02-s11p2 as pbsuser1: job 186.testdev-02-s11p2 2017-08-10 09:52:31,402 INFO job: executable set to /bin/sleep with arguments: 100 2017-08-10 09:52:31,403 INFOCLI testdev-02-s11p2: sudo -H -u pbsuser2 /opt/pbs/bin/qsub -- /bin/sleep 100 2017-08-10 09:52:31,462 INFO submit to testdev-02-s11p2 as pbsuser2: job 187.testdev-02-s11p2 2017-08-10 09:52:31,462 INFO job: executable set to /bin/sleep with arguments: 100 2017-08-10 09:52:31,463 INFOCLI testdev-02-s11p2: sudo -H -u pbsuser3 /opt/pbs/bin/qsub -- /bin/sleep 100 2017-08-10 09:52:31,522 INFO submit to testdev-02-s11p2 as pbsuser3: job 188.testdev-02-s11p2 2017-08-10 09:52:31,522 INFO manager on testdev-02-s11p2: set server {'scheduling': 'True'} 2017-08-10 09:52:31,522 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-08-10 09:52:31,582 INFO scheduler testdev-02-s11p2 log match: searching for "185.testdev-02-s11p2;Formula Evaluation = 0.1253" - No match 2017-08-10 09:52:32,583 INFO scheduler testdev-02-s11p2 log match: searching for "185.testdev-02-s11p2;Formula Evaluation = 0.1253"... OK 2017-08-10 09:52:32,584 INFO scheduler testdev-02-s11p2 log match: searching for "186.testdev-02-s11p2;Formula Evaluation = 0.1253"... OK 2017-08-10 09:52:32,584 INFO scheduler testdev-02-s11p2 log match: searching for "187.testdev-02-s11p2;Formula Evaluation = 0.5004"... OK 2017-08-10 09:52:32,584 INFO scheduler testdev-02-s11p2 log match: searching for "188.testdev-02-s11p2;Formula Evaluation = 0.8330"... OK 2017-08-10 09:52:32,585 INFO =============================== 2017-08-10 09:52:32,585 INFO Entered TestFairshare tearDown 2017-08-10 09:52:32,585 INFO =============================== 2017-08-10 09:52:32,585 INFO ================================ 2017-08-10 09:52:32,585 INFO Completed TestFairshare tearDown 2017-08-10 09:52:32,585 INFO ================================ 2017-08-10 09:52:32,589 INFO ok 2017-08-10 09:52:32,590 INFO test name: test_fairshare_formula2 (tests.functional.pbs_fairshare.TestFairshare)... 2017-08-10 09:52:32,590 INFO test start time: Thu Aug 10 09:52:32 2017 2017-08-10 09:52:32,590 INFO test docstring: Test fairshare in the formula. Make sure the fairshare_perc is correct 2017-08-10 09:52:32,590 INFO ============================ 2017-08-10 09:52:32,590 INFO Entered TestFairshare setUp 2017-08-10 09:52:32,590 INFO ============================ 2017-08-10 09:52:32,591 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -Bf testdev-02-s11p2.pbspro.com 2017-08-10 09:52:32,607 INFO status on testdev-02-s11p2: server 2017-08-10 09:52:32,608 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -Bf testdev-02-s11p2.pbspro.com 2017-08-10 09:52:32,621 INFO manager on testdev-02-s11p2: set server {'managers': (3, 'anamika@*')} 2017-08-10 09:52:32,622 INFOCLI testdev-02-s11p2: sudo -H /opt/pbs/bin/qmgr -c set server managers-=anamika@* 2017-08-10 09:52:32,742 INFO manager on testdev-02-s11p2: set server {'managers': (2, 'anamika@*')} 2017-08-10 09:52:32,742 INFOCLI testdev-02-s11p2: sudo -H /opt/pbs/bin/qmgr -c set server managers+=anamika@* 2017-08-10 09:52:32,802 INFO server testdev-02-s11p2: reverting configuration to defaults 2017-08-10 09:52:32,802 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -Bf testdev-02-s11p2.pbspro.com 2017-08-10 09:52:32,816 INFO select on testdev-02-s11p2: __ALL__ 2017-08-10 09:52:32,817 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qselect 2017-08-10 09:52:32,830 INFO delete job on testdev-02-s11p2: 185.testdev-02-s11p2, 186.testdev-02-s11p2, 187.testdev-02-s11p2, 188.testdev-02-s11p2 2017-08-10 09:52:32,830 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qdel -W force 185.testdev-02-s11p2 186.testdev-02-s11p2 187.testdev-02-s11p2 188.testdev-02-s11p2 2017-08-10 09:52:33,462 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -f 185.testdev-02-s11p2 2017-08-10 09:52:33,482 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -f 186.testdev-02-s11p2 2017-08-10 09:52:33,500 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -f 187.testdev-02-s11p2 2017-08-10 09:52:33,518 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -f 188.testdev-02-s11p2 2017-08-10 09:52:33,537 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -f @testdev-02-s11p2.pbspro.com 2017-08-10 09:52:33,551 INFO expect on server testdev-02-s11p2: job_state set 0 job ... OK 2017-08-10 09:52:33,551 INFOCLI testdev-02-s11p2: /opt/pbs/bin/pbs_rstat -f 2017-08-10 09:52:33,564 INFO manager on testdev-02-s11p2: unset server ['job_sort_formula'] 2017-08-10 09:52:33,565 INFOCLI testdev-02-s11p2: sudo -H /opt/pbs/bin/qmgr -c unset server job_sort_formula 2017-08-10 09:52:33,702 INFOCLI testdev-02-s11p2: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-08-10 09:52:33,724 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -Qf @testdev-02-s11p2.pbspro.com 2017-08-10 09:52:33,738 INFO status on testdev-02-s11p2: node 2017-08-10 09:52:33,739 INFOCLI testdev-02-s11p2: /opt/pbs/bin/pbsnodes -s testdev-02-s11p2.pbspro.com -v -a 2017-08-10 09:52:33,752 INFO manager on testdev-02-s11p2: delete queue workq 2017-08-10 09:52:33,753 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c delete queue workq 2017-08-10 09:52:33,951 INFO server testdev-02-s11p2: expect offset set to 0.5 2017-08-10 09:52:34,452 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -Qf workq@testdev-02-s11p2.pbspro.com 2017-08-10 09:52:34,468 INFO expect on server testdev-02-s11p2: unset queue workq ... OK 2017-08-10 09:52:34,468 INFO manager on testdev-02-s11p2: create queue workq {'started': 'True', 'queue_type': 'Execution', 'enabled': 'True'} 2017-08-10 09:52:34,468 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c create queue workq started=True,queue_type=Execution,enabled=True 2017-08-10 09:52:34,681 INFO server testdev-02-s11p2: expect offset set to 0.5 2017-08-10 09:52:35,183 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -Qf workq@testdev-02-s11p2.pbspro.com 2017-08-10 09:52:35,198 INFO expect on server testdev-02-s11p2: started set True || queue_type set Execution || enabled set True queue workq ... OK 2017-08-10 09:52:35,199 INFO manager on testdev-02-s11p2: set server {'default_queue': 'workq'} 2017-08-10 09:52:35,199 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c set server default_queue=workq 2017-08-10 09:52:35,311 INFO status on testdev-02-s11p2: resource 2017-08-10 09:52:35,312 INFO manager on testdev-02-s11p2: list resource 2017-08-10 09:52:35,312 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c list resource 2017-08-10 09:52:35,327 INFOCLI status on testdev-02-s11p2: server license_count 2017-08-10 09:52:35,327 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -Bf testdev-02-s11p2.pbspro.com 2017-08-10 09:52:35,341 INFO server: testdev-02-s11p2.pbspro.com licensed 2017-08-10 09:52:35,352 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/server_priv/comm.lock 2017-08-10 09:52:35,375 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/sched_priv/sched.lock 2017-08-10 09:52:35,387 INFO scheduler testdev-02-s11p2: reverting configuration to defaults 2017-08-10 09:52:35,388 INFO manager on testdev-02-s11p2: delete sched @default 2017-08-10 09:52:35,388 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c delete sched @default 2017-08-10 09:52:35,401 INFO manager on testdev-02-s11p2: list sched 2017-08-10 09:52:35,402 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c list sched 2017-08-10 09:52:35,415 INFO manager on testdev-02-s11p2: unset sched ['sched_cycle_length'] 2017-08-10 09:52:35,416 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c unset sched sched_cycle_length 2017-08-10 09:52:35,461 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/sched_priv/dedicated_time 2017-08-10 09:52:35,474 INFOCLI2 testdev-02-s11p2: cmp /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:35,479 ERROR err: ['cmp: /var/spool/pbs/sched_priv/resource_group: Permission denied'] 2017-08-10 09:52:35,479 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:35,491 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:35,503 INFO scheduler testdev-02-s11p2: reverting holidays file to default 2017-08-10 09:52:35,503 INFOCLI2 testdev-02-s11p2: cmp /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-08-10 09:52:35,508 ERROR err: ['cmp: /var/spool/pbs/sched_priv/holidays: Permission denied'] 2017-08-10 09:52:35,508 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-08-10 09:52:35,520 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/holidays 2017-08-10 09:52:35,532 INFO scheduler testdev-02-s11p2: Parsing holidays file 2017-08-10 09:52:35,532 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/sched_priv/holidays 2017-08-10 09:52:35,544 INFOCLI2 testdev-02-s11p2: cmp /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-08-10 09:52:35,548 ERROR err: ['cmp: /var/spool/pbs/sched_priv/sched_config: Permission denied'] 2017-08-10 09:52:35,549 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-08-10 09:52:35,561 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-08-10 09:52:35,572 INFO scheduler testdev-02-s11p2: sent signal -HUP 2017-08-10 09:52:35,573 INFOCLI2 testdev-02-s11p2: sudo -H kill -HUP 12179 2017-08-10 09:52:35,584 INFOCLI2 testdev-02-s11p2: sudo -H /opt/pbs/sbin/pbsfs -e 2017-08-10 09:52:35,624 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/sched_priv/sched.lock 2017-08-10 09:52:35,645 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/mom_priv/mom.lock 2017-08-10 09:52:35,657 INFO mom testdev-02-s11p2: reverting configuration to defaults 2017-08-10 09:52:35,657 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/rm -f /var/spool/pbs/mom_priv/epilogue 2017-08-10 09:52:35,669 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/rm -f /var/spool/pbs/mom_priv/prologue 2017-08-10 09:52:35,681 INFOCLI testdev-02-s11p2: sudo -H /opt/pbs/sbin/pbs_mom -s list 2017-08-10 09:52:35,699 INFO mom testdev-02-s11p2: sent signal -HUP 2017-08-10 09:52:35,700 INFOCLI2 testdev-02-s11p2: sudo -H kill -HUP 12176 2017-08-10 09:52:35,723 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/mom_priv/mom.lock 2017-08-10 09:52:35,734 INFOCLI testdev-02-s11p2: /opt/pbs/bin/pbsnodes -s testdev-02-s11p2.pbspro.com -v -a 2017-08-10 09:52:35,748 INFO status on testdev-02-s11p2: node testdev-02-s11p2 2017-08-10 09:52:35,749 INFOCLI testdev-02-s11p2: /opt/pbs/bin/pbsnodes -s testdev-02-s11p2.pbspro.com -v testdev-02-s11p2 2017-08-10 09:52:35,763 INFOCLI testdev-02-s11p2: /opt/pbs/bin/pbsnodes -s testdev-02-s11p2.pbspro.com -v testdev-02-s11p2 2017-08-10 09:52:35,777 INFO expect on server testdev-02-s11p2: state = free node testdev-02-s11p2 ... OK 2017-08-10 09:52:35,777 INFO ============================= 2017-08-10 09:52:35,777 INFO Completed TestFairshare setUp 2017-08-10 09:52:35,777 INFO ============================= 2017-08-10 09:52:35,777 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:35,789 INFO creating tree node: group1 2017-08-10 09:52:35,790 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /tmp/PtlPbstUtEWY /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:35,802 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:35,814 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chown root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:35,825 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chgrp root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:35,837 INFO creating tree node: group2 2017-08-10 09:52:35,837 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /tmp/PtlPbsZkP95y /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:35,849 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:35,861 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chown root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:35,872 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chgrp root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:35,884 INFO creating tree node: pbsuser 2017-08-10 09:52:35,885 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /tmp/PtlPbsJ5fjIR /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:35,898 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:35,909 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chown root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:35,921 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chgrp root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:35,933 INFO creating tree node: pbsuser1 2017-08-10 09:52:35,934 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /tmp/PtlPbsV4QvGV /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:35,946 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:35,958 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chown root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:35,969 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chgrp root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:35,981 INFO creating tree node: pbsuser2 2017-08-10 09:52:35,982 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /tmp/PtlPbs5VpzTD /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:35,993 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:36,005 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chown root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:36,017 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chgrp root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:36,028 INFO creating tree node: pbsuser3 2017-08-10 09:52:36,029 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /tmp/PtlPbsSp3Ti1 /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:36,041 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:36,053 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chown root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:36,064 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chgrp root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:36,076 INFOCLI2 testdev-02-s11p2: sudo -H /opt/pbs/sbin/pbsfs -s pbsuser 100 2017-08-10 09:52:36,092 INFOCLI2 testdev-02-s11p2: sudo -H /opt/pbs/sbin/pbsfs -s pbsuser1 100 2017-08-10 09:52:36,108 INFOCLI2 testdev-02-s11p2: sudo -H /opt/pbs/sbin/pbsfs -s pbsuser3 1000 2017-08-10 09:52:36,125 INFO scheduler testdev-02-s11p2: config {'log_filter': 2048} 2017-08-10 09:52:36,126 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /var/spool/pbs/sched_priv/sched_config /var/spool/pbs/sched_priv/sched_config.bak 2017-08-10 09:52:36,138 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /tmp/PtlPbsbNPY_r /var/spool/pbs/sched_priv/sched_config 2017-08-10 09:52:36,150 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-08-10 09:52:36,162 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chown root /var/spool/pbs/sched_priv/sched_config 2017-08-10 09:52:36,173 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chgrp root /var/spool/pbs/sched_priv/sched_config 2017-08-10 09:52:36,185 INFO scheduler testdev-02-s11p2: sent signal -HUP 2017-08-10 09:52:36,185 INFOCLI2 testdev-02-s11p2: sudo -H kill -HUP 12179 2017-08-10 09:52:36,197 INFO scheduler testdev-02-s11p2 log match: searching for "Error reading line" - No match 2017-08-10 09:52:37,198 INFO manager on testdev-02-s11p2: set server {'scheduling': 'False'} 2017-08-10 09:52:37,199 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c set server scheduling=False 2017-08-10 09:52:37,592 INFO manager on testdev-02-s11p2: set server {'job_sort_formula': 'fairshare_perc'} 2017-08-10 09:52:37,593 INFOCLI testdev-02-s11p2: sudo -H /opt/pbs/bin/qmgr -c set server job_sort_formula=fairshare_perc 2017-08-10 09:52:37,753 INFO job: executable set to /bin/sleep with arguments: 100 2017-08-10 09:52:37,753 INFOCLI testdev-02-s11p2: sudo -H -u pbsuser /opt/pbs/bin/qsub -- /bin/sleep 100 2017-08-10 09:52:37,812 INFO submit to testdev-02-s11p2 as pbsuser: job 189.testdev-02-s11p2 2017-08-10 09:52:37,813 INFO job: executable set to /bin/sleep with arguments: 100 2017-08-10 09:52:37,813 INFOCLI testdev-02-s11p2: sudo -H -u pbsuser1 /opt/pbs/bin/qsub -- /bin/sleep 100 2017-08-10 09:52:37,862 INFO submit to testdev-02-s11p2 as pbsuser1: job 190.testdev-02-s11p2 2017-08-10 09:52:37,863 INFO job: executable set to /bin/sleep with arguments: 100 2017-08-10 09:52:37,863 INFOCLI testdev-02-s11p2: sudo -H -u pbsuser2 /opt/pbs/bin/qsub -- /bin/sleep 100 2017-08-10 09:52:37,932 INFO submit to testdev-02-s11p2 as pbsuser2: job 191.testdev-02-s11p2 2017-08-10 09:52:37,933 INFO job: executable set to /bin/sleep with arguments: 100 2017-08-10 09:52:37,933 INFOCLI testdev-02-s11p2: sudo -H -u pbsuser3 /opt/pbs/bin/qsub -- /bin/sleep 100 2017-08-10 09:52:38,012 INFO submit to testdev-02-s11p2 as pbsuser3: job 192.testdev-02-s11p2 2017-08-10 09:52:38,012 INFO manager on testdev-02-s11p2: set server {'scheduling': 'True'} 2017-08-10 09:52:38,012 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-08-10 09:52:38,112 INFO scheduler testdev-02-s11p2 log match: searching for "189.testdev-02-s11p2;Formula Evaluation = 0.2" - No match 2017-08-10 09:52:39,113 INFO scheduler testdev-02-s11p2 log match: searching for "189.testdev-02-s11p2;Formula Evaluation = 0.2"... OK 2017-08-10 09:52:39,114 INFO scheduler testdev-02-s11p2 log match: searching for "190.testdev-02-s11p2;Formula Evaluation = 0.2"... OK 2017-08-10 09:52:39,114 INFO scheduler testdev-02-s11p2 log match: searching for "191.testdev-02-s11p2;Formula Evaluation = 0.36"... OK 2017-08-10 09:52:39,114 INFO scheduler testdev-02-s11p2 log match: searching for "192.testdev-02-s11p2;Formula Evaluation = 0.24"... OK 2017-08-10 09:52:39,114 INFO =============================== 2017-08-10 09:52:39,115 INFO Entered TestFairshare tearDown 2017-08-10 09:52:39,115 INFO =============================== 2017-08-10 09:52:39,115 INFO ================================ 2017-08-10 09:52:39,115 INFO Completed TestFairshare tearDown 2017-08-10 09:52:39,115 INFO ================================ 2017-08-10 09:52:39,116 INFO ok 2017-08-10 09:52:39,117 INFO test name: test_fairshare_formula3 (tests.functional.pbs_fairshare.TestFairshare)... 2017-08-10 09:52:39,117 INFO test start time: Thu Aug 10 09:52:39 2017 2017-08-10 09:52:39,117 INFO test docstring: Test fairshare in the formula. Make sure entities with small usage are negatively affected by their high usage siblings. Make sure that jobs run in the correct order. Use fairshare_tree_usage in a pow() formula 2017-08-10 09:52:39,117 INFO ============================ 2017-08-10 09:52:39,117 INFO Entered TestFairshare setUp 2017-08-10 09:52:39,118 INFO ============================ 2017-08-10 09:52:39,118 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -Bf testdev-02-s11p2.pbspro.com 2017-08-10 09:52:39,134 INFO status on testdev-02-s11p2: server 2017-08-10 09:52:39,135 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -Bf testdev-02-s11p2.pbspro.com 2017-08-10 09:52:39,149 INFO manager on testdev-02-s11p2: set server {'managers': (3, 'anamika@*')} 2017-08-10 09:52:39,149 INFOCLI testdev-02-s11p2: sudo -H /opt/pbs/bin/qmgr -c set server managers-=anamika@* 2017-08-10 09:52:39,322 INFO manager on testdev-02-s11p2: set server {'managers': (2, 'anamika@*')} 2017-08-10 09:52:39,323 INFOCLI testdev-02-s11p2: sudo -H /opt/pbs/bin/qmgr -c set server managers+=anamika@* 2017-08-10 09:52:39,424 INFO server testdev-02-s11p2: reverting configuration to defaults 2017-08-10 09:52:39,424 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -Bf testdev-02-s11p2.pbspro.com 2017-08-10 09:52:39,439 INFO select on testdev-02-s11p2: __ALL__ 2017-08-10 09:52:39,440 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qselect 2017-08-10 09:52:39,453 INFO delete job on testdev-02-s11p2: 189.testdev-02-s11p2, 190.testdev-02-s11p2, 191.testdev-02-s11p2, 192.testdev-02-s11p2 2017-08-10 09:52:39,453 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qdel -W force 189.testdev-02-s11p2 190.testdev-02-s11p2 191.testdev-02-s11p2 192.testdev-02-s11p2 2017-08-10 09:52:40,532 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -f 189.testdev-02-s11p2 2017-08-10 09:52:40,553 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -f 190.testdev-02-s11p2 2017-08-10 09:52:40,571 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -f 191.testdev-02-s11p2 2017-08-10 09:52:40,589 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -f 192.testdev-02-s11p2 2017-08-10 09:52:40,608 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -f @testdev-02-s11p2.pbspro.com 2017-08-10 09:52:40,622 INFO expect on server testdev-02-s11p2: job_state set 0 job ... OK 2017-08-10 09:52:40,622 INFOCLI testdev-02-s11p2: /opt/pbs/bin/pbs_rstat -f 2017-08-10 09:52:40,636 INFO manager on testdev-02-s11p2: unset server ['job_sort_formula'] 2017-08-10 09:52:40,636 INFOCLI testdev-02-s11p2: sudo -H /opt/pbs/bin/qmgr -c unset server job_sort_formula 2017-08-10 09:52:40,863 INFOCLI testdev-02-s11p2: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-08-10 09:52:40,884 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -Qf @testdev-02-s11p2.pbspro.com 2017-08-10 09:52:40,898 INFO status on testdev-02-s11p2: node 2017-08-10 09:52:40,899 INFOCLI testdev-02-s11p2: /opt/pbs/bin/pbsnodes -s testdev-02-s11p2.pbspro.com -v -a 2017-08-10 09:52:40,912 INFO manager on testdev-02-s11p2: delete queue workq 2017-08-10 09:52:40,913 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c delete queue workq 2017-08-10 09:52:41,122 INFO server testdev-02-s11p2: expect offset set to 0.5 2017-08-10 09:52:41,623 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -Qf workq@testdev-02-s11p2.pbspro.com 2017-08-10 09:52:41,638 INFO expect on server testdev-02-s11p2: unset queue workq ... OK 2017-08-10 09:52:41,639 INFO manager on testdev-02-s11p2: create queue workq {'started': 'True', 'queue_type': 'Execution', 'enabled': 'True'} 2017-08-10 09:52:41,639 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c create queue workq started=True,queue_type=Execution,enabled=True 2017-08-10 09:52:41,892 INFO server testdev-02-s11p2: expect offset set to 0.5 2017-08-10 09:52:42,393 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -Qf workq@testdev-02-s11p2.pbspro.com 2017-08-10 09:52:42,409 INFO expect on server testdev-02-s11p2: started set True || queue_type set Execution || enabled set True queue workq ... OK 2017-08-10 09:52:42,409 INFO manager on testdev-02-s11p2: set server {'default_queue': 'workq'} 2017-08-10 09:52:42,409 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c set server default_queue=workq 2017-08-10 09:52:42,552 INFO status on testdev-02-s11p2: resource 2017-08-10 09:52:42,552 INFO manager on testdev-02-s11p2: list resource 2017-08-10 09:52:42,552 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c list resource 2017-08-10 09:52:42,568 INFOCLI status on testdev-02-s11p2: server license_count 2017-08-10 09:52:42,568 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -Bf testdev-02-s11p2.pbspro.com 2017-08-10 09:52:42,582 INFO server: testdev-02-s11p2.pbspro.com licensed 2017-08-10 09:52:42,593 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/server_priv/comm.lock 2017-08-10 09:52:42,615 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/sched_priv/sched.lock 2017-08-10 09:52:42,626 INFO scheduler testdev-02-s11p2: reverting configuration to defaults 2017-08-10 09:52:42,627 INFO manager on testdev-02-s11p2: delete sched @default 2017-08-10 09:52:42,627 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c delete sched @default 2017-08-10 09:52:42,640 INFO manager on testdev-02-s11p2: list sched 2017-08-10 09:52:42,641 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c list sched 2017-08-10 09:52:42,655 INFO manager on testdev-02-s11p2: unset sched ['sched_cycle_length'] 2017-08-10 09:52:42,655 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c unset sched sched_cycle_length 2017-08-10 09:52:42,712 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/sched_priv/dedicated_time 2017-08-10 09:52:42,724 INFOCLI2 testdev-02-s11p2: cmp /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:42,729 ERROR err: ['cmp: /var/spool/pbs/sched_priv/resource_group: Permission denied'] 2017-08-10 09:52:42,730 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:42,742 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:42,753 INFO scheduler testdev-02-s11p2: reverting holidays file to default 2017-08-10 09:52:42,754 INFOCLI2 testdev-02-s11p2: cmp /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-08-10 09:52:42,759 ERROR err: ['cmp: /var/spool/pbs/sched_priv/holidays: Permission denied'] 2017-08-10 09:52:42,759 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-08-10 09:52:42,771 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/holidays 2017-08-10 09:52:42,782 INFO scheduler testdev-02-s11p2: Parsing holidays file 2017-08-10 09:52:42,783 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/sched_priv/holidays 2017-08-10 09:52:42,794 INFOCLI2 testdev-02-s11p2: cmp /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-08-10 09:52:42,799 ERROR err: ['cmp: /var/spool/pbs/sched_priv/sched_config: Permission denied'] 2017-08-10 09:52:42,799 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-08-10 09:52:42,811 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-08-10 09:52:42,823 INFO scheduler testdev-02-s11p2: sent signal -HUP 2017-08-10 09:52:42,823 INFOCLI2 testdev-02-s11p2: sudo -H kill -HUP 12179 2017-08-10 09:52:42,834 INFOCLI2 testdev-02-s11p2: sudo -H /opt/pbs/sbin/pbsfs -e 2017-08-10 09:52:42,873 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/sched_priv/sched.lock 2017-08-10 09:52:42,895 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/mom_priv/mom.lock 2017-08-10 09:52:42,906 INFO mom testdev-02-s11p2: reverting configuration to defaults 2017-08-10 09:52:42,906 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/rm -f /var/spool/pbs/mom_priv/epilogue 2017-08-10 09:52:42,918 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/rm -f /var/spool/pbs/mom_priv/prologue 2017-08-10 09:52:42,929 INFOCLI testdev-02-s11p2: sudo -H /opt/pbs/sbin/pbs_mom -s list 2017-08-10 09:52:42,948 INFO mom testdev-02-s11p2: sent signal -HUP 2017-08-10 09:52:42,948 INFOCLI2 testdev-02-s11p2: sudo -H kill -HUP 12176 2017-08-10 09:52:42,971 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/mom_priv/mom.lock 2017-08-10 09:52:42,983 INFOCLI testdev-02-s11p2: /opt/pbs/bin/pbsnodes -s testdev-02-s11p2.pbspro.com -v -a 2017-08-10 09:52:42,997 INFO status on testdev-02-s11p2: node testdev-02-s11p2 2017-08-10 09:52:42,997 INFOCLI testdev-02-s11p2: /opt/pbs/bin/pbsnodes -s testdev-02-s11p2.pbspro.com -v testdev-02-s11p2 2017-08-10 09:52:43,010 INFOCLI testdev-02-s11p2: /opt/pbs/bin/pbsnodes -s testdev-02-s11p2.pbspro.com -v testdev-02-s11p2 2017-08-10 09:52:43,024 INFO expect on server testdev-02-s11p2: state = free node testdev-02-s11p2 ... OK 2017-08-10 09:52:43,024 INFO ============================= 2017-08-10 09:52:43,025 INFO Completed TestFairshare setUp 2017-08-10 09:52:43,025 INFO ============================= 2017-08-10 09:52:43,025 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:43,037 INFO creating tree node: group1 2017-08-10 09:52:43,037 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /tmp/PtlPbsUXus6B /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:43,049 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:43,061 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chown root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:43,072 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chgrp root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:43,084 INFO creating tree node: group2 2017-08-10 09:52:43,084 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /tmp/PtlPbsRu9aqr /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:43,096 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:43,108 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chown root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:43,119 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chgrp root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:43,131 INFO creating tree node: pbsuser 2017-08-10 09:52:43,131 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /tmp/PtlPbsbL6Edp /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:43,143 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:43,155 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chown root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:43,166 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chgrp root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:43,178 INFO creating tree node: pbsuser1 2017-08-10 09:52:43,178 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /tmp/PtlPbsdo_S3y /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:43,190 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:43,201 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chown root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:43,213 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chgrp root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:43,224 INFO creating tree node: pbsuser2 2017-08-10 09:52:43,225 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /tmp/PtlPbsxUyZdm /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:43,237 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:43,249 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chown root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:43,261 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chgrp root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:43,272 INFO creating tree node: pbsuser3 2017-08-10 09:52:43,273 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /tmp/PtlPbsL9rAkm /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:43,285 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:43,296 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chown root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:43,307 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chgrp root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:43,319 INFOCLI2 testdev-02-s11p2: sudo -H /opt/pbs/sbin/pbsfs -s pbsuser 100 2017-08-10 09:52:43,335 INFOCLI2 testdev-02-s11p2: sudo -H /opt/pbs/sbin/pbsfs -s pbsuser1 100 2017-08-10 09:52:43,351 INFOCLI2 testdev-02-s11p2: sudo -H /opt/pbs/sbin/pbsfs -s pbsuser3 1000 2017-08-10 09:52:43,367 INFO scheduler testdev-02-s11p2: config {'log_filter': 2048} 2017-08-10 09:52:43,368 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /var/spool/pbs/sched_priv/sched_config /var/spool/pbs/sched_priv/sched_config.bak 2017-08-10 09:52:43,380 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /tmp/PtlPbs_qLj_Y /var/spool/pbs/sched_priv/sched_config 2017-08-10 09:52:43,392 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-08-10 09:52:43,404 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chown root /var/spool/pbs/sched_priv/sched_config 2017-08-10 09:52:43,415 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chgrp root /var/spool/pbs/sched_priv/sched_config 2017-08-10 09:52:43,426 INFO scheduler testdev-02-s11p2: sent signal -HUP 2017-08-10 09:52:43,427 INFOCLI2 testdev-02-s11p2: sudo -H kill -HUP 12179 2017-08-10 09:52:43,438 INFO scheduler testdev-02-s11p2 log match: searching for "Error reading line" - No match 2017-08-10 09:52:44,440 INFO manager on testdev-02-s11p2: set server {'scheduling': 'False'} 2017-08-10 09:52:44,440 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c set server scheduling=False 2017-08-10 09:52:44,662 INFO manager on testdev-02-s11p2: set server {'job_sort_formula': '"pow(2,-(fairshare_tree_usage/fairshare_perc))"'} 2017-08-10 09:52:44,663 INFOCLI testdev-02-s11p2: sudo -H /opt/pbs/bin/qmgr -c set server job_sort_formula="pow(2,-(fairshare_tree_usage/fairshare_perc))" 2017-08-10 09:52:44,783 INFO job: executable set to /bin/sleep with arguments: 100 2017-08-10 09:52:44,783 INFOCLI testdev-02-s11p2: sudo -H -u pbsuser2 /opt/pbs/bin/qsub -- /bin/sleep 100 2017-08-10 09:52:44,842 INFO submit to testdev-02-s11p2 as pbsuser2: job 193.testdev-02-s11p2 2017-08-10 09:52:44,843 INFO job: executable set to /bin/sleep with arguments: 100 2017-08-10 09:52:44,843 INFOCLI testdev-02-s11p2: sudo -H -u pbsuser3 /opt/pbs/bin/qsub -- /bin/sleep 100 2017-08-10 09:52:44,892 INFO submit to testdev-02-s11p2 as pbsuser3: job 194.testdev-02-s11p2 2017-08-10 09:52:44,893 INFO job: executable set to /bin/sleep with arguments: 100 2017-08-10 09:52:44,894 INFOCLI testdev-02-s11p2: sudo -H -u pbsuser /opt/pbs/bin/qsub -- /bin/sleep 100 2017-08-10 09:52:44,982 INFO submit to testdev-02-s11p2 as pbsuser: job 195.testdev-02-s11p2 2017-08-10 09:52:44,983 INFO job: executable set to /bin/sleep with arguments: 100 2017-08-10 09:52:44,983 INFOCLI testdev-02-s11p2: sudo -H -u pbsuser1 /opt/pbs/bin/qsub -- /bin/sleep 100 2017-08-10 09:52:45,042 INFO submit to testdev-02-s11p2 as pbsuser1: job 196.testdev-02-s11p2 2017-08-10 09:52:45,043 INFO manager on testdev-02-s11p2: set server {'scheduling': 'True'} 2017-08-10 09:52:45,043 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-08-10 09:52:45,132 INFO scheduler testdev-02-s11p2 log match: searching for "193.testdev-02-s11p2;Formula Evaluation = 0.3816" - No match 2017-08-10 09:52:46,133 INFO scheduler testdev-02-s11p2 log match: searching for "193.testdev-02-s11p2;Formula Evaluation = 0.3816"... OK 2017-08-10 09:52:46,134 INFO scheduler testdev-02-s11p2 log match: searching for "194.testdev-02-s11p2;Formula Evaluation = 0.0902"... OK 2017-08-10 09:52:46,134 INFO scheduler testdev-02-s11p2 log match: searching for "195.testdev-02-s11p2;Formula Evaluation = 0.6477"... OK 2017-08-10 09:52:46,134 INFO scheduler testdev-02-s11p2 log match: searching for "196.testdev-02-s11p2;Formula Evaluation = 0.6477"... OK 2017-08-10 09:52:46,135 INFO scheduler testdev-02-s11p2 log match: searching for "Leaving Scheduling Cycle"... OK 2017-08-10 09:52:46,154 INFO =============================== 2017-08-10 09:52:46,154 INFO Entered TestFairshare tearDown 2017-08-10 09:52:46,154 INFO =============================== 2017-08-10 09:52:46,154 INFO ================================ 2017-08-10 09:52:46,155 INFO Completed TestFairshare tearDown 2017-08-10 09:52:46,155 INFO ================================ 2017-08-10 09:52:46,156 INFO ok 2017-08-10 09:52:46,156 INFO test name: test_fairshare_formula4 (tests.functional.pbs_fairshare.TestFairshare)... 2017-08-10 09:52:46,157 INFO test start time: Thu Aug 10 09:52:46 2017 2017-08-10 09:52:46,157 INFO test docstring: Test fairshare in the formula. Make sure entities with small usage are negatively affected by their high usage siblings. Make sure that jobs run in the correct order. Use keyword fairshare_factor 2017-08-10 09:52:46,157 INFO ============================ 2017-08-10 09:52:46,157 INFO Entered TestFairshare setUp 2017-08-10 09:52:46,157 INFO ============================ 2017-08-10 09:52:46,158 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -Bf testdev-02-s11p2.pbspro.com 2017-08-10 09:52:46,174 INFO status on testdev-02-s11p2: server 2017-08-10 09:52:46,175 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -Bf testdev-02-s11p2.pbspro.com 2017-08-10 09:52:46,188 INFO manager on testdev-02-s11p2: set server {'managers': (3, 'anamika@*')} 2017-08-10 09:52:46,189 INFOCLI testdev-02-s11p2: sudo -H /opt/pbs/bin/qmgr -c set server managers-=anamika@* 2017-08-10 09:52:46,312 INFO manager on testdev-02-s11p2: set server {'managers': (2, 'anamika@*')} 2017-08-10 09:52:46,313 INFOCLI testdev-02-s11p2: sudo -H /opt/pbs/bin/qmgr -c set server managers+=anamika@* 2017-08-10 09:52:46,392 INFO server testdev-02-s11p2: reverting configuration to defaults 2017-08-10 09:52:46,393 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -Bf testdev-02-s11p2.pbspro.com 2017-08-10 09:52:46,408 INFO select on testdev-02-s11p2: __ALL__ 2017-08-10 09:52:46,408 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qselect 2017-08-10 09:52:46,421 INFO delete job on testdev-02-s11p2: 193.testdev-02-s11p2, 194.testdev-02-s11p2, 195.testdev-02-s11p2, 196.testdev-02-s11p2 2017-08-10 09:52:46,422 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qdel -W force 193.testdev-02-s11p2 194.testdev-02-s11p2 195.testdev-02-s11p2 196.testdev-02-s11p2 2017-08-10 09:52:47,035 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -f 193.testdev-02-s11p2 2017-08-10 09:52:47,056 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -f 194.testdev-02-s11p2 2017-08-10 09:52:47,074 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -f 195.testdev-02-s11p2 2017-08-10 09:52:47,092 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -f 196.testdev-02-s11p2 2017-08-10 09:52:47,110 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -f @testdev-02-s11p2.pbspro.com 2017-08-10 09:52:47,124 INFO expect on server testdev-02-s11p2: job_state set 0 job ... OK 2017-08-10 09:52:47,124 INFOCLI testdev-02-s11p2: /opt/pbs/bin/pbs_rstat -f 2017-08-10 09:52:47,138 INFO manager on testdev-02-s11p2: unset server ['job_sort_formula'] 2017-08-10 09:52:47,138 INFOCLI testdev-02-s11p2: sudo -H /opt/pbs/bin/qmgr -c unset server job_sort_formula 2017-08-10 09:52:47,303 INFOCLI testdev-02-s11p2: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-08-10 09:52:47,324 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -Qf @testdev-02-s11p2.pbspro.com 2017-08-10 09:52:47,339 INFO status on testdev-02-s11p2: node 2017-08-10 09:52:47,339 INFOCLI testdev-02-s11p2: /opt/pbs/bin/pbsnodes -s testdev-02-s11p2.pbspro.com -v -a 2017-08-10 09:52:47,353 INFO manager on testdev-02-s11p2: delete queue workq 2017-08-10 09:52:47,353 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c delete queue workq 2017-08-10 09:52:47,452 INFO server testdev-02-s11p2: expect offset set to 0.5 2017-08-10 09:52:47,953 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -Qf workq@testdev-02-s11p2.pbspro.com 2017-08-10 09:52:47,970 INFO expect on server testdev-02-s11p2: unset queue workq ... OK 2017-08-10 09:52:47,971 INFO manager on testdev-02-s11p2: create queue workq {'started': 'True', 'queue_type': 'Execution', 'enabled': 'True'} 2017-08-10 09:52:47,971 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c create queue workq started=True,queue_type=Execution,enabled=True 2017-08-10 09:52:48,172 INFO server testdev-02-s11p2: expect offset set to 0.5 2017-08-10 09:52:48,673 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -Qf workq@testdev-02-s11p2.pbspro.com 2017-08-10 09:52:48,689 INFO expect on server testdev-02-s11p2: started set True || queue_type set Execution || enabled set True queue workq ... OK 2017-08-10 09:52:48,690 INFO manager on testdev-02-s11p2: set server {'default_queue': 'workq'} 2017-08-10 09:52:48,690 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c set server default_queue=workq 2017-08-10 09:52:48,782 INFO status on testdev-02-s11p2: resource 2017-08-10 09:52:48,783 INFO manager on testdev-02-s11p2: list resource 2017-08-10 09:52:48,783 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c list resource 2017-08-10 09:52:48,798 INFOCLI status on testdev-02-s11p2: server license_count 2017-08-10 09:52:48,799 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -Bf testdev-02-s11p2.pbspro.com 2017-08-10 09:52:48,812 INFO server: testdev-02-s11p2.pbspro.com licensed 2017-08-10 09:52:48,823 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/server_priv/comm.lock 2017-08-10 09:52:48,845 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/sched_priv/sched.lock 2017-08-10 09:52:48,857 INFO scheduler testdev-02-s11p2: reverting configuration to defaults 2017-08-10 09:52:48,857 INFO manager on testdev-02-s11p2: delete sched @default 2017-08-10 09:52:48,858 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c delete sched @default 2017-08-10 09:52:48,871 INFO manager on testdev-02-s11p2: list sched 2017-08-10 09:52:48,872 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c list sched 2017-08-10 09:52:48,885 INFO manager on testdev-02-s11p2: unset sched ['sched_cycle_length'] 2017-08-10 09:52:48,886 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c unset sched sched_cycle_length 2017-08-10 09:52:48,942 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/sched_priv/dedicated_time 2017-08-10 09:52:48,954 INFOCLI2 testdev-02-s11p2: cmp /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:48,959 ERROR err: ['cmp: /var/spool/pbs/sched_priv/resource_group: Permission denied'] 2017-08-10 09:52:48,960 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:48,972 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:48,983 INFO scheduler testdev-02-s11p2: reverting holidays file to default 2017-08-10 09:52:48,984 INFOCLI2 testdev-02-s11p2: cmp /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-08-10 09:52:48,988 ERROR err: ['cmp: /var/spool/pbs/sched_priv/holidays: Permission denied'] 2017-08-10 09:52:48,989 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-08-10 09:52:49,001 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/holidays 2017-08-10 09:52:49,013 INFO scheduler testdev-02-s11p2: Parsing holidays file 2017-08-10 09:52:49,013 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/sched_priv/holidays 2017-08-10 09:52:49,025 INFOCLI2 testdev-02-s11p2: cmp /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-08-10 09:52:49,031 ERROR err: ['cmp: /var/spool/pbs/sched_priv/sched_config: Permission denied'] 2017-08-10 09:52:49,031 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-08-10 09:52:49,043 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-08-10 09:52:49,055 INFO scheduler testdev-02-s11p2: sent signal -HUP 2017-08-10 09:52:49,056 INFOCLI2 testdev-02-s11p2: sudo -H kill -HUP 12179 2017-08-10 09:52:49,067 INFOCLI2 testdev-02-s11p2: sudo -H /opt/pbs/sbin/pbsfs -e 2017-08-10 09:52:49,107 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/sched_priv/sched.lock 2017-08-10 09:52:49,129 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/mom_priv/mom.lock 2017-08-10 09:52:49,141 INFO mom testdev-02-s11p2: reverting configuration to defaults 2017-08-10 09:52:49,141 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/rm -f /var/spool/pbs/mom_priv/epilogue 2017-08-10 09:52:49,153 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/rm -f /var/spool/pbs/mom_priv/prologue 2017-08-10 09:52:49,165 INFOCLI testdev-02-s11p2: sudo -H /opt/pbs/sbin/pbs_mom -s list 2017-08-10 09:52:49,183 INFO mom testdev-02-s11p2: sent signal -HUP 2017-08-10 09:52:49,184 INFOCLI2 testdev-02-s11p2: sudo -H kill -HUP 12176 2017-08-10 09:52:49,206 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/mom_priv/mom.lock 2017-08-10 09:52:49,218 INFOCLI testdev-02-s11p2: /opt/pbs/bin/pbsnodes -s testdev-02-s11p2.pbspro.com -v -a 2017-08-10 09:52:49,232 INFO status on testdev-02-s11p2: node testdev-02-s11p2 2017-08-10 09:52:49,233 INFOCLI testdev-02-s11p2: /opt/pbs/bin/pbsnodes -s testdev-02-s11p2.pbspro.com -v testdev-02-s11p2 2017-08-10 09:52:49,246 INFOCLI testdev-02-s11p2: /opt/pbs/bin/pbsnodes -s testdev-02-s11p2.pbspro.com -v testdev-02-s11p2 2017-08-10 09:52:49,260 INFO expect on server testdev-02-s11p2: state = free node testdev-02-s11p2 ... OK 2017-08-10 09:52:49,261 INFO ============================= 2017-08-10 09:52:49,261 INFO Completed TestFairshare setUp 2017-08-10 09:52:49,261 INFO ============================= 2017-08-10 09:52:49,261 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:49,273 INFO creating tree node: group1 2017-08-10 09:52:49,274 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /tmp/PtlPbsty4wWc /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:49,286 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:49,298 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chown root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:49,309 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chgrp root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:49,321 INFO creating tree node: group2 2017-08-10 09:52:49,321 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /tmp/PtlPbsDOuZHe /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:49,333 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:49,345 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chown root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:49,357 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chgrp root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:49,368 INFO creating tree node: pbsuser 2017-08-10 09:52:49,369 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /tmp/PtlPbs878rad /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:49,381 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:49,392 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chown root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:49,404 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chgrp root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:49,416 INFO creating tree node: pbsuser1 2017-08-10 09:52:49,416 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /tmp/PtlPbsnfMwxJ /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:49,428 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:49,440 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chown root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:49,451 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chgrp root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:49,463 INFO creating tree node: pbsuser2 2017-08-10 09:52:49,464 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /tmp/PtlPbsonZTv2 /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:49,476 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:49,487 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chown root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:49,499 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chgrp root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:49,510 INFO creating tree node: pbsuser3 2017-08-10 09:52:49,511 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /tmp/PtlPbsIB89rx /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:49,523 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:49,535 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chown root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:49,546 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chgrp root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:49,558 INFOCLI2 testdev-02-s11p2: sudo -H /opt/pbs/sbin/pbsfs -s pbsuser 100 2017-08-10 09:52:49,574 INFOCLI2 testdev-02-s11p2: sudo -H /opt/pbs/sbin/pbsfs -s pbsuser1 100 2017-08-10 09:52:49,591 INFOCLI2 testdev-02-s11p2: sudo -H /opt/pbs/sbin/pbsfs -s pbsuser3 1000 2017-08-10 09:52:49,608 INFO scheduler testdev-02-s11p2: config {'log_filter': 2048} 2017-08-10 09:52:49,609 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /var/spool/pbs/sched_priv/sched_config /var/spool/pbs/sched_priv/sched_config.bak 2017-08-10 09:52:49,621 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /tmp/PtlPbsqte32s /var/spool/pbs/sched_priv/sched_config 2017-08-10 09:52:49,634 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-08-10 09:52:49,646 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chown root /var/spool/pbs/sched_priv/sched_config 2017-08-10 09:52:49,658 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chgrp root /var/spool/pbs/sched_priv/sched_config 2017-08-10 09:52:49,670 INFO scheduler testdev-02-s11p2: sent signal -HUP 2017-08-10 09:52:49,670 INFOCLI2 testdev-02-s11p2: sudo -H kill -HUP 12179 2017-08-10 09:52:49,681 INFO scheduler testdev-02-s11p2 log match: searching for "Error reading line" - No match 2017-08-10 09:52:50,683 INFO manager on testdev-02-s11p2: set server {'scheduling': 'False'} 2017-08-10 09:52:50,684 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c set server scheduling=False 2017-08-10 09:52:50,972 INFO manager on testdev-02-s11p2: set server {'job_sort_formula': 'fairshare_factor'} 2017-08-10 09:52:50,973 INFOCLI testdev-02-s11p2: sudo -H /opt/pbs/bin/qmgr -c set server job_sort_formula=fairshare_factor 2017-08-10 09:52:51,063 INFO job: executable set to /bin/sleep with arguments: 100 2017-08-10 09:52:51,064 INFOCLI testdev-02-s11p2: sudo -H -u pbsuser2 /opt/pbs/bin/qsub -- /bin/sleep 100 2017-08-10 09:52:51,122 INFO submit to testdev-02-s11p2 as pbsuser2: job 197.testdev-02-s11p2 2017-08-10 09:52:51,123 INFO job: executable set to /bin/sleep with arguments: 100 2017-08-10 09:52:51,124 INFOCLI testdev-02-s11p2: sudo -H -u pbsuser3 /opt/pbs/bin/qsub -- /bin/sleep 100 2017-08-10 09:52:51,242 INFO submit to testdev-02-s11p2 as pbsuser3: job 198.testdev-02-s11p2 2017-08-10 09:52:51,243 INFO job: executable set to /bin/sleep with arguments: 100 2017-08-10 09:52:51,244 INFOCLI testdev-02-s11p2: sudo -H -u pbsuser /opt/pbs/bin/qsub -- /bin/sleep 100 2017-08-10 09:52:51,322 INFO submit to testdev-02-s11p2 as pbsuser: job 199.testdev-02-s11p2 2017-08-10 09:52:51,323 INFO job: executable set to /bin/sleep with arguments: 100 2017-08-10 09:52:51,324 INFOCLI testdev-02-s11p2: sudo -H -u pbsuser1 /opt/pbs/bin/qsub -- /bin/sleep 100 2017-08-10 09:52:51,382 INFO submit to testdev-02-s11p2 as pbsuser1: job 200.testdev-02-s11p2 2017-08-10 09:52:51,383 INFO manager on testdev-02-s11p2: set server {'scheduling': 'True'} 2017-08-10 09:52:51,383 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-08-10 09:52:51,453 INFO scheduler testdev-02-s11p2 log match: searching for "197.testdev-02-s11p2;Formula Evaluation = 0.3816" - No match 2017-08-10 09:52:52,455 INFO scheduler testdev-02-s11p2 log match: searching for "197.testdev-02-s11p2;Formula Evaluation = 0.3816"... OK 2017-08-10 09:52:52,455 INFO scheduler testdev-02-s11p2 log match: searching for "198.testdev-02-s11p2;Formula Evaluation = 0.0902"... OK 2017-08-10 09:52:52,455 INFO scheduler testdev-02-s11p2 log match: searching for "199.testdev-02-s11p2;Formula Evaluation = 0.6477"... OK 2017-08-10 09:52:52,455 INFO scheduler testdev-02-s11p2 log match: searching for "200.testdev-02-s11p2;Formula Evaluation = 0.6477"... OK 2017-08-10 09:52:52,456 INFO scheduler testdev-02-s11p2 log match: searching for "Leaving Scheduling Cycle"... OK 2017-08-10 09:52:52,478 INFO =============================== 2017-08-10 09:52:52,478 INFO Entered TestFairshare tearDown 2017-08-10 09:52:52,479 INFO =============================== 2017-08-10 09:52:52,479 INFO ================================ 2017-08-10 09:52:52,479 INFO Completed TestFairshare tearDown 2017-08-10 09:52:52,479 INFO ================================ 2017-08-10 09:52:52,480 INFO ok 2017-08-10 09:52:52,481 INFO test name: test_fairshare_formula5 (tests.functional.pbs_fairshare.TestFairshare)... 2017-08-10 09:52:52,481 INFO test start time: Thu Aug 10 09:52:52 2017 2017-08-10 09:52:52,481 INFO test docstring: Test fairshare in the formula with fair_share set to true in scheduler. Make sure formula takes precedence over fairshare usage. Output will be same as in test_fairshare_formula4. 2017-08-10 09:52:52,481 INFO ============================ 2017-08-10 09:52:52,481 INFO Entered TestFairshare setUp 2017-08-10 09:52:52,482 INFO ============================ 2017-08-10 09:52:52,482 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -Bf testdev-02-s11p2.pbspro.com 2017-08-10 09:52:52,499 INFO status on testdev-02-s11p2: server 2017-08-10 09:52:52,499 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -Bf testdev-02-s11p2.pbspro.com 2017-08-10 09:52:52,513 INFO manager on testdev-02-s11p2: set server {'managers': (3, 'anamika@*')} 2017-08-10 09:52:52,514 INFOCLI testdev-02-s11p2: sudo -H /opt/pbs/bin/qmgr -c set server managers-=anamika@* 2017-08-10 09:52:52,783 INFO manager on testdev-02-s11p2: set server {'managers': (2, 'anamika@*')} 2017-08-10 09:52:52,784 INFOCLI testdev-02-s11p2: sudo -H /opt/pbs/bin/qmgr -c set server managers+=anamika@* 2017-08-10 09:52:52,853 INFO server testdev-02-s11p2: reverting configuration to defaults 2017-08-10 09:52:52,853 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -Bf testdev-02-s11p2.pbspro.com 2017-08-10 09:52:52,868 INFO select on testdev-02-s11p2: __ALL__ 2017-08-10 09:52:52,868 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qselect 2017-08-10 09:52:52,881 INFO delete job on testdev-02-s11p2: 197.testdev-02-s11p2, 198.testdev-02-s11p2, 199.testdev-02-s11p2, 200.testdev-02-s11p2 2017-08-10 09:52:52,882 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qdel -W force 197.testdev-02-s11p2 198.testdev-02-s11p2 199.testdev-02-s11p2 200.testdev-02-s11p2 2017-08-10 09:52:53,576 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -f 197.testdev-02-s11p2 2017-08-10 09:52:53,597 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -f 198.testdev-02-s11p2 2017-08-10 09:52:53,615 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -f 199.testdev-02-s11p2 2017-08-10 09:52:53,633 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -f 200.testdev-02-s11p2 2017-08-10 09:52:53,651 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -f @testdev-02-s11p2.pbspro.com 2017-08-10 09:52:53,665 INFO expect on server testdev-02-s11p2: job_state set 0 job ... OK 2017-08-10 09:52:53,665 INFOCLI testdev-02-s11p2: /opt/pbs/bin/pbs_rstat -f 2017-08-10 09:52:53,678 INFO manager on testdev-02-s11p2: unset server ['job_sort_formula'] 2017-08-10 09:52:53,679 INFOCLI testdev-02-s11p2: sudo -H /opt/pbs/bin/qmgr -c unset server job_sort_formula 2017-08-10 09:52:53,853 INFOCLI testdev-02-s11p2: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-08-10 09:52:53,875 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -Qf @testdev-02-s11p2.pbspro.com 2017-08-10 09:52:53,890 INFO status on testdev-02-s11p2: node 2017-08-10 09:52:53,890 INFOCLI testdev-02-s11p2: /opt/pbs/bin/pbsnodes -s testdev-02-s11p2.pbspro.com -v -a 2017-08-10 09:52:53,904 INFO manager on testdev-02-s11p2: delete queue workq 2017-08-10 09:52:53,904 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c delete queue workq 2017-08-10 09:52:53,992 INFO server testdev-02-s11p2: expect offset set to 0.5 2017-08-10 09:52:54,493 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -Qf workq@testdev-02-s11p2.pbspro.com 2017-08-10 09:52:54,509 INFO expect on server testdev-02-s11p2: unset queue workq ... OK 2017-08-10 09:52:54,509 INFO manager on testdev-02-s11p2: create queue workq {'started': 'True', 'queue_type': 'Execution', 'enabled': 'True'} 2017-08-10 09:52:54,510 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c create queue workq started=True,queue_type=Execution,enabled=True 2017-08-10 09:52:54,763 INFO server testdev-02-s11p2: expect offset set to 0.5 2017-08-10 09:52:55,264 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -Qf workq@testdev-02-s11p2.pbspro.com 2017-08-10 09:52:55,280 INFO expect on server testdev-02-s11p2: started set True || queue_type set Execution || enabled set True queue workq ... OK 2017-08-10 09:52:55,280 INFO manager on testdev-02-s11p2: set server {'default_queue': 'workq'} 2017-08-10 09:52:55,280 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c set server default_queue=workq 2017-08-10 09:52:55,432 INFO status on testdev-02-s11p2: resource 2017-08-10 09:52:55,433 INFO manager on testdev-02-s11p2: list resource 2017-08-10 09:52:55,433 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c list resource 2017-08-10 09:52:55,448 INFOCLI status on testdev-02-s11p2: server license_count 2017-08-10 09:52:55,448 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -Bf testdev-02-s11p2.pbspro.com 2017-08-10 09:52:55,462 INFO server: testdev-02-s11p2.pbspro.com licensed 2017-08-10 09:52:55,473 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/server_priv/comm.lock 2017-08-10 09:52:55,495 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/sched_priv/sched.lock 2017-08-10 09:52:55,507 INFO scheduler testdev-02-s11p2: reverting configuration to defaults 2017-08-10 09:52:55,507 INFO manager on testdev-02-s11p2: delete sched @default 2017-08-10 09:52:55,507 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c delete sched @default 2017-08-10 09:52:55,521 INFO manager on testdev-02-s11p2: list sched 2017-08-10 09:52:55,521 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c list sched 2017-08-10 09:52:55,535 INFO manager on testdev-02-s11p2: unset sched ['sched_cycle_length'] 2017-08-10 09:52:55,535 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c unset sched sched_cycle_length 2017-08-10 09:52:55,593 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/sched_priv/dedicated_time 2017-08-10 09:52:55,605 INFOCLI2 testdev-02-s11p2: cmp /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:55,610 ERROR err: ['cmp: /var/spool/pbs/sched_priv/resource_group: Permission denied'] 2017-08-10 09:52:55,611 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:55,623 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:55,636 INFO scheduler testdev-02-s11p2: reverting holidays file to default 2017-08-10 09:52:55,637 INFOCLI2 testdev-02-s11p2: cmp /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-08-10 09:52:55,642 ERROR err: ['cmp: /var/spool/pbs/sched_priv/holidays: Permission denied'] 2017-08-10 09:52:55,642 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-08-10 09:52:55,654 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/holidays 2017-08-10 09:52:55,666 INFO scheduler testdev-02-s11p2: Parsing holidays file 2017-08-10 09:52:55,666 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/sched_priv/holidays 2017-08-10 09:52:55,678 INFOCLI2 testdev-02-s11p2: cmp /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-08-10 09:52:55,683 ERROR err: ['cmp: /var/spool/pbs/sched_priv/sched_config: Permission denied'] 2017-08-10 09:52:55,684 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-08-10 09:52:55,695 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-08-10 09:52:55,707 INFO scheduler testdev-02-s11p2: sent signal -HUP 2017-08-10 09:52:55,708 INFOCLI2 testdev-02-s11p2: sudo -H kill -HUP 12179 2017-08-10 09:52:55,719 INFOCLI2 testdev-02-s11p2: sudo -H /opt/pbs/sbin/pbsfs -e 2017-08-10 09:52:55,759 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/sched_priv/sched.lock 2017-08-10 09:52:55,781 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/mom_priv/mom.lock 2017-08-10 09:52:55,792 INFO mom testdev-02-s11p2: reverting configuration to defaults 2017-08-10 09:52:55,793 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/rm -f /var/spool/pbs/mom_priv/epilogue 2017-08-10 09:52:55,804 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/rm -f /var/spool/pbs/mom_priv/prologue 2017-08-10 09:52:55,816 INFOCLI testdev-02-s11p2: sudo -H /opt/pbs/sbin/pbs_mom -s list 2017-08-10 09:52:55,835 INFO mom testdev-02-s11p2: sent signal -HUP 2017-08-10 09:52:55,835 INFOCLI2 testdev-02-s11p2: sudo -H kill -HUP 12176 2017-08-10 09:52:55,858 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/mom_priv/mom.lock 2017-08-10 09:52:55,869 INFOCLI testdev-02-s11p2: /opt/pbs/bin/pbsnodes -s testdev-02-s11p2.pbspro.com -v -a 2017-08-10 09:52:55,884 INFO status on testdev-02-s11p2: node testdev-02-s11p2 2017-08-10 09:52:55,884 INFOCLI testdev-02-s11p2: /opt/pbs/bin/pbsnodes -s testdev-02-s11p2.pbspro.com -v testdev-02-s11p2 2017-08-10 09:52:55,898 INFOCLI testdev-02-s11p2: /opt/pbs/bin/pbsnodes -s testdev-02-s11p2.pbspro.com -v testdev-02-s11p2 2017-08-10 09:52:55,911 INFO expect on server testdev-02-s11p2: state = free node testdev-02-s11p2 ... OK 2017-08-10 09:52:55,912 INFO ============================= 2017-08-10 09:52:55,912 INFO Completed TestFairshare setUp 2017-08-10 09:52:55,912 INFO ============================= 2017-08-10 09:52:55,912 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:55,924 INFO creating tree node: group1 2017-08-10 09:52:55,925 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /tmp/PtlPbstKCWtB /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:55,937 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:55,948 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chown root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:55,960 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chgrp root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:55,971 INFO creating tree node: group2 2017-08-10 09:52:55,972 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /tmp/PtlPbsZ8L2NA /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:55,984 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:55,996 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chown root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:56,007 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chgrp root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:56,019 INFO creating tree node: pbsuser 2017-08-10 09:52:56,019 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /tmp/PtlPbsssUJ_k /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:56,031 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:56,043 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chown root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:56,055 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chgrp root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:56,066 INFO creating tree node: pbsuser1 2017-08-10 09:52:56,067 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /tmp/PtlPbskxwpYz /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:56,079 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:56,090 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chown root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:56,102 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chgrp root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:56,113 INFO creating tree node: pbsuser2 2017-08-10 09:52:56,114 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /tmp/PtlPbsYSRNUS /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:56,126 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:56,138 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chown root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:56,149 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chgrp root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:56,161 INFO creating tree node: pbsuser3 2017-08-10 09:52:56,161 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /tmp/PtlPbs_7GDNs /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:56,173 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:56,185 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chown root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:56,197 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chgrp root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:52:56,208 INFOCLI2 testdev-02-s11p2: sudo -H /opt/pbs/sbin/pbsfs -s pbsuser 100 2017-08-10 09:52:56,224 INFOCLI2 testdev-02-s11p2: sudo -H /opt/pbs/sbin/pbsfs -s pbsuser1 100 2017-08-10 09:52:56,241 INFOCLI2 testdev-02-s11p2: sudo -H /opt/pbs/sbin/pbsfs -s pbsuser3 1000 2017-08-10 09:52:56,257 INFO scheduler testdev-02-s11p2: config {'log_filter': 2048, 'fair_share': 'True All'} 2017-08-10 09:52:56,258 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /var/spool/pbs/sched_priv/sched_config /var/spool/pbs/sched_priv/sched_config.bak 2017-08-10 09:52:56,270 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /tmp/PtlPbs2ZxHi2 /var/spool/pbs/sched_priv/sched_config 2017-08-10 09:52:56,283 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-08-10 09:52:56,294 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chown root /var/spool/pbs/sched_priv/sched_config 2017-08-10 09:52:56,306 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chgrp root /var/spool/pbs/sched_priv/sched_config 2017-08-10 09:52:56,317 INFO scheduler testdev-02-s11p2: sent signal -HUP 2017-08-10 09:52:56,318 INFOCLI2 testdev-02-s11p2: sudo -H kill -HUP 12179 2017-08-10 09:52:56,329 INFO scheduler testdev-02-s11p2 log match: searching for "Error reading line" - No match 2017-08-10 09:52:57,331 INFO manager on testdev-02-s11p2: set server {'scheduling': 'False'} 2017-08-10 09:52:57,331 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c set server scheduling=False 2017-08-10 09:52:57,493 INFO manager on testdev-02-s11p2: set server {'job_sort_formula': 'fairshare_factor'} 2017-08-10 09:52:57,493 INFOCLI testdev-02-s11p2: sudo -H /opt/pbs/bin/qmgr -c set server job_sort_formula=fairshare_factor 2017-08-10 09:52:57,563 INFO job: executable set to /bin/sleep with arguments: 100 2017-08-10 09:52:57,564 INFOCLI testdev-02-s11p2: sudo -H -u pbsuser2 /opt/pbs/bin/qsub -l cput=10 -- /bin/sleep 100 2017-08-10 09:52:57,653 INFO submit to testdev-02-s11p2 as pbsuser2: job 201.testdev-02-s11p2 OrderedDict([('Resource_List.cput', 10)]) 2017-08-10 09:52:57,658 INFO job: executable set to /bin/sleep with arguments: 100 2017-08-10 09:52:57,658 INFOCLI testdev-02-s11p2: sudo -H -u pbsuser3 /opt/pbs/bin/qsub -l cput=20 -- /bin/sleep 100 2017-08-10 09:52:57,723 INFO submit to testdev-02-s11p2 as pbsuser3: job 202.testdev-02-s11p2 OrderedDict([('Resource_List.cput', 20)]) 2017-08-10 09:52:57,724 INFO job: executable set to /bin/sleep with arguments: 100 2017-08-10 09:52:57,724 INFOCLI testdev-02-s11p2: sudo -H -u pbsuser /opt/pbs/bin/qsub -l cput=30 -- /bin/sleep 100 2017-08-10 09:52:57,793 INFO submit to testdev-02-s11p2 as pbsuser: job 203.testdev-02-s11p2 OrderedDict([('Resource_List.cput', 30)]) 2017-08-10 09:52:57,794 INFO job: executable set to /bin/sleep with arguments: 100 2017-08-10 09:52:57,794 INFOCLI testdev-02-s11p2: sudo -H -u pbsuser1 /opt/pbs/bin/qsub -l cput=40 -- /bin/sleep 100 2017-08-10 09:52:57,863 INFO submit to testdev-02-s11p2 as pbsuser1: job 204.testdev-02-s11p2 OrderedDict([('Resource_List.cput', 40)]) 2017-08-10 09:52:57,863 INFO manager on testdev-02-s11p2: set server {'scheduling': 'True'} 2017-08-10 09:52:57,864 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-08-10 09:52:57,913 INFO scheduler testdev-02-s11p2 log match: searching for "201.testdev-02-s11p2;Formula Evaluation = 0.3816" - No match 2017-08-10 09:52:58,913 INFO scheduler testdev-02-s11p2 log match: searching for "201.testdev-02-s11p2;Formula Evaluation = 0.3816"... OK 2017-08-10 09:52:58,913 INFO scheduler testdev-02-s11p2 log match: searching for "202.testdev-02-s11p2;Formula Evaluation = 0.0902"... OK 2017-08-10 09:52:58,914 INFO scheduler testdev-02-s11p2 log match: searching for "203.testdev-02-s11p2;Formula Evaluation = 0.6477"... OK 2017-08-10 09:52:58,914 INFO scheduler testdev-02-s11p2 log match: searching for "204.testdev-02-s11p2;Formula Evaluation = 0.6477"... OK 2017-08-10 09:52:58,914 INFO scheduler testdev-02-s11p2 log match: searching for "Leaving Scheduling Cycle"... OK 2017-08-10 09:52:58,929 INFO =============================== 2017-08-10 09:52:58,929 INFO Entered TestFairshare tearDown 2017-08-10 09:52:58,930 INFO =============================== 2017-08-10 09:52:58,930 INFO ================================ 2017-08-10 09:52:58,930 INFO Completed TestFairshare tearDown 2017-08-10 09:52:58,930 INFO ================================ 2017-08-10 09:52:58,931 INFO ok 2017-08-10 09:52:58,932 INFO test name: test_fairshare_formula6 (tests.functional.pbs_fairshare.TestFairshare)... 2017-08-10 09:52:58,932 INFO test start time: Thu Aug 10 09:52:58 2017 2017-08-10 09:52:58,932 INFO test docstring: Test fairshare in the formula. Make sure entities with small usage are negatively affected by their high usage siblings. Make sure that jobs run in the correct order. Use keyword fairshare_factor with ncpus/walltime 2017-08-10 09:52:58,932 INFO ============================ 2017-08-10 09:52:58,933 INFO Entered TestFairshare setUp 2017-08-10 09:52:58,933 INFO ============================ 2017-08-10 09:52:58,933 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -Bf testdev-02-s11p2.pbspro.com 2017-08-10 09:52:58,950 INFO status on testdev-02-s11p2: server 2017-08-10 09:52:58,951 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -Bf testdev-02-s11p2.pbspro.com 2017-08-10 09:52:58,967 INFO manager on testdev-02-s11p2: set server {'managers': (3, 'anamika@*')} 2017-08-10 09:52:58,968 INFOCLI testdev-02-s11p2: sudo -H /opt/pbs/bin/qmgr -c set server managers-=anamika@* 2017-08-10 09:52:59,031 INFO manager on testdev-02-s11p2: set server {'managers': (2, 'anamika@*')} 2017-08-10 09:52:59,032 INFOCLI testdev-02-s11p2: sudo -H /opt/pbs/bin/qmgr -c set server managers+=anamika@* 2017-08-10 09:52:59,091 INFO server testdev-02-s11p2: reverting configuration to defaults 2017-08-10 09:52:59,092 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -Bf testdev-02-s11p2.pbspro.com 2017-08-10 09:52:59,110 INFO select on testdev-02-s11p2: __ALL__ 2017-08-10 09:52:59,110 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qselect 2017-08-10 09:52:59,126 INFO delete job on testdev-02-s11p2: 201.testdev-02-s11p2, 202.testdev-02-s11p2, 203.testdev-02-s11p2, 204.testdev-02-s11p2 2017-08-10 09:52:59,126 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qdel -W force 201.testdev-02-s11p2 202.testdev-02-s11p2 203.testdev-02-s11p2 204.testdev-02-s11p2 2017-08-10 09:52:59,904 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -f 201.testdev-02-s11p2 2017-08-10 09:52:59,925 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -f 202.testdev-02-s11p2 2017-08-10 09:52:59,943 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -f 203.testdev-02-s11p2 2017-08-10 09:52:59,961 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -f 204.testdev-02-s11p2 2017-08-10 09:52:59,978 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -f @testdev-02-s11p2.pbspro.com 2017-08-10 09:52:59,992 INFO expect on server testdev-02-s11p2: job_state set 0 job ... OK 2017-08-10 09:52:59,992 INFOCLI testdev-02-s11p2: /opt/pbs/bin/pbs_rstat -f 2017-08-10 09:53:00,005 INFO manager on testdev-02-s11p2: unset server ['job_sort_formula'] 2017-08-10 09:53:00,006 INFOCLI testdev-02-s11p2: sudo -H /opt/pbs/bin/qmgr -c unset server job_sort_formula 2017-08-10 09:53:00,192 INFOCLI testdev-02-s11p2: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-08-10 09:53:00,213 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -Qf @testdev-02-s11p2.pbspro.com 2017-08-10 09:53:00,228 INFO status on testdev-02-s11p2: node 2017-08-10 09:53:00,228 INFOCLI testdev-02-s11p2: /opt/pbs/bin/pbsnodes -s testdev-02-s11p2.pbspro.com -v -a 2017-08-10 09:53:00,242 INFO manager on testdev-02-s11p2: delete queue workq 2017-08-10 09:53:00,242 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c delete queue workq 2017-08-10 09:53:00,341 INFO server testdev-02-s11p2: expect offset set to 0.5 2017-08-10 09:53:00,842 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -Qf workq@testdev-02-s11p2.pbspro.com 2017-08-10 09:53:00,857 INFO expect on server testdev-02-s11p2: unset queue workq ... OK 2017-08-10 09:53:00,858 INFO manager on testdev-02-s11p2: create queue workq {'started': 'True', 'queue_type': 'Execution', 'enabled': 'True'} 2017-08-10 09:53:00,858 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c create queue workq started=True,queue_type=Execution,enabled=True 2017-08-10 09:53:01,051 INFO server testdev-02-s11p2: expect offset set to 0.5 2017-08-10 09:53:01,552 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -Qf workq@testdev-02-s11p2.pbspro.com 2017-08-10 09:53:01,568 INFO expect on server testdev-02-s11p2: started set True || queue_type set Execution || enabled set True queue workq ... OK 2017-08-10 09:53:01,568 INFO manager on testdev-02-s11p2: set server {'default_queue': 'workq'} 2017-08-10 09:53:01,568 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c set server default_queue=workq 2017-08-10 09:53:01,721 INFO status on testdev-02-s11p2: resource 2017-08-10 09:53:01,721 INFO manager on testdev-02-s11p2: list resource 2017-08-10 09:53:01,722 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c list resource 2017-08-10 09:53:01,737 INFOCLI status on testdev-02-s11p2: server license_count 2017-08-10 09:53:01,737 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -Bf testdev-02-s11p2.pbspro.com 2017-08-10 09:53:01,751 INFO server: testdev-02-s11p2.pbspro.com licensed 2017-08-10 09:53:01,763 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/server_priv/comm.lock 2017-08-10 09:53:01,785 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/sched_priv/sched.lock 2017-08-10 09:53:01,796 INFO scheduler testdev-02-s11p2: reverting configuration to defaults 2017-08-10 09:53:01,797 INFO manager on testdev-02-s11p2: delete sched @default 2017-08-10 09:53:01,797 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c delete sched @default 2017-08-10 09:53:01,811 INFO manager on testdev-02-s11p2: list sched 2017-08-10 09:53:01,811 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c list sched 2017-08-10 09:53:01,825 INFO manager on testdev-02-s11p2: unset sched ['sched_cycle_length'] 2017-08-10 09:53:01,825 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c unset sched sched_cycle_length 2017-08-10 09:53:01,921 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/sched_priv/dedicated_time 2017-08-10 09:53:01,933 INFOCLI2 testdev-02-s11p2: cmp /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:53:01,938 ERROR err: ['cmp: /var/spool/pbs/sched_priv/resource_group: Permission denied'] 2017-08-10 09:53:01,939 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:53:01,951 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:53:01,962 INFO scheduler testdev-02-s11p2: reverting holidays file to default 2017-08-10 09:53:01,963 INFOCLI2 testdev-02-s11p2: cmp /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-08-10 09:53:01,968 ERROR err: ['cmp: /var/spool/pbs/sched_priv/holidays: Permission denied'] 2017-08-10 09:53:01,968 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-08-10 09:53:01,980 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/holidays 2017-08-10 09:53:01,992 INFO scheduler testdev-02-s11p2: Parsing holidays file 2017-08-10 09:53:01,992 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/sched_priv/holidays 2017-08-10 09:53:02,004 INFOCLI2 testdev-02-s11p2: cmp /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-08-10 09:53:02,009 ERROR err: ['cmp: /var/spool/pbs/sched_priv/sched_config: Permission denied'] 2017-08-10 09:53:02,009 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-08-10 09:53:02,021 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-08-10 09:53:02,033 INFO scheduler testdev-02-s11p2: sent signal -HUP 2017-08-10 09:53:02,033 INFOCLI2 testdev-02-s11p2: sudo -H kill -HUP 12179 2017-08-10 09:53:02,044 INFOCLI2 testdev-02-s11p2: sudo -H /opt/pbs/sbin/pbsfs -e 2017-08-10 09:53:02,084 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/sched_priv/sched.lock 2017-08-10 09:53:02,106 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/mom_priv/mom.lock 2017-08-10 09:53:02,118 INFO mom testdev-02-s11p2: reverting configuration to defaults 2017-08-10 09:53:02,118 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/rm -f /var/spool/pbs/mom_priv/epilogue 2017-08-10 09:53:02,130 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/rm -f /var/spool/pbs/mom_priv/prologue 2017-08-10 09:53:02,141 INFOCLI testdev-02-s11p2: sudo -H /opt/pbs/sbin/pbs_mom -s list 2017-08-10 09:53:02,160 INFO mom testdev-02-s11p2: sent signal -HUP 2017-08-10 09:53:02,160 INFOCLI2 testdev-02-s11p2: sudo -H kill -HUP 12176 2017-08-10 09:53:02,183 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/mom_priv/mom.lock 2017-08-10 09:53:02,195 INFOCLI testdev-02-s11p2: /opt/pbs/bin/pbsnodes -s testdev-02-s11p2.pbspro.com -v -a 2017-08-10 09:53:02,209 INFO status on testdev-02-s11p2: node testdev-02-s11p2 2017-08-10 09:53:02,209 INFOCLI testdev-02-s11p2: /opt/pbs/bin/pbsnodes -s testdev-02-s11p2.pbspro.com -v testdev-02-s11p2 2017-08-10 09:53:02,223 INFOCLI testdev-02-s11p2: /opt/pbs/bin/pbsnodes -s testdev-02-s11p2.pbspro.com -v testdev-02-s11p2 2017-08-10 09:53:02,237 INFO expect on server testdev-02-s11p2: state = free node testdev-02-s11p2 ... OK 2017-08-10 09:53:02,238 INFO ============================= 2017-08-10 09:53:02,238 INFO Completed TestFairshare setUp 2017-08-10 09:53:02,238 INFO ============================= 2017-08-10 09:53:02,238 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:53:02,250 INFO creating tree node: group1 2017-08-10 09:53:02,251 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /tmp/PtlPbsQYpRCy /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:53:02,263 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:53:02,274 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chown root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:53:02,286 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chgrp root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:53:02,298 INFO creating tree node: group2 2017-08-10 09:53:02,298 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /tmp/PtlPbsn5rTth /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:53:02,311 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:53:02,322 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chown root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:53:02,334 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chgrp root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:53:02,345 INFO creating tree node: pbsuser 2017-08-10 09:53:02,346 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /tmp/PtlPbsISrOq4 /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:53:02,358 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:53:02,370 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chown root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:53:02,381 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chgrp root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:53:02,393 INFO creating tree node: pbsuser1 2017-08-10 09:53:02,394 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /tmp/PtlPbsiPmgUA /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:53:02,408 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:53:02,420 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chown root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:53:02,432 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chgrp root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:53:02,443 INFO creating tree node: pbsuser2 2017-08-10 09:53:02,444 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /tmp/PtlPbsm4nYHf /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:53:02,456 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:53:02,468 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chown root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:53:02,480 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chgrp root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:53:02,493 INFO creating tree node: pbsuser3 2017-08-10 09:53:02,494 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /tmp/PtlPbs16l1DQ /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:53:02,506 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:53:02,519 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chown root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:53:02,531 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chgrp root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:53:02,542 INFOCLI2 testdev-02-s11p2: sudo -H /opt/pbs/sbin/pbsfs -s pbsuser 100 2017-08-10 09:53:02,559 INFOCLI2 testdev-02-s11p2: sudo -H /opt/pbs/sbin/pbsfs -s pbsuser1 100 2017-08-10 09:53:02,575 INFOCLI2 testdev-02-s11p2: sudo -H /opt/pbs/sbin/pbsfs -s pbsuser3 1000 2017-08-10 09:53:02,591 INFO scheduler testdev-02-s11p2: config {'log_filter': 2048} 2017-08-10 09:53:02,592 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /var/spool/pbs/sched_priv/sched_config /var/spool/pbs/sched_priv/sched_config.bak 2017-08-10 09:53:02,605 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /tmp/PtlPbsYupIwZ /var/spool/pbs/sched_priv/sched_config 2017-08-10 09:53:02,617 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-08-10 09:53:02,629 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chown root /var/spool/pbs/sched_priv/sched_config 2017-08-10 09:53:02,641 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chgrp root /var/spool/pbs/sched_priv/sched_config 2017-08-10 09:53:02,652 INFO scheduler testdev-02-s11p2: sent signal -HUP 2017-08-10 09:53:02,653 INFOCLI2 testdev-02-s11p2: sudo -H kill -HUP 12179 2017-08-10 09:53:02,664 INFO scheduler testdev-02-s11p2 log match: searching for "Error reading line" - No match 2017-08-10 09:53:03,666 INFO manager on testdev-02-s11p2: set server {'scheduling': 'False'} 2017-08-10 09:53:03,666 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c set server scheduling=False 2017-08-10 09:53:03,751 INFO manager on testdev-02-s11p2: set server {'job_sort_formula': '"fairshare_factor + (walltime/ncpus)"'} 2017-08-10 09:53:03,751 INFOCLI testdev-02-s11p2: sudo -H /opt/pbs/bin/qmgr -c set server job_sort_formula="fairshare_factor + (walltime/ncpus)" 2017-08-10 09:53:03,842 INFO job: executable set to /bin/sleep with arguments: 100 2017-08-10 09:53:03,843 INFOCLI testdev-02-s11p2: sudo -H -u pbsuser2 /opt/pbs/bin/qsub -l ncpus=1 -l walltime=00:01:00 -- /bin/sleep 100 2017-08-10 09:53:03,891 INFO submit to testdev-02-s11p2 as pbsuser2: job 205.testdev-02-s11p2 OrderedDict([('Resource_List.ncpus', 1), ('Resource_List.walltime', '00:01:00')]) 2017-08-10 09:53:03,892 INFO job: executable set to /bin/sleep with arguments: 100 2017-08-10 09:53:03,892 INFOCLI testdev-02-s11p2: sudo -H -u pbsuser3 /opt/pbs/bin/qsub -l ncpus=2 -l walltime=00:01:00 -- /bin/sleep 100 2017-08-10 09:53:03,941 INFO submit to testdev-02-s11p2 as pbsuser3: job 206.testdev-02-s11p2 OrderedDict([('Resource_List.ncpus', 2), ('Resource_List.walltime', '00:01:00')]) 2017-08-10 09:53:03,942 INFO job: executable set to /bin/sleep with arguments: 100 2017-08-10 09:53:03,942 INFOCLI testdev-02-s11p2: sudo -H -u pbsuser /opt/pbs/bin/qsub -l ncpus=3 -l walltime=00:02:00 -- /bin/sleep 100 2017-08-10 09:53:03,991 INFO submit to testdev-02-s11p2 as pbsuser: job 207.testdev-02-s11p2 OrderedDict([('Resource_List.ncpus', 3), ('Resource_List.walltime', '00:02:00')]) 2017-08-10 09:53:03,992 INFO job: executable set to /bin/sleep with arguments: 100 2017-08-10 09:53:03,992 INFOCLI testdev-02-s11p2: sudo -H -u pbsuser1 /opt/pbs/bin/qsub -l ncpus=4 -l walltime=00:02:00 -- /bin/sleep 100 2017-08-10 09:53:04,071 INFO submit to testdev-02-s11p2 as pbsuser1: job 208.testdev-02-s11p2 OrderedDict([('Resource_List.ncpus', 4), ('Resource_List.walltime', '00:02:00')]) 2017-08-10 09:53:04,072 INFO manager on testdev-02-s11p2: set server {'scheduling': 'True'} 2017-08-10 09:53:04,072 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-08-10 09:53:04,121 INFO scheduler testdev-02-s11p2 log match: searching for "205.testdev-02-s11p2;Formula Evaluation = 60.3816" - No match 2017-08-10 09:53:05,123 INFO scheduler testdev-02-s11p2 log match: searching for "205.testdev-02-s11p2;Formula Evaluation = 60.3816"... OK 2017-08-10 09:53:05,124 INFO scheduler testdev-02-s11p2 log match: searching for "206.testdev-02-s11p2;Formula Evaluation = 30.0902"... OK 2017-08-10 09:53:05,124 INFO scheduler testdev-02-s11p2 log match: searching for "207.testdev-02-s11p2;Formula Evaluation = 40.6477"... OK 2017-08-10 09:53:05,124 INFO scheduler testdev-02-s11p2 log match: searching for "208.testdev-02-s11p2;Formula Evaluation = 30.6477"... OK 2017-08-10 09:53:05,124 INFO scheduler testdev-02-s11p2 log match: searching for "Leaving Scheduling Cycle"... OK 2017-08-10 09:53:05,141 INFO =============================== 2017-08-10 09:53:05,141 INFO Entered TestFairshare tearDown 2017-08-10 09:53:05,142 INFO =============================== 2017-08-10 09:53:05,142 INFO ================================ 2017-08-10 09:53:05,142 INFO Completed TestFairshare tearDown 2017-08-10 09:53:05,142 INFO ================================ 2017-08-10 09:53:05,143 INFO ok 2017-08-10 09:53:05,144 INFO test name: test_formula_keyword (tests.functional.pbs_fairshare.TestFairshare)... 2017-08-10 09:53:05,144 INFO test start time: Thu Aug 10 09:53:05 2017 2017-08-10 09:53:05,144 INFO test docstring: Test to see if 'fairshare_tree_usage' and 'fairshare_perc' are allowed to be set in the job_sort_formula 2017-08-10 09:53:05,144 INFO ============================ 2017-08-10 09:53:05,144 INFO Entered TestFairshare setUp 2017-08-10 09:53:05,144 INFO ============================ 2017-08-10 09:53:05,145 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -Bf testdev-02-s11p2.pbspro.com 2017-08-10 09:53:05,161 INFO status on testdev-02-s11p2: server 2017-08-10 09:53:05,162 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -Bf testdev-02-s11p2.pbspro.com 2017-08-10 09:53:05,176 INFO manager on testdev-02-s11p2: set server {'managers': (3, 'anamika@*')} 2017-08-10 09:53:05,176 INFOCLI testdev-02-s11p2: sudo -H /opt/pbs/bin/qmgr -c set server managers-=anamika@* 2017-08-10 09:53:05,281 INFO manager on testdev-02-s11p2: set server {'managers': (2, 'anamika@*')} 2017-08-10 09:53:05,282 INFOCLI testdev-02-s11p2: sudo -H /opt/pbs/bin/qmgr -c set server managers+=anamika@* 2017-08-10 09:53:05,351 INFO server testdev-02-s11p2: reverting configuration to defaults 2017-08-10 09:53:05,352 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -Bf testdev-02-s11p2.pbspro.com 2017-08-10 09:53:05,366 INFO select on testdev-02-s11p2: __ALL__ 2017-08-10 09:53:05,367 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qselect 2017-08-10 09:53:05,380 INFO delete job on testdev-02-s11p2: 205.testdev-02-s11p2, 206.testdev-02-s11p2, 207.testdev-02-s11p2, 208.testdev-02-s11p2 2017-08-10 09:53:05,381 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qdel -W force 205.testdev-02-s11p2 206.testdev-02-s11p2 207.testdev-02-s11p2 208.testdev-02-s11p2 2017-08-10 09:53:06,063 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -f 205.testdev-02-s11p2 2017-08-10 09:53:06,084 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -f 206.testdev-02-s11p2 2017-08-10 09:53:06,103 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -f 207.testdev-02-s11p2 2017-08-10 09:53:06,121 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -f 208.testdev-02-s11p2 2017-08-10 09:53:06,138 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -f @testdev-02-s11p2.pbspro.com 2017-08-10 09:53:06,152 INFO expect on server testdev-02-s11p2: job_state set 0 job ... OK 2017-08-10 09:53:06,152 INFOCLI testdev-02-s11p2: /opt/pbs/bin/pbs_rstat -f 2017-08-10 09:53:06,165 INFO manager on testdev-02-s11p2: unset server ['job_sort_formula'] 2017-08-10 09:53:06,166 INFOCLI testdev-02-s11p2: sudo -H /opt/pbs/bin/qmgr -c unset server job_sort_formula 2017-08-10 09:53:06,362 INFOCLI testdev-02-s11p2: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-08-10 09:53:06,384 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -Qf @testdev-02-s11p2.pbspro.com 2017-08-10 09:53:06,398 INFO status on testdev-02-s11p2: node 2017-08-10 09:53:06,399 INFOCLI testdev-02-s11p2: /opt/pbs/bin/pbsnodes -s testdev-02-s11p2.pbspro.com -v -a 2017-08-10 09:53:06,413 INFO manager on testdev-02-s11p2: delete queue workq 2017-08-10 09:53:06,413 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c delete queue workq 2017-08-10 09:53:06,511 INFO server testdev-02-s11p2: expect offset set to 0.5 2017-08-10 09:53:07,012 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -Qf workq@testdev-02-s11p2.pbspro.com 2017-08-10 09:53:07,028 INFO expect on server testdev-02-s11p2: unset queue workq ... OK 2017-08-10 09:53:07,028 INFO manager on testdev-02-s11p2: create queue workq {'started': 'True', 'queue_type': 'Execution', 'enabled': 'True'} 2017-08-10 09:53:07,028 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c create queue workq started=True,queue_type=Execution,enabled=True 2017-08-10 09:53:07,281 INFO server testdev-02-s11p2: expect offset set to 0.5 2017-08-10 09:53:07,783 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -Qf workq@testdev-02-s11p2.pbspro.com 2017-08-10 09:53:07,798 INFO expect on server testdev-02-s11p2: started set True || queue_type set Execution || enabled set True queue workq ... OK 2017-08-10 09:53:07,799 INFO manager on testdev-02-s11p2: set server {'default_queue': 'workq'} 2017-08-10 09:53:07,799 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c set server default_queue=workq 2017-08-10 09:53:07,931 INFO status on testdev-02-s11p2: resource 2017-08-10 09:53:07,932 INFO manager on testdev-02-s11p2: list resource 2017-08-10 09:53:07,932 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c list resource 2017-08-10 09:53:07,947 INFOCLI status on testdev-02-s11p2: server license_count 2017-08-10 09:53:07,948 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -Bf testdev-02-s11p2.pbspro.com 2017-08-10 09:53:07,961 INFO server: testdev-02-s11p2.pbspro.com licensed 2017-08-10 09:53:07,973 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/server_priv/comm.lock 2017-08-10 09:53:07,995 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/sched_priv/sched.lock 2017-08-10 09:53:08,007 INFO scheduler testdev-02-s11p2: reverting configuration to defaults 2017-08-10 09:53:08,007 INFO manager on testdev-02-s11p2: delete sched @default 2017-08-10 09:53:08,007 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c delete sched @default 2017-08-10 09:53:08,022 INFO manager on testdev-02-s11p2: list sched 2017-08-10 09:53:08,023 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c list sched 2017-08-10 09:53:08,038 INFO manager on testdev-02-s11p2: unset sched ['sched_cycle_length'] 2017-08-10 09:53:08,038 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c unset sched sched_cycle_length 2017-08-10 09:53:08,151 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/sched_priv/dedicated_time 2017-08-10 09:53:08,164 INFOCLI2 testdev-02-s11p2: cmp /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:53:08,169 ERROR err: ['cmp: /var/spool/pbs/sched_priv/resource_group: Permission denied'] 2017-08-10 09:53:08,170 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:53:08,182 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:53:08,194 INFO scheduler testdev-02-s11p2: reverting holidays file to default 2017-08-10 09:53:08,194 INFOCLI2 testdev-02-s11p2: cmp /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-08-10 09:53:08,199 ERROR err: ['cmp: /var/spool/pbs/sched_priv/holidays: Permission denied'] 2017-08-10 09:53:08,199 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-08-10 09:53:08,211 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/holidays 2017-08-10 09:53:08,223 INFO scheduler testdev-02-s11p2: Parsing holidays file 2017-08-10 09:53:08,223 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/sched_priv/holidays 2017-08-10 09:53:08,235 INFOCLI2 testdev-02-s11p2: cmp /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-08-10 09:53:08,240 ERROR err: ['cmp: /var/spool/pbs/sched_priv/sched_config: Permission denied'] 2017-08-10 09:53:08,241 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-08-10 09:53:08,252 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-08-10 09:53:08,264 INFO scheduler testdev-02-s11p2: sent signal -HUP 2017-08-10 09:53:08,264 INFOCLI2 testdev-02-s11p2: sudo -H kill -HUP 12179 2017-08-10 09:53:08,275 INFOCLI2 testdev-02-s11p2: sudo -H /opt/pbs/sbin/pbsfs -e 2017-08-10 09:53:08,315 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/sched_priv/sched.lock 2017-08-10 09:53:08,336 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/mom_priv/mom.lock 2017-08-10 09:53:08,348 INFO mom testdev-02-s11p2: reverting configuration to defaults 2017-08-10 09:53:08,349 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/rm -f /var/spool/pbs/mom_priv/epilogue 2017-08-10 09:53:08,361 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/rm -f /var/spool/pbs/mom_priv/prologue 2017-08-10 09:53:08,373 INFOCLI testdev-02-s11p2: sudo -H /opt/pbs/sbin/pbs_mom -s list 2017-08-10 09:53:08,392 INFO mom testdev-02-s11p2: sent signal -HUP 2017-08-10 09:53:08,392 INFOCLI2 testdev-02-s11p2: sudo -H kill -HUP 12176 2017-08-10 09:53:08,415 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/mom_priv/mom.lock 2017-08-10 09:53:08,427 INFOCLI testdev-02-s11p2: /opt/pbs/bin/pbsnodes -s testdev-02-s11p2.pbspro.com -v -a 2017-08-10 09:53:08,441 INFO status on testdev-02-s11p2: node testdev-02-s11p2 2017-08-10 09:53:08,441 INFOCLI testdev-02-s11p2: /opt/pbs/bin/pbsnodes -s testdev-02-s11p2.pbspro.com -v testdev-02-s11p2 2017-08-10 09:53:08,456 INFOCLI testdev-02-s11p2: /opt/pbs/bin/pbsnodes -s testdev-02-s11p2.pbspro.com -v testdev-02-s11p2 2017-08-10 09:53:08,469 INFO expect on server testdev-02-s11p2: state = free node testdev-02-s11p2 ... OK 2017-08-10 09:53:08,470 INFO ============================= 2017-08-10 09:53:08,470 INFO Completed TestFairshare setUp 2017-08-10 09:53:08,470 INFO ============================= 2017-08-10 09:53:08,470 INFO manager on testdev-02-s11p2: set server {'job_sort_formula': 'fairshare_tree_usage'} 2017-08-10 09:53:08,470 INFOCLI testdev-02-s11p2: sudo -H /opt/pbs/bin/qmgr -c set server job_sort_formula=fairshare_tree_usage 2017-08-10 09:53:09,772 INFO manager on testdev-02-s11p2: set server {'job_sort_formula': 'fairshare_perc'} 2017-08-10 09:53:09,773 INFOCLI testdev-02-s11p2: sudo -H /opt/pbs/bin/qmgr -c set server job_sort_formula=fairshare_perc 2017-08-10 09:53:09,862 INFO manager on testdev-02-s11p2: set server {'job_sort_formula': '"pow(2,-(fairshare_tree_usage/fairshare_perc))"'} 2017-08-10 09:53:09,862 INFOCLI testdev-02-s11p2: sudo -H /opt/pbs/bin/qmgr -c set server job_sort_formula="pow(2,-(fairshare_tree_usage/fairshare_perc))" 2017-08-10 09:53:09,942 INFO manager on testdev-02-s11p2: set server {'job_sort_formula': 'fairshare_factor'} 2017-08-10 09:53:09,942 INFOCLI testdev-02-s11p2: sudo -H /opt/pbs/bin/qmgr -c set server job_sort_formula=fairshare_factor 2017-08-10 09:53:10,065 INFO manager on testdev-02-s11p2: set server {'job_sort_formula': 'fair_share_factor'} 2017-08-10 09:53:10,065 INFOCLI testdev-02-s11p2: sudo -H /opt/pbs/bin/qmgr -c set server job_sort_formula=fair_share_factor 2017-08-10 09:53:10,087 ERROR err: ['qmgr obj= svr=default: Formula contains invalid keyword', 'qmgr: Error (15124) returned from server'] 2017-08-10 09:53:10,088 INFO =============================== 2017-08-10 09:53:10,088 INFO Entered TestFairshare tearDown 2017-08-10 09:53:10,088 INFO =============================== 2017-08-10 09:53:10,088 INFO ================================ 2017-08-10 09:53:10,088 INFO Completed TestFairshare tearDown 2017-08-10 09:53:10,089 INFO ================================ 2017-08-10 09:53:10,090 INFO ok 2017-08-10 09:53:10,090 INFO test name: test_pbsfs (tests.functional.pbs_fairshare.TestFairshare)... 2017-08-10 09:53:10,090 INFO test start time: Thu Aug 10 09:53:10 2017 2017-08-10 09:53:10,091 INFO test docstring: Test to see if running pbsfs affects the scheduler's view of the fairshare usage. This is done by calling the Scheduler()'s revert_to_defaults(). This will call pbsfs -e to remove all usage. 2017-08-10 09:53:10,091 INFO ============================ 2017-08-10 09:53:10,091 INFO Entered TestFairshare setUp 2017-08-10 09:53:10,091 INFO ============================ 2017-08-10 09:53:10,091 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -Bf testdev-02-s11p2.pbspro.com 2017-08-10 09:53:10,107 INFO status on testdev-02-s11p2: server 2017-08-10 09:53:10,107 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -Bf testdev-02-s11p2.pbspro.com 2017-08-10 09:53:10,121 INFO manager on testdev-02-s11p2: set server {'managers': (3, 'anamika@*')} 2017-08-10 09:53:10,121 INFOCLI testdev-02-s11p2: sudo -H /opt/pbs/bin/qmgr -c set server managers-=anamika@* 2017-08-10 09:53:10,182 INFO manager on testdev-02-s11p2: set server {'managers': (2, 'anamika@*')} 2017-08-10 09:53:10,182 INFOCLI testdev-02-s11p2: sudo -H /opt/pbs/bin/qmgr -c set server managers+=anamika@* 2017-08-10 09:53:10,332 INFO server testdev-02-s11p2: reverting configuration to defaults 2017-08-10 09:53:10,332 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -Bf testdev-02-s11p2.pbspro.com 2017-08-10 09:53:10,347 INFO select on testdev-02-s11p2: __ALL__ 2017-08-10 09:53:10,347 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qselect 2017-08-10 09:53:10,361 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -f @testdev-02-s11p2.pbspro.com 2017-08-10 09:53:10,374 INFO expect on server testdev-02-s11p2: job_state set 0 job ... OK 2017-08-10 09:53:10,375 INFOCLI testdev-02-s11p2: /opt/pbs/bin/pbs_rstat -f 2017-08-10 09:53:10,388 INFO manager on testdev-02-s11p2: unset server ['job_sort_formula'] 2017-08-10 09:53:10,388 INFOCLI testdev-02-s11p2: sudo -H /opt/pbs/bin/qmgr -c unset server job_sort_formula 2017-08-10 09:53:10,512 INFOCLI testdev-02-s11p2: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-08-10 09:53:10,533 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -Qf @testdev-02-s11p2.pbspro.com 2017-08-10 09:53:10,547 INFO status on testdev-02-s11p2: node 2017-08-10 09:53:10,547 INFOCLI testdev-02-s11p2: /opt/pbs/bin/pbsnodes -s testdev-02-s11p2.pbspro.com -v -a 2017-08-10 09:53:10,561 INFO manager on testdev-02-s11p2: delete queue workq 2017-08-10 09:53:10,561 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c delete queue workq 2017-08-10 09:53:10,661 INFO server testdev-02-s11p2: expect offset set to 0.5 2017-08-10 09:53:11,162 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -Qf workq@testdev-02-s11p2.pbspro.com 2017-08-10 09:53:11,178 INFO expect on server testdev-02-s11p2: unset queue workq ... OK 2017-08-10 09:53:11,178 INFO manager on testdev-02-s11p2: create queue workq {'started': 'True', 'queue_type': 'Execution', 'enabled': 'True'} 2017-08-10 09:53:11,179 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c create queue workq started=True,queue_type=Execution,enabled=True 2017-08-10 09:53:11,441 INFO server testdev-02-s11p2: expect offset set to 0.5 2017-08-10 09:53:11,943 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -Qf workq@testdev-02-s11p2.pbspro.com 2017-08-10 09:53:11,959 INFO expect on server testdev-02-s11p2: started set True || queue_type set Execution || enabled set True queue workq ... OK 2017-08-10 09:53:11,959 INFO manager on testdev-02-s11p2: set server {'default_queue': 'workq'} 2017-08-10 09:53:11,959 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c set server default_queue=workq 2017-08-10 09:53:12,071 INFO status on testdev-02-s11p2: resource 2017-08-10 09:53:12,072 INFO manager on testdev-02-s11p2: list resource 2017-08-10 09:53:12,072 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c list resource 2017-08-10 09:53:12,087 INFOCLI status on testdev-02-s11p2: server license_count 2017-08-10 09:53:12,087 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -Bf testdev-02-s11p2.pbspro.com 2017-08-10 09:53:12,101 INFO server: testdev-02-s11p2.pbspro.com licensed 2017-08-10 09:53:12,112 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/server_priv/comm.lock 2017-08-10 09:53:12,134 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/sched_priv/sched.lock 2017-08-10 09:53:12,145 INFO scheduler testdev-02-s11p2: reverting configuration to defaults 2017-08-10 09:53:12,146 INFO manager on testdev-02-s11p2: delete sched @default 2017-08-10 09:53:12,146 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c delete sched @default 2017-08-10 09:53:12,159 INFO manager on testdev-02-s11p2: list sched 2017-08-10 09:53:12,160 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c list sched 2017-08-10 09:53:12,173 INFO manager on testdev-02-s11p2: unset sched ['sched_cycle_length'] 2017-08-10 09:53:12,174 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c unset sched sched_cycle_length 2017-08-10 09:53:12,261 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/sched_priv/dedicated_time 2017-08-10 09:53:12,274 INFOCLI2 testdev-02-s11p2: cmp /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:53:12,279 ERROR err: ['cmp: /var/spool/pbs/sched_priv/resource_group: Permission denied'] 2017-08-10 09:53:12,279 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:53:12,291 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:53:12,303 INFO scheduler testdev-02-s11p2: reverting holidays file to default 2017-08-10 09:53:12,303 INFOCLI2 testdev-02-s11p2: cmp /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-08-10 09:53:12,308 ERROR err: ['cmp: /var/spool/pbs/sched_priv/holidays: Permission denied'] 2017-08-10 09:53:12,309 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-08-10 09:53:12,321 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/holidays 2017-08-10 09:53:12,332 INFO scheduler testdev-02-s11p2: Parsing holidays file 2017-08-10 09:53:12,332 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/sched_priv/holidays 2017-08-10 09:53:12,344 INFOCLI2 testdev-02-s11p2: cmp /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-08-10 09:53:12,349 ERROR err: ['cmp: /var/spool/pbs/sched_priv/sched_config: Permission denied'] 2017-08-10 09:53:12,349 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-08-10 09:53:12,361 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-08-10 09:53:12,374 INFO scheduler testdev-02-s11p2: sent signal -HUP 2017-08-10 09:53:12,374 INFOCLI2 testdev-02-s11p2: sudo -H kill -HUP 12179 2017-08-10 09:53:12,385 INFOCLI2 testdev-02-s11p2: sudo -H /opt/pbs/sbin/pbsfs -e 2017-08-10 09:53:12,429 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/sched_priv/sched.lock 2017-08-10 09:53:12,451 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/mom_priv/mom.lock 2017-08-10 09:53:12,463 INFO mom testdev-02-s11p2: reverting configuration to defaults 2017-08-10 09:53:12,463 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/rm -f /var/spool/pbs/mom_priv/epilogue 2017-08-10 09:53:12,475 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/rm -f /var/spool/pbs/mom_priv/prologue 2017-08-10 09:53:12,486 INFOCLI testdev-02-s11p2: sudo -H /opt/pbs/sbin/pbs_mom -s list 2017-08-10 09:53:12,505 INFO mom testdev-02-s11p2: sent signal -HUP 2017-08-10 09:53:12,506 INFOCLI2 testdev-02-s11p2: sudo -H kill -HUP 12176 2017-08-10 09:53:12,529 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/mom_priv/mom.lock 2017-08-10 09:53:12,541 INFOCLI testdev-02-s11p2: /opt/pbs/bin/pbsnodes -s testdev-02-s11p2.pbspro.com -v -a 2017-08-10 09:53:12,555 INFO status on testdev-02-s11p2: node testdev-02-s11p2 2017-08-10 09:53:12,556 INFOCLI testdev-02-s11p2: /opt/pbs/bin/pbsnodes -s testdev-02-s11p2.pbspro.com -v testdev-02-s11p2 2017-08-10 09:53:12,570 INFOCLI testdev-02-s11p2: /opt/pbs/bin/pbsnodes -s testdev-02-s11p2.pbspro.com -v testdev-02-s11p2 2017-08-10 09:53:12,583 INFO expect on server testdev-02-s11p2: state = free node testdev-02-s11p2 ... OK 2017-08-10 09:53:12,584 INFO ============================= 2017-08-10 09:53:12,584 INFO Completed TestFairshare setUp 2017-08-10 09:53:12,584 INFO ============================= 2017-08-10 09:53:12,584 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:53:12,596 INFO creating tree node: pbsuser 2017-08-10 09:53:12,597 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /tmp/PtlPbsaI8TCO /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:53:12,609 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:53:12,621 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chown root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:53:12,632 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chgrp root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:53:12,644 INFO creating tree node: pbsuser1 2017-08-10 09:53:12,645 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /tmp/PtlPbs52Essa /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:53:12,656 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:53:12,668 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chown root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:53:12,680 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chgrp root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:53:12,691 INFO scheduler testdev-02-s11p2: config {'fair_share': 'True'} 2017-08-10 09:53:12,692 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /var/spool/pbs/sched_priv/sched_config /var/spool/pbs/sched_priv/sched_config.bak 2017-08-10 09:53:12,705 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /tmp/PtlPbsWSxeth /var/spool/pbs/sched_priv/sched_config 2017-08-10 09:53:12,717 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-08-10 09:53:12,729 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chown root /var/spool/pbs/sched_priv/sched_config 2017-08-10 09:53:12,740 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chgrp root /var/spool/pbs/sched_priv/sched_config 2017-08-10 09:53:12,752 INFO scheduler testdev-02-s11p2: sent signal -HUP 2017-08-10 09:53:12,752 INFOCLI2 testdev-02-s11p2: sudo -H kill -HUP 12179 2017-08-10 09:53:12,763 INFO scheduler testdev-02-s11p2 log match: searching for "Error reading line" - No match 2017-08-10 09:53:13,765 INFOCLI2 testdev-02-s11p2: sudo -H /opt/pbs/sbin/pbsfs -s pbsuser 100 2017-08-10 09:53:13,783 INFO manager on testdev-02-s11p2: set server {'scheduling': 'False'} 2017-08-10 09:53:13,784 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c set server scheduling=False 2017-08-10 09:53:14,472 INFO job: executable set to /bin/sleep with arguments: 100 2017-08-10 09:53:14,473 INFOCLI testdev-02-s11p2: sudo -H -u pbsuser /opt/pbs/bin/qsub -- /bin/sleep 100 2017-08-10 09:53:14,582 INFO submit to testdev-02-s11p2 as pbsuser: job 209.testdev-02-s11p2 2017-08-10 09:53:14,583 INFO job: executable set to /bin/sleep with arguments: 100 2017-08-10 09:53:14,583 INFOCLI testdev-02-s11p2: sudo -H -u pbsuser1 /opt/pbs/bin/qsub -- /bin/sleep 100 2017-08-10 09:53:14,672 INFO submit to testdev-02-s11p2 as pbsuser1: job 210.testdev-02-s11p2 2017-08-10 09:53:14,672 INFO manager on testdev-02-s11p2: set server {'scheduling': 'True'} 2017-08-10 09:53:14,673 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-08-10 09:53:14,743 INFO scheduler testdev-02-s11p2 log match: searching for "Leaving Scheduling Cycle" - No match 2017-08-10 09:53:15,759 INFO scheduler testdev-02-s11p2 log match: searching for "Leaving Scheduling Cycle"... OK 2017-08-10 09:53:15,790 INFO delete job on testdev-02-s11p2: 209.testdev-02-s11p2 2017-08-10 09:53:15,790 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qdel 209.testdev-02-s11p2 2017-08-10 09:53:15,952 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -f 209.testdev-02-s11p2 2017-08-10 09:53:15,979 INFO delete job on testdev-02-s11p2: 210.testdev-02-s11p2 2017-08-10 09:53:15,979 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qdel 210.testdev-02-s11p2 2017-08-10 09:53:16,146 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -f 210.testdev-02-s11p2 2017-08-10 09:53:16,175 INFO job: executable set to /bin/sleep with arguments: 100 2017-08-10 09:53:16,176 INFO expect on server testdev-02-s11p2: queue unset job 210.testdev-02-s11p2 got: queue = workq 2017-08-10 09:53:16,677 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-08-10 09:53:16,732 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -f 210.testdev-02-s11p2 2017-08-10 09:53:16,747 INFO expect on server testdev-02-s11p2: queue unset job 210.testdev-02-s11p2 attempt: 2 got: queue = workq 2017-08-10 09:53:17,248 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-08-10 09:53:17,322 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -f 210.testdev-02-s11p2 2017-08-10 09:53:17,337 INFO expect on server testdev-02-s11p2: queue unset job 210.testdev-02-s11p2 attempt: 3 got: queue = workq 2017-08-10 09:53:17,838 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-08-10 09:53:17,972 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -f 210.testdev-02-s11p2 2017-08-10 09:53:17,988 INFO expect on server testdev-02-s11p2: queue unset job 210.testdev-02-s11p2 attempt: 4 got: queue = workq 2017-08-10 09:53:18,489 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-08-10 09:53:18,572 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qstat -f 210.testdev-02-s11p2 2017-08-10 09:53:18,593 INFO scheduler testdev-02-s11p2: reverting configuration to defaults 2017-08-10 09:53:18,593 INFO manager on testdev-02-s11p2: delete sched @default 2017-08-10 09:53:18,593 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c delete sched @default 2017-08-10 09:53:18,608 INFO manager on testdev-02-s11p2: list sched 2017-08-10 09:53:18,608 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c list sched 2017-08-10 09:53:18,623 INFO manager on testdev-02-s11p2: unset sched ['sched_cycle_length'] 2017-08-10 09:53:18,623 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c unset sched sched_cycle_length 2017-08-10 09:53:18,702 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/sched_priv/dedicated_time 2017-08-10 09:53:18,714 INFOCLI2 testdev-02-s11p2: cmp /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:53:18,719 ERROR err: ['cmp: /var/spool/pbs/sched_priv/resource_group: Permission denied'] 2017-08-10 09:53:18,720 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:53:18,732 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:53:18,744 INFO scheduler testdev-02-s11p2: reverting holidays file to default 2017-08-10 09:53:18,744 INFOCLI2 testdev-02-s11p2: cmp /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-08-10 09:53:18,749 ERROR err: ['cmp: /var/spool/pbs/sched_priv/holidays: Permission denied'] 2017-08-10 09:53:18,749 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-08-10 09:53:18,761 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/holidays 2017-08-10 09:53:18,773 INFO scheduler testdev-02-s11p2: Parsing holidays file 2017-08-10 09:53:18,773 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/sched_priv/holidays 2017-08-10 09:53:18,785 INFOCLI2 testdev-02-s11p2: cmp /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-08-10 09:53:18,789 ERROR err: ['cmp: /var/spool/pbs/sched_priv/sched_config: Permission denied'] 2017-08-10 09:53:18,790 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-08-10 09:53:18,802 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-08-10 09:53:18,813 INFO scheduler testdev-02-s11p2: sent signal -HUP 2017-08-10 09:53:18,814 INFOCLI2 testdev-02-s11p2: sudo -H kill -HUP 12179 2017-08-10 09:53:18,824 INFOCLI2 testdev-02-s11p2: sudo -H /opt/pbs/sbin/pbsfs -e 2017-08-10 09:53:18,865 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/sched_priv/sched.lock 2017-08-10 09:53:18,876 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cat /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:53:18,888 INFO creating tree node: pbsuser 2017-08-10 09:53:18,889 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /tmp/PtlPbszQjAUC /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:53:18,901 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:53:18,914 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chown root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:53:18,926 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chgrp root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:53:18,937 INFO creating tree node: pbsuser1 2017-08-10 09:53:18,938 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /tmp/PtlPbscMuWXi /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:53:18,950 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:53:18,962 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chown root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:53:18,973 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chgrp root /var/spool/pbs/sched_priv/resource_group 2017-08-10 09:53:18,985 INFO scheduler testdev-02-s11p2: config {'fair_share': 'True'} 2017-08-10 09:53:18,986 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /var/spool/pbs/sched_priv/sched_config /var/spool/pbs/sched_priv/sched_config.bak 2017-08-10 09:53:18,998 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/cp -p /tmp/PtlPbsruhfbn /var/spool/pbs/sched_priv/sched_config 2017-08-10 09:53:19,010 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-08-10 09:53:19,022 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chown root /var/spool/pbs/sched_priv/sched_config 2017-08-10 09:53:19,034 INFOCLI2 testdev-02-s11p2: sudo -H /opt/tools/wrappers/chgrp root /var/spool/pbs/sched_priv/sched_config 2017-08-10 09:53:19,045 INFO scheduler testdev-02-s11p2: sent signal -HUP 2017-08-10 09:53:19,045 INFOCLI2 testdev-02-s11p2: sudo -H kill -HUP 12179 2017-08-10 09:53:19,057 INFO scheduler testdev-02-s11p2 log match: searching for "Error reading line" - No match 2017-08-10 09:53:20,059 INFOCLI2 testdev-02-s11p2: sudo -H /opt/pbs/sbin/pbsfs -s pbsuser1 50 2017-08-10 09:53:20,077 INFO manager on testdev-02-s11p2: set server {'scheduling': 'False'} 2017-08-10 09:53:20,078 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c set server scheduling=False 2017-08-10 09:53:20,343 INFO job: executable set to /bin/sleep with arguments: 100 2017-08-10 09:53:20,343 INFOCLI testdev-02-s11p2: sudo -H -u pbsuser /opt/pbs/bin/qsub -- /bin/sleep 100 2017-08-10 09:53:20,432 INFO submit to testdev-02-s11p2 as pbsuser: job 211.testdev-02-s11p2 2017-08-10 09:53:20,433 INFO job: executable set to /bin/sleep with arguments: 100 2017-08-10 09:53:20,433 INFOCLI testdev-02-s11p2: sudo -H -u pbsuser1 /opt/pbs/bin/qsub -- /bin/sleep 100 2017-08-10 09:53:20,482 INFO submit to testdev-02-s11p2 as pbsuser1: job 212.testdev-02-s11p2 2017-08-10 09:53:20,483 INFO manager on testdev-02-s11p2: set server {'scheduling': 'True'} 2017-08-10 09:53:20,483 INFOCLI testdev-02-s11p2: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-08-10 09:53:20,544 INFO scheduler testdev-02-s11p2 log match: searching for "Leaving Scheduling Cycle" - No match 2017-08-10 09:53:21,560 INFO scheduler testdev-02-s11p2 log match: searching for "Leaving Scheduling Cycle"... OK 2017-08-10 09:53:21,594 INFO =============================== 2017-08-10 09:53:21,594 INFO Entered TestFairshare tearDown 2017-08-10 09:53:21,594 INFO =============================== 2017-08-10 09:53:21,594 INFO ================================ 2017-08-10 09:53:21,595 INFO Completed TestFairshare tearDown 2017-08-10 09:53:21,595 INFO ================================ 2017-08-10 09:53:21,596 INFO ok 2017-08-10 09:53:21,605 INFO ERROR 2017-08-10 09:53:21,607 INFO ERROR 2017-08-10 09:53:21,607 INFO 2017-08-10 09:53:21,608 INFO ====================================================================== 2017-08-10 09:53:21,608 INFO ERROR: Failure: ImportError (No module named functional.pbs_interactivetestsuite_ext) 2017-08-10 09:53:21,608 INFO ___m_oo_m___ 2017-08-10 09:53:21,608 INFO Traceback (most recent call last): File "/home/anamika/PTL/lib/python2.7/site-packages/nose/case.py", line 133, in run self.runTest(result) File "/home/anamika/PTL/lib/python2.7/site-packages/nose/case.py", line 151, in runTest test(result) File "/opt/pbs/python/lib/python2.7/unittest/case.py", line 393, in __call__ return self.run(*args, **kwds) File "/opt/pbs/python/lib/python2.7/unittest/case.py", line 304, in run result.startTest(self) File "/home/anamika/PTL/lib/python2.7/site-packages/nose/proxy.py", line 169, in startTest self.plugins.startTest(self.test) File "/home/anamika/PTL/lib/python2.7/site-packages/nose/plugins/manager.py", line 99, in __call__ return self.call(*arg, **kw) File "/home/anamika/PTL/lib/python2.7/site-packages/nose/plugins/manager.py", line 167, in simple result = meth(*arg, **kw) File "/home/anamika/PTL/lib/python2.7/site-packages/ptl/utils/plugins/ptl_test_runner.py", line 599, in startTest timeout = self.__get_timeout(test) File "/home/anamika/PTL/lib/python2.7/site-packages/ptl/utils/plugins/ptl_test_runner.py", line 551, in __get_timeout __conf = getattr(test.test, 'conf') AttributeError: 'Failure' object has no attribute 'conf' 2017-08-10 09:53:21,608 INFO ====================================================================== 2017-08-10 09:53:21,608 INFO ERROR: Failure: ImportError (No module named performance.pbs_benchtestsuite_ext) 2017-08-10 09:53:21,609 INFO ___m_oo_m___ 2017-08-10 09:53:21,609 INFO Traceback (most recent call last): File "/home/anamika/PTL/lib/python2.7/site-packages/nose/case.py", line 133, in run self.runTest(result) File "/home/anamika/PTL/lib/python2.7/site-packages/nose/case.py", line 151, in runTest test(result) File "/opt/pbs/python/lib/python2.7/unittest/case.py", line 393, in __call__ return self.run(*args, **kwds) File "/opt/pbs/python/lib/python2.7/unittest/case.py", line 304, in run result.startTest(self) File "/home/anamika/PTL/lib/python2.7/site-packages/nose/proxy.py", line 169, in startTest self.plugins.startTest(self.test) File "/home/anamika/PTL/lib/python2.7/site-packages/nose/plugins/manager.py", line 99, in __call__ return self.call(*arg, **kw) File "/home/anamika/PTL/lib/python2.7/site-packages/nose/plugins/manager.py", line 167, in simple result = meth(*arg, **kw) File "/home/anamika/PTL/lib/python2.7/site-packages/ptl/utils/plugins/ptl_test_runner.py", line 599, in startTest timeout = self.__get_timeout(test) File "/home/anamika/PTL/lib/python2.7/site-packages/ptl/utils/plugins/ptl_test_runner.py", line 551, in __get_timeout __conf = getattr(test.test, 'conf') AttributeError: 'Failure' object has no attribute 'conf' 2017-08-10 09:53:21,609 INFO ================================================================================ error: Failure: ImportError (No module named functional.pbs_interactivetestsuite_ext) error: Failure: ImportError (No module named performance.pbs_benchtestsuite_ext) Test cases with failures: Failure.runTest Test suites with failures: Failure run: 10, succeeded: 8, failed: 0, errors: 2, skipped: 0, timedout: 0 Tests run in 0:00:55.013664 2017-08-10 09:53:21,609 INFO Cleaning up temporary files