2017-05-17 13:37:41,268 INFO input command: pbs_benchpress -f pbs_accumulate_resc_used.py -p moms=centosvm:centosvm2:centosvm_nfs -o PP-745_verfiy_PP-409.txt 2017-05-17 13:37:41,268 INFO param: moms=centosvm:centosvm2:centosvm_nfs 2017-05-17 13:37:41,268 INFO ptl version: 1.0.0 2017-05-17 13:37:41,268 INFO platform: Linux centosvm 3.10.0-514.10.2.el7.x86_64 #1 SMP Fri Mar 3 00:04:05 UTC 2017 x86_64 x86_64 2017-05-17 13:37:41,268 INFO python version: 2.7.5 2017-05-17 13:37:41,268 INFO user: dilipk 2017-05-17 13:37:41,268 INFO -------------------------------------------------------------------------------- 2017-05-17 13:37:41,268 INFO Cleaning up temporary files 2017-05-17 13:37:41,276 INFO ====================================================================== 2017-05-17 13:37:41,276 INFO suite name: TestPbsAccumulateRescUsed 2017-05-17 13:37:41,276 INFO suite docstring: This tests the feature in PBS that enables mom hooks to accumulate resources_used values for resources beside cput, cpupercent, and mem. This includes accumulation of custom resources. The mom hooks supported this feature are: exechost_periodic, execjob_prologue, and execjob_epilogue. PRE: Have a cluster of PBS with 3 mom hosts, with an exechost_startup that adds custom resources. POST: When a job ends, accounting_logs reflect the aggregated resources_used values. And with job_history_enable=true, one can do a 'qstat -x -f ' to obtain information of a previous job. 2017-05-17 13:37:41,276 INFO ====================================================================== 2017-05-17 13:37:41,276 INFO ============================================= 2017-05-17 13:37:41,276 INFO Entered TestPbsAccumulateRescUsed setUpClass 2017-05-17 13:37:41,276 INFO ============================================= 2017-05-17 13:37:41,277 INFOCLI2 centosvm: id pbsuser 2017-05-17 13:37:41,281 INFOCLI2 centosvm: id pbsuser1 2017-05-17 13:37:41,283 INFOCLI2 centosvm: id pbsuser2 2017-05-17 13:37:41,286 INFOCLI2 centosvm: id pbsuser3 2017-05-17 13:37:41,299 INFO server centosvm: server operating mode set to cli 2017-05-17 13:37:41,299 INFOCLI centosvm: /opt/pbs/bin/qstat -Bf centosvm 2017-05-17 13:37:41,310 INFO server centosvm: version 14.0.1 2017-05-17 13:37:41,310 INFO expect action: created new action kicksched 2017-05-17 13:37:41,310 INFO expect action: added action kicksched to server centosvm 2017-05-17 13:37:41,332 INFOCLI2 centosvm: sudo -H /opt/pbs/sbin/pbsfs 2017-05-17 13:37:41,352 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2017-05-17 13:37:41,365 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list sched 2017-05-17 13:37:41,376 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2017-05-17 13:37:41,397 INFO server centosvm: server operating mode set to cli 2017-05-17 13:37:41,398 INFOCLI centosvm: /opt/pbs/bin/qstat -Bf centosvm 2017-05-17 13:37:41,411 INFO server centosvm: version 14.0.1 2017-05-17 13:37:41,415 INFOCLI2 centosvm2: ssh centosvm2 python -c "import os;print [False, os.environ['PBS_CONF_FILE']]['PBS_CONF_FILE' in os.environ]" 2017-05-17 13:37:43,795 INFO server centosvm2: server operating mode set to cli 2017-05-17 13:37:43,796 INFOCLI centosvm: /opt/pbs/bin/qstat -Bf centosvm2 2017-05-17 13:37:43,814 ERROR err: ['Connection refused', 'qstat: cannot connect to server centosvm2 (errno=111)'] 2017-05-17 13:37:43,815 INFO server centosvm2: version unknown 2017-05-17 13:37:45,295 INFOCLI2 centosvm_nfs: ssh centosvm_nfs python -c "import os;print [False, os.environ['PBS_CONF_FILE']]['PBS_CONF_FILE' in os.environ]" 2017-05-17 13:37:47,658 INFO server centosvm_nfs: server operating mode set to cli 2017-05-17 13:37:47,658 INFOCLI centosvm: /opt/pbs/bin/qstat -Bf centosvm_nfs 2017-05-17 13:37:47,708 ERROR err: ['Connection refused', 'qstat: cannot connect to server centosvm_nfs (errno=111)'] 2017-05-17 13:37:47,709 INFO server centosvm_nfs: version unknown 2017-05-17 13:37:48,791 INFO ============================================== 2017-05-17 13:37:48,791 INFO Completed TestPbsAccumulateRescUsed setUpClass 2017-05-17 13:37:48,791 INFO ============================================== 2017-05-17 13:37:48,792 INFO test name: test_epi_pro (tests.functional.pbs_accumulate_resc_used.TestPbsAccumulateRescUsed)... 2017-05-17 13:37:48,792 INFO test start time: Wed May 17 13:37:48 2017 2017-05-17 13:37:48,792 INFO test docstring: Test that epilogue and prologue changing same and different resources. Values of same resource would get overwriteen by the last hook. 2017-05-17 13:37:48,792 INFO ======================================== 2017-05-17 13:37:48,793 INFO Entered TestPbsAccumulateRescUsed setUp 2017-05-17 13:37:48,793 INFO ======================================== 2017-05-17 13:37:48,793 INFOCLI centosvm: /opt/pbs/bin/qstat -Bf centosvm 2017-05-17 13:37:48,818 INFO status on centosvm: server 2017-05-17 13:37:48,818 INFOCLI centosvm: /opt/pbs/bin/qstat -Bf centosvm 2017-05-17 13:37:48,844 INFO manager on centosvm: set server {'managers': (3, 'dilipk@*')} 2017-05-17 13:37:48,845 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c set server managers-=dilipk@* 2017-05-17 13:37:48,873 INFO manager on centosvm: set server {'managers': (2, 'dilipk@*')} 2017-05-17 13:37:48,874 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c set server managers+=dilipk@* 2017-05-17 13:37:48,899 INFO server centosvm: reverting configuration to defaults 2017-05-17 13:37:48,899 INFOCLI centosvm: /opt/pbs/bin/qstat -Bf centosvm 2017-05-17 13:37:48,910 INFO select on centosvm: __ALL__ 2017-05-17 13:37:48,910 INFOCLI centosvm: /opt/pbs/bin/qselect 2017-05-17 13:37:48,928 INFOCLI centosvm: /opt/pbs/bin/qstat -f @centosvm 2017-05-17 13:37:48,939 INFO expect on server centosvm: job_state set 0 job ... OK 2017-05-17 13:37:48,940 INFOCLI centosvm: /opt/pbs/bin/pbs_rstat -f 2017-05-17 13:37:48,950 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-05-17 13:37:48,969 INFOCLI centosvm: /opt/pbs/bin/qstat -Qf @centosvm 2017-05-17 13:37:48,982 INFO manager on centosvm: create queue workq {'started': 'True', 'queue_type': 'Execution', 'enabled': 'True'} 2017-05-17 13:37:48,983 INFOCLI centosvm: /opt/pbs/bin/qmgr -c create queue workq started=True,queue_type=Execution,enabled=True 2017-05-17 13:37:49,000 INFOCLI centosvm: /opt/pbs/bin/qstat -Qf workq@centosvm 2017-05-17 13:37:49,012 INFO expect on server centosvm: started set True || queue_type set Execution || enabled set True queue workq ... OK 2017-05-17 13:37:49,013 INFO manager on centosvm: set server {'default_queue': 'workq'} 2017-05-17 13:37:49,013 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server default_queue=workq 2017-05-17 13:37:49,027 INFO status on centosvm: resource 2017-05-17 13:37:49,027 INFO manager on centosvm: list resource 2017-05-17 13:37:49,027 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource 2017-05-17 13:37:49,042 INFOCLI status on centosvm: server license_count 2017-05-17 13:37:49,043 INFOCLI centosvm: /opt/pbs/bin/qstat -Bf centosvm 2017-05-17 13:37:49,056 INFO server: centosvm licensed 2017-05-17 13:37:49,075 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/server_priv/comm.lock 2017-05-17 13:37:49,105 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-05-17 13:37:49,115 INFO scheduler centosvm: reverting configuration to defaults 2017-05-17 13:37:49,116 INFO manager on centosvm: list sched 2017-05-17 13:37:49,116 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list sched 2017-05-17 13:37:49,130 INFO manager on centosvm: unset sched ['sched_cycle_length'] 2017-05-17 13:37:49,130 INFOCLI centosvm: /opt/pbs/bin/qmgr -c unset sched sched_cycle_length 2017-05-17 13:37:49,144 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/dedicated_time 2017-05-17 13:37:49,155 INFOCLI2 centosvm: cmp /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-05-17 13:37:49,158 ERROR err: ['cmp: /var/spool/pbs/sched_priv/resource_group: Permission denied'] 2017-05-17 13:37:49,158 INFOCLI2 centosvm: which cp 2017-05-17 13:37:49,160 INFOCLI2 centosvm: sudo -H /usr/bin/cp -p /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-05-17 13:37:49,173 INFOCLI2 centosvm: which chmod 2017-05-17 13:37:49,176 INFOCLI2 centosvm: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-05-17 13:37:49,187 INFO scheduler centosvm: reverting holidays file to default 2017-05-17 13:37:49,187 INFOCLI2 centosvm: cmp /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-05-17 13:37:49,191 ERROR err: ['cmp: /var/spool/pbs/sched_priv/holidays: Permission denied'] 2017-05-17 13:37:49,191 INFOCLI2 centosvm: sudo -H /usr/bin/cp -p /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-05-17 13:37:49,201 INFOCLI2 centosvm: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/holidays 2017-05-17 13:37:49,214 INFO scheduler centosvm: Parsing holidays file 2017-05-17 13:37:49,214 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2017-05-17 13:37:49,226 INFOCLI2 centosvm: cmp /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-05-17 13:37:49,229 ERROR err: ['cmp: /var/spool/pbs/sched_priv/sched_config: Permission denied'] 2017-05-17 13:37:49,229 INFOCLI2 centosvm: sudo -H /usr/bin/cp -p /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-05-17 13:37:49,242 INFOCLI2 centosvm: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-05-17 13:37:49,253 INFO scheduler centosvm: sent signal -HUP 2017-05-17 13:37:49,253 INFOCLI2 centosvm: sudo -H kill -HUP 84281 2017-05-17 13:37:49,267 INFOCLI2 centosvm: which rm 2017-05-17 13:37:49,271 INFOCLI2 centosvm: sudo -H /usr/bin/rm /var/spool/pbs/sched_priv/usage 2017-05-17 13:37:49,284 ERROR err: ['/usr/bin/rm: cannot remove \xe2\x80\x98/var/spool/pbs/sched_priv/usage\xe2\x80\x99: No such file or directory'] 2017-05-17 13:37:49,310 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-05-17 13:37:49,322 INFOCLI2 centosvm: sudo -H cmp /opt/pbs/sbin/pbs_mom /opt/pbs/sbin/pbs_mom.cpuset 2017-05-17 13:37:49,348 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:37:49,360 INFOCLI2 centosvm: sudo -H /opt/pbs/sbin/pbs_mom --version 2017-05-17 13:37:49,373 INFO mom centosvm: reverting configuration to defaults 2017-05-17 13:37:49,374 INFOCLI2 centosvm: sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2017-05-17 13:37:49,384 INFOCLI2 centosvm: sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2017-05-17 13:37:49,397 INFOCLI2 centosvm: sudo -H ls /var/spool/pbs/mom_priv/config.d 2017-05-17 13:37:49,410 INFOCLI2 centosvm: sudo -H /usr/bin/cp -p /tmp/PtlPbsrY0oGH /var/spool/pbs/mom_priv/config 2017-05-17 13:37:49,423 INFOCLI2 centosvm: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/mom_priv/config 2017-05-17 13:37:49,433 INFOCLI2 centosvm: which chown 2017-05-17 13:37:49,437 INFOCLI2 centosvm: sudo -H /usr/bin/chown root /var/spool/pbs/mom_priv/config 2017-05-17 13:37:49,450 INFOCLI2 centosvm: which chgrp 2017-05-17 13:37:49,452 INFOCLI2 centosvm: sudo -H /usr/bin/chgrp root /var/spool/pbs/mom_priv/config 2017-05-17 13:37:49,464 INFO mom centosvm: sent signal -HUP 2017-05-17 13:37:49,464 INFOCLI2 centosvm: sudo -H kill -HUP 84278 2017-05-17 13:37:49,494 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:37:49,506 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v -a 2017-05-17 13:37:49,523 INFO server centosvm: server operating mode set to cli 2017-05-17 13:37:49,523 INFOCLI centosvm: /opt/pbs/bin/qstat -Bf centosvm 2017-05-17 13:37:49,535 INFO server centosvm: version 14.0.1 2017-05-17 13:37:49,539 INFO server centosvm2: server operating mode set to cli 2017-05-17 13:37:49,540 INFOCLI centosvm: /opt/pbs/bin/qstat -Bf centosvm2 2017-05-17 13:37:49,546 ERROR err: ['Connection refused', 'qstat: cannot connect to server centosvm2 (errno=111)'] 2017-05-17 13:37:49,547 INFO server centosvm2: version unknown 2017-05-17 13:37:49,549 INFO server centosvm_nfs: server operating mode set to cli 2017-05-17 13:37:49,549 INFOCLI centosvm: /opt/pbs/bin/qstat -Bf centosvm_nfs 2017-05-17 13:37:49,559 ERROR err: ['Connection refused', 'qstat: cannot connect to server centosvm_nfs (errno=111)'] 2017-05-17 13:37:49,559 INFO server centosvm_nfs: version unknown 2017-05-17 13:37:49,560 INFO status on centosvm: node centosvm 2017-05-17 13:37:49,560 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:37:49,573 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:37:49,585 INFO expect on server centosvm: state = free node centosvm ... OK 2017-05-17 13:37:49,585 INFOCLI2 centosvm2: ssh centosvm2 sudo -H cmp /opt/pbs/sbin/pbs_mom /opt/pbs/sbin/pbs_mom.cpuset 2017-05-17 13:37:50,577 INFOCLI2 centosvm2: ssh centosvm2 /usr/bin/python -c "import sys; print sys.platform" 2017-05-17 13:37:51,732 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:37:52,188 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /opt/pbs/sbin/pbs_mom --version 2017-05-17 13:37:52,661 INFO mom centosvm2: reverting configuration to defaults 2017-05-17 13:37:52,661 INFOCLI2 centosvm2: ssh centosvm2 which rm 2017-05-17 13:37:53,140 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2017-05-17 13:37:53,704 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2017-05-17 13:37:54,163 INFOCLI2 centosvm2: ssh centosvm2 sudo -H ls /var/spool/pbs/mom_priv/config.d 2017-05-17 13:37:54,675 INFOCLI2 centosvm2: ssh centosvm2 python -c "import tempfile;print tempfile.mkstemp('PtlPbstmpcopy')[1]" 2017-05-17 13:37:55,086 INFOCLI2 centosvm2: ssh centosvm2 which scp 2017-05-17 13:37:55,501 INFOCLI2 centosvm: /usr/bin/scp -p /tmp/PtlPbs6LbRNM centosvm2:/tmp/tmpADQpNTPtlPbstmpcopy 2017-05-17 13:37:55,963 INFOCLI2 centosvm2: ssh centosvm2 which mv 2017-05-17 13:37:56,425 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/mv /tmp/tmpADQpNTPtlPbstmpcopy /var/spool/pbs/mom_priv/config 2017-05-17 13:37:57,009 INFOCLI2 centosvm2: ssh centosvm2 which chmod 2017-05-17 13:37:57,387 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/chmod 0644 /var/spool/pbs/mom_priv/config 2017-05-17 13:37:57,860 INFOCLI2 centosvm2: ssh centosvm2 which chown 2017-05-17 13:37:58,382 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/chown root /var/spool/pbs/mom_priv/config 2017-05-17 13:37:58,771 INFOCLI2 centosvm2: ssh centosvm2 which chgrp 2017-05-17 13:37:59,234 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/chgrp root /var/spool/pbs/mom_priv/config 2017-05-17 13:37:59,734 INFO mom centosvm2: sent signal -HUP 2017-05-17 13:37:59,735 INFOCLI2 centosvm2: ssh centosvm2 sudo -H kill -HUP 9078 2017-05-17 13:38:00,796 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:38:01,218 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v -a 2017-05-17 13:38:01,250 INFO status on centosvm: node centosvm2 2017-05-17 13:38:01,251 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm2 2017-05-17 13:38:01,278 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm2 2017-05-17 13:38:01,296 INFO expect on server centosvm: state = free node centosvm2 ... OK 2017-05-17 13:38:01,297 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H cmp /opt/pbs/sbin/pbs_mom /opt/pbs/sbin/pbs_mom.cpuset 2017-05-17 13:38:02,455 INFOCLI2 centosvm_nfs: ssh centosvm_nfs /usr/bin/python -c "import sys; print sys.platform" 2017-05-17 13:38:03,677 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:38:04,080 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H /opt/pbs/sbin/pbs_mom --version 2017-05-17 13:38:04,629 INFO mom centosvm_nfs: reverting configuration to defaults 2017-05-17 13:38:04,630 INFOCLI2 centosvm_nfs: ssh centosvm_nfs which rm 2017-05-17 13:38:05,135 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2017-05-17 13:38:05,759 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2017-05-17 13:38:06,207 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H ls /var/spool/pbs/mom_priv/config.d 2017-05-17 13:38:06,648 INFOCLI2 centosvm_nfs: ssh centosvm_nfs python -c "import tempfile;print tempfile.mkstemp('PtlPbstmpcopy')[1]" 2017-05-17 13:38:07,108 INFOCLI2 centosvm_nfs: ssh centosvm_nfs which scp 2017-05-17 13:38:07,552 INFOCLI2 centosvm: /usr/bin/scp -p /tmp/PtlPbsM11_yc centosvm_nfs:/tmp/tmpOah26UPtlPbstmpcopy 2017-05-17 13:38:08,110 INFOCLI2 centosvm_nfs: ssh centosvm_nfs which mv 2017-05-17 13:38:08,507 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H /usr/bin/mv /tmp/tmpOah26UPtlPbstmpcopy /var/spool/pbs/mom_priv/config 2017-05-17 13:38:09,001 INFOCLI2 centosvm_nfs: ssh centosvm_nfs which chmod 2017-05-17 13:38:09,454 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H /usr/bin/chmod 0644 /var/spool/pbs/mom_priv/config 2017-05-17 13:38:09,933 INFOCLI2 centosvm_nfs: ssh centosvm_nfs which chown 2017-05-17 13:38:10,474 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H /usr/bin/chown root /var/spool/pbs/mom_priv/config 2017-05-17 13:38:10,912 INFOCLI2 centosvm_nfs: ssh centosvm_nfs which chgrp 2017-05-17 13:38:11,357 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H /usr/bin/chgrp root /var/spool/pbs/mom_priv/config 2017-05-17 13:38:11,796 INFO mom centosvm_nfs: sent signal -HUP 2017-05-17 13:38:11,796 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H kill -HUP 76746 2017-05-17 13:38:12,801 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:38:13,346 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v -a 2017-05-17 13:38:13,362 INFO status on centosvm: node centosvm_nfs 2017-05-17 13:38:13,363 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm_nfs 2017-05-17 13:38:13,375 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm_nfs 2017-05-17 13:38:13,388 INFO expect on server centosvm: state = free node centosvm_nfs ... OK 2017-05-17 13:38:13,389 INFO ========================================= 2017-05-17 13:38:13,389 INFO Completed TestPbsAccumulateRescUsed setUp 2017-05-17 13:38:13,389 INFO ========================================= 2017-05-17 13:38:13,389 INFO len moms = 3 2017-05-17 13:38:13,389 INFOCLI2 centosvm: sudo -H /usr/bin/rm -rf /var/spool/pbs/mom_priv/config.d 2017-05-17 13:38:13,402 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/rm -rf /var/spool/pbs/mom_priv/config.d 2017-05-17 13:38:13,848 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H /usr/bin/rm -rf /var/spool/pbs/mom_priv/config.d 2017-05-17 13:38:14,322 INFO manager on centosvm: delete node 2017-05-17 13:38:14,322 INFOCLI centosvm: /opt/pbs/bin/qmgr -c delete node @default 2017-05-17 13:38:14,358 INFO manager on centosvm: create node centosvm 2017-05-17 13:38:14,359 INFOCLI centosvm: /opt/pbs/bin/qmgr -c create node centosvm 2017-05-17 13:38:14,388 INFO manager on centosvm: create node centosvm2 2017-05-17 13:38:14,388 INFOCLI centosvm: /opt/pbs/bin/qmgr -c create node centosvm2 2017-05-17 13:38:14,411 INFO manager on centosvm: create node centosvm_nfs 2017-05-17 13:38:14,412 INFOCLI centosvm: /opt/pbs/bin/qmgr -c create node centosvm_nfs 2017-05-17 13:38:14,431 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:38:14,452 INFO expect on server centosvm: state = free node centosvm got: state = state-unknown,down 2017-05-17 13:38:14,954 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:38:14,996 INFO expect on server centosvm: state = free node centosvm attempt: 2 got: state = state-unknown,down 2017-05-17 13:38:15,499 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:38:15,543 INFO expect on server centosvm: state = free node centosvm attempt: 3 got: state = state-unknown,down 2017-05-17 13:38:16,046 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:38:16,098 INFO expect on server centosvm: state = free node centosvm attempt: 4 ... OK 2017-05-17 13:38:16,099 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm2 2017-05-17 13:38:16,128 INFO expect on server centosvm: state = free node centosvm2 got: state = initializing 2017-05-17 13:38:16,629 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm2 2017-05-17 13:38:16,661 INFO expect on server centosvm: state = free node centosvm2 attempt: 2 ... OK 2017-05-17 13:38:16,662 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm_nfs 2017-05-17 13:38:16,699 INFO expect on server centosvm: state = free node centosvm_nfs ... OK 2017-05-17 13:38:16,699 INFO status on centosvm: hook 2017-05-17 13:38:16,700 INFO manager on centosvm: list hook 2017-05-17 13:38:16,700 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-05-17 13:38:16,752 INFO status on centosvm: hook 2017-05-17 13:38:16,753 INFO manager on centosvm: list hook 2017-05-17 13:38:16,753 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-05-17 13:38:16,798 INFO manager on centosvm: create hook start 2017-05-17 13:38:16,798 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c create hook start 2017-05-17 13:38:16,839 INFO manager on centosvm: set hook start {'enabled': 'True', 'event': 'exechost_startup'} 2017-05-17 13:38:16,839 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c set hook start enabled=True,event=exechost_startup 2017-05-17 13:38:16,880 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook start 2017-05-17 13:38:16,922 INFO expect on server centosvm: enabled set True || event set exechost_startup hook start ... OK 2017-05-17 13:38:16,923 INFO manager on centosvm: import hook start {'content-type': 'application/x-python', 'input-file': '/tmp/PtlPbsWcKTeM', 'content-encoding': 'default'} 2017-05-17 13:38:16,924 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c import hook start application/x-python default /tmp/PtlPbsWcKTeM 2017-05-17 13:38:16,955 INFO server centosvm: imported hook body --- import pbs e=pbs.event() localnode=pbs.get_local_nodename() e.vnode_list[localnode].resources_available['foo_i'] = 7 e.vnode_list[localnode].resources_available['foo_f'] = 5.0 e.vnode_list[localnode].resources_available['foo_str'] = "seventyseven" --- 2017-05-17 13:38:16,956 INFO mom centosvm: sent signal -HUP 2017-05-17 13:38:16,956 INFOCLI2 centosvm: sudo -H kill -HUP 84278 2017-05-17 13:38:16,968 INFO mom centosvm2: sent signal -HUP 2017-05-17 13:38:16,969 INFOCLI2 centosvm2: ssh centosvm2 sudo -H kill -HUP 9078 2017-05-17 13:38:17,357 INFO mom centosvm_nfs: sent signal -HUP 2017-05-17 13:38:17,358 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H kill -HUP 76746 2017-05-17 13:38:17,815 INFO manager on centosvm: set server {'job_history_enable': 'True'} 2017-05-17 13:38:17,817 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server job_history_enable=True 2017-05-17 13:38:17,867 INFO manager on centosvm as root: create resource foo_str2 {'flag': 'h', 'type': 'string'} 2017-05-17 13:38:17,867 INFOCLI centosvm: sudo -H -u root /opt/pbs/bin/qmgr -c create resource foo_str2 flag=h,type=string 2017-05-17 13:38:17,953 INFO mom centosvm log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:38:18,393 INFO mom centosvm2 log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:38:19,018 INFO mom centosvm_nfs log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:38:19,020 INFO manager on centosvm as root: create resource foo_str3 {'flag': 'h', 'type': 'string'} 2017-05-17 13:38:19,021 INFOCLI centosvm: sudo -H -u root /opt/pbs/bin/qmgr -c create resource foo_str3 flag=h,type=string 2017-05-17 13:38:19,108 INFO mom centosvm log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:38:19,653 INFO mom centosvm2 log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:38:20,324 INFO mom centosvm_nfs log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:38:20,325 INFO manager on centosvm as root: create resource foo_str4 {'flag': 'h', 'type': 'string'} 2017-05-17 13:38:20,325 INFOCLI centosvm: sudo -H -u root /opt/pbs/bin/qmgr -c create resource foo_str4 flag=h,type=string 2017-05-17 13:38:20,391 INFO mom centosvm log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:38:20,847 INFO mom centosvm2 log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:38:21,441 INFO mom centosvm_nfs log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:38:21,441 INFO manager on centosvm as root: create resource stra {'flag': 'h', 'type': 'string_array'} 2017-05-17 13:38:21,442 INFOCLI centosvm: sudo -H -u root /opt/pbs/bin/qmgr -c create resource stra flag=h,type=string_array 2017-05-17 13:38:21,499 INFO mom centosvm log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:38:21,872 INFO mom centosvm2 log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:38:22,431 INFO mom centosvm_nfs log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:38:22,432 INFO status on centosvm: hook 2017-05-17 13:38:22,433 INFO manager on centosvm: list hook 2017-05-17 13:38:22,433 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-05-17 13:38:22,490 INFO status on centosvm: hook 2017-05-17 13:38:22,490 INFO manager on centosvm: list hook 2017-05-17 13:38:22,490 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-05-17 13:38:22,525 INFO manager on centosvm: create hook pro 2017-05-17 13:38:22,526 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c create hook pro 2017-05-17 13:38:22,559 INFO manager on centosvm: set hook pro {'enabled': 'True', 'event': 'execjob_prologue'} 2017-05-17 13:38:22,559 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c set hook pro enabled=True,event=execjob_prologue 2017-05-17 13:38:22,592 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook pro 2017-05-17 13:38:22,616 INFO expect on server centosvm: enabled set True || event set execjob_prologue hook pro ... OK 2017-05-17 13:38:22,617 INFO manager on centosvm: import hook pro {'content-type': 'application/x-python', 'input-file': '/tmp/PtlPbs8zmoQ7', 'content-encoding': 'default'} 2017-05-17 13:38:22,617 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c import hook pro application/x-python default /tmp/PtlPbs8zmoQ7 2017-05-17 13:38:22,638 INFO server centosvm: imported hook body --- import pbs e=pbs.event() pbs.logmsg(pbs.LOG_DEBUG, "In prologue hook") e.job.resources_used["foo_i"] = 10 e.job.resources_used["foo_f"] = 0.10 --- 2017-05-17 13:38:22,639 INFO status on centosvm: hook 2017-05-17 13:38:22,639 INFO manager on centosvm: list hook 2017-05-17 13:38:22,639 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-05-17 13:38:22,660 INFO status on centosvm: hook 2017-05-17 13:38:22,660 INFO manager on centosvm: list hook 2017-05-17 13:38:22,660 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-05-17 13:38:22,679 INFO manager on centosvm: create hook epi 2017-05-17 13:38:22,679 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c create hook epi 2017-05-17 13:38:22,708 INFO manager on centosvm: set hook epi {'enabled': 'True', 'event': 'execjob_epilogue'} 2017-05-17 13:38:22,708 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c set hook epi enabled=True,event=execjob_epilogue 2017-05-17 13:38:22,735 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook epi 2017-05-17 13:38:22,755 INFO expect on server centosvm: enabled set True || event set execjob_epilogue hook epi ... OK 2017-05-17 13:38:22,756 INFO manager on centosvm: import hook epi {'content-type': 'application/x-python', 'input-file': '/tmp/PtlPbsrpSp7S', 'content-encoding': 'default'} 2017-05-17 13:38:22,756 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c import hook epi application/x-python default /tmp/PtlPbsrpSp7S 2017-05-17 13:38:22,780 INFO server centosvm: imported hook body --- import pbs e=pbs.event() pbs.logmsg(pbs.LOG_DEBUG, "In epilogue hook") e.job.resources_used["foo_f"] = 0.20 e.job.resources_used["cput"] = 10 --- 2017-05-17 13:38:22,781 INFO job: executable set to /bin/sleep with arguments: 100 2017-05-17 13:38:22,782 INFOCLI2 centosvm: sudo -H /usr/bin/chmod 0755 /tmp/PtlPbsJobScriptAzDGJH 2017-05-17 13:38:22,793 INFOCLI centosvm: sudo -H -u pbsuser /opt/pbs/bin/qsub -l place=scatter -l select=3:ncpus=1 /tmp/PtlPbsJobScriptAzDGJH 2017-05-17 13:38:22,814 INFO submit to centosvm as pbsuser: job 0.centosvm OrderedDict([('Resource_List.place', 'scatter'), ('Resource_List.select', '3:ncpus=1')]) 2017-05-17 13:38:22,814 INFOCLI job script /tmp/PtlPbsJobScriptAzDGJH --- pbsdsh -n 1 hostname pbsdsh -n 2 hostname sleep 5 --- 2017-05-17 13:38:22,814 INFO server centosvm: expect offset set to 5 2017-05-17 13:38:27,820 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 0.centosvm 2017-05-17 13:38:27,867 INFO expect on server centosvm: resources_used.cput = 30 || job_state = F || resources_used.foo_i = 30 || resources_used.foo_f = 0.6 job 0.centosvm got: resources_used.cput = 0 2017-05-17 13:38:28,869 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:38:28,896 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 0.centosvm 2017-05-17 13:38:28,931 INFO expect on server centosvm: resources_used.cput = 30 || job_state = F || resources_used.foo_i = 30 || resources_used.foo_f = 0.6 job 0.centosvm attempt: 2 got: resources_used.cput = 0 2017-05-17 13:38:29,934 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:38:29,979 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 0.centosvm 2017-05-17 13:38:30,011 INFO expect on server centosvm: resources_used.cput = 30 || job_state = F || resources_used.foo_i = 30 || resources_used.foo_f = 0.6 job 0.centosvm attempt: 3 ... OK 2017-05-17 13:38:30,012 INFOCLI2 centosvm: sudo -H /usr/bin/chmod 0755 /tmp/PtlPbsJobScriptmFCasR 2017-05-17 13:38:30,034 INFOCLI centosvm: /opt/pbs/bin/qsub -l place=scatter -l select=3:ncpus=1 /tmp/PtlPbsJobScriptmFCasR 2017-05-17 13:38:30,050 INFO submit to centosvm as None: job 1.centosvm OrderedDict([('Resource_List.place', 'scatter'), ('Resource_List.select', '3:ncpus=1')]) 2017-05-17 13:38:30,050 INFOCLI job script /tmp/PtlPbsJobScriptmFCasR --- pbsdsh -n 1 hostname pbsdsh -n 2 hostname sleep 300 --- 2017-05-17 13:38:30,050 INFOCLI centosvm: /opt/pbs/bin/qstat -f 1.centosvm 2017-05-17 13:38:30,063 INFO expect on server centosvm: job_state = R && resources_used.foo_i = 30 && resources_used.foo_f = 0.3 && substate = 42 job 1.centosvm got: job_state = Q 2017-05-17 13:38:32,066 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:38:32,113 INFOCLI centosvm: /opt/pbs/bin/qstat -f 1.centosvm 2017-05-17 13:38:32,158 INFO expect on server centosvm: job_state = R && resources_used.foo_i = 30 && resources_used.foo_f = 0.3 && substate = 42 job 1.centosvm attempt: 2 got: resources_used.foo_i = 10 2017-05-17 13:38:34,163 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:38:34,211 INFOCLI centosvm: /opt/pbs/bin/qstat -f 1.centosvm 2017-05-17 13:38:34,248 INFO expect on server centosvm: job_state = R && resources_used.foo_i = 30 && resources_used.foo_f = 0.3 && substate = 42 job 1.centosvm attempt: 3 got: resources_used.foo_i = 10 2017-05-17 13:38:36,252 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:38:36,297 INFOCLI centosvm: /opt/pbs/bin/qstat -f 1.centosvm 2017-05-17 13:38:36,334 INFO expect on server centosvm: job_state = R && resources_used.foo_i = 30 && resources_used.foo_f = 0.3 && substate = 42 job 1.centosvm attempt: 4 got: resources_used.foo_i = 10 2017-05-17 13:38:38,338 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:38:38,354 INFOCLI centosvm: /opt/pbs/bin/qstat -f 1.centosvm 2017-05-17 13:38:38,369 INFO expect on server centosvm: job_state = R && resources_used.foo_i = 30 && resources_used.foo_f = 0.3 && substate = 42 job 1.centosvm attempt: 5 got: resources_used.foo_i = 10 2017-05-17 13:38:40,372 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:38:40,419 INFOCLI centosvm: /opt/pbs/bin/qstat -f 1.centosvm 2017-05-17 13:38:40,453 INFO expect on server centosvm: job_state = R && resources_used.foo_i = 30 && resources_used.foo_f = 0.3 && substate = 42 job 1.centosvm attempt: 6 got: resources_used.foo_i = 10 2017-05-17 13:38:42,459 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:38:42,509 INFOCLI centosvm: /opt/pbs/bin/qstat -f 1.centosvm 2017-05-17 13:38:42,523 INFO expect on server centosvm: job_state = R && resources_used.foo_i = 30 && resources_used.foo_f = 0.3 && substate = 42 job 1.centosvm attempt: 7 got: resources_used.foo_i = 10 2017-05-17 13:38:44,528 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:38:44,573 INFOCLI centosvm: /opt/pbs/bin/qstat -f 1.centosvm 2017-05-17 13:38:44,603 INFO expect on server centosvm: job_state = R && resources_used.foo_i = 30 && resources_used.foo_f = 0.3 && substate = 42 job 1.centosvm attempt: 8 got: resources_used.foo_i = 10 2017-05-17 13:38:46,609 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:38:46,661 INFOCLI centosvm: /opt/pbs/bin/qstat -f 1.centosvm 2017-05-17 13:38:46,709 INFO expect on server centosvm: job_state = R && resources_used.foo_i = 30 && resources_used.foo_f = 0.3 && substate = 42 job 1.centosvm attempt: 9 got: resources_used.foo_i = 10 2017-05-17 13:38:48,712 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:38:48,732 INFOCLI centosvm: /opt/pbs/bin/qstat -f 1.centosvm 2017-05-17 13:38:48,744 INFO expect on server centosvm: job_state = R && resources_used.foo_i = 30 && resources_used.foo_f = 0.3 && substate = 42 job 1.centosvm attempt: 10 got: resources_used.foo_i = 10 2017-05-17 13:38:50,751 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:38:50,798 INFOCLI centosvm: /opt/pbs/bin/qstat -f 1.centosvm 2017-05-17 13:38:50,834 INFO expect on server centosvm: job_state = R && resources_used.foo_i = 30 && resources_used.foo_f = 0.3 && substate = 42 job 1.centosvm attempt: 11 got: resources_used.foo_i = 10 2017-05-17 13:38:52,839 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:38:52,908 INFOCLI centosvm: /opt/pbs/bin/qstat -f 1.centosvm 2017-05-17 13:38:52,944 INFO expect on server centosvm: job_state = R && resources_used.foo_i = 30 && resources_used.foo_f = 0.3 && substate = 42 job 1.centosvm attempt: 12 got: resources_used.foo_i = 10 2017-05-17 13:38:54,948 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:38:55,004 INFOCLI centosvm: /opt/pbs/bin/qstat -f 1.centosvm 2017-05-17 13:38:55,036 INFO expect on server centosvm: job_state = R && resources_used.foo_i = 30 && resources_used.foo_f = 0.3 && substate = 42 job 1.centosvm attempt: 13 got: resources_used.foo_i = 10 2017-05-17 13:38:57,041 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:38:57,100 INFOCLI centosvm: /opt/pbs/bin/qstat -f 1.centosvm 2017-05-17 13:38:57,139 INFO expect on server centosvm: job_state = R && resources_used.foo_i = 30 && resources_used.foo_f = 0.3 && substate = 42 job 1.centosvm attempt: 14 got: resources_used.foo_i = 10 2017-05-17 13:38:59,142 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:38:59,191 INFOCLI centosvm: /opt/pbs/bin/qstat -f 1.centosvm 2017-05-17 13:38:59,218 INFO expect on server centosvm: job_state = R && resources_used.foo_i = 30 && resources_used.foo_f = 0.3 && substate = 42 job 1.centosvm attempt: 15 ... OK 2017-05-17 13:38:59,219 INFO delete job on centosvm: 1.centosvm 2017-05-17 13:38:59,219 INFOCLI centosvm: /opt/pbs/bin/qdel -W force 1.centosvm 2017-05-17 13:38:59,254 INFOCLI centosvm: /opt/pbs/bin/qstat -f 1.centosvm 2017-05-17 13:38:59,288 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 1.centosvm 2017-05-17 13:38:59,310 INFO job: executable set to /bin/sleep with arguments: 100 2017-05-17 13:38:59,311 INFO expect on server centosvm: resources_used.foo_i = 30 || resources_used.foo_f = 0.3 job 1.centosvm ... OK 2017-05-17 13:38:59,311 INFO =========================================== 2017-05-17 13:38:59,311 INFO Entered TestPbsAccumulateRescUsed tearDown 2017-05-17 13:38:59,311 INFO =========================================== 2017-05-17 13:38:59,311 INFO ============================================ 2017-05-17 13:38:59,311 INFO Completed TestPbsAccumulateRescUsed tearDown 2017-05-17 13:38:59,312 INFO ============================================ 2017-05-17 13:38:59,312 INFO ok 2017-05-17 13:38:59,313 INFO test name: test_epilogue (tests.functional.pbs_accumulate_resc_used.TestPbsAccumulateRescUsed)... 2017-05-17 13:38:59,313 INFO test start time: Wed May 17 13:38:59 2017 2017-05-17 13:38:59,313 INFO test docstring: Test accumulatinon of resources of a multinode job from an exechost_epilogue hook. 2017-05-17 13:38:59,313 INFO ======================================== 2017-05-17 13:38:59,313 INFO Entered TestPbsAccumulateRescUsed setUp 2017-05-17 13:38:59,314 INFO ======================================== 2017-05-17 13:38:59,314 INFOCLI centosvm: /opt/pbs/bin/qstat -Bf centosvm 2017-05-17 13:38:59,333 INFO status on centosvm: server 2017-05-17 13:38:59,333 INFOCLI centosvm: /opt/pbs/bin/qstat -Bf centosvm 2017-05-17 13:38:59,354 INFO manager on centosvm: set server {'managers': (3, 'dilipk@*')} 2017-05-17 13:38:59,354 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c set server managers-=dilipk@* 2017-05-17 13:38:59,395 INFO manager on centosvm: set server {'managers': (2, 'dilipk@*')} 2017-05-17 13:38:59,396 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c set server managers+=dilipk@* 2017-05-17 13:38:59,448 INFO server centosvm: reverting configuration to defaults 2017-05-17 13:38:59,448 INFOCLI centosvm: /opt/pbs/bin/qstat -Bf centosvm 2017-05-17 13:38:59,474 INFO select on centosvm: __ALL__ 2017-05-17 13:38:59,474 INFOCLI centosvm: /opt/pbs/bin/qselect -x 2017-05-17 13:38:59,492 INFO delete job on centosvm: 0.centosvm, 1.centosvm 2017-05-17 13:38:59,493 INFOCLI centosvm: /opt/pbs/bin/qdel -W force -x 0.centosvm 1.centosvm 2017-05-17 13:38:59,529 INFOCLI centosvm: /opt/pbs/bin/qstat -f 0.centosvm 2017-05-17 13:38:59,552 INFOCLI centosvm: /opt/pbs/bin/qstat -f 1.centosvm 2017-05-17 13:38:59,582 INFOCLI centosvm: /opt/pbs/bin/qstat -f @centosvm 2017-05-17 13:38:59,601 INFO expect on server centosvm: job_state set 0 job ... OK 2017-05-17 13:38:59,601 INFOCLI centosvm: /opt/pbs/bin/pbs_rstat -f 2017-05-17 13:38:59,617 INFO manager on centosvm: unset server ['job_history_enable'] 2017-05-17 13:38:59,618 INFOCLI centosvm: /opt/pbs/bin/qmgr -c unset server job_history_enable 2017-05-17 13:38:59,639 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-05-17 13:38:59,670 INFO manager on centosvm: delete hook start,pro,epi 2017-05-17 13:38:59,670 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c delete hook start,pro,epi 2017-05-17 13:38:59,714 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook start 2017-05-17 13:38:59,750 INFO expect on server centosvm: unset hook start ... OK 2017-05-17 13:38:59,751 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook pro 2017-05-17 13:38:59,782 INFO expect on server centosvm: unset hook pro ... OK 2017-05-17 13:38:59,782 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook epi 2017-05-17 13:38:59,809 INFO expect on server centosvm: unset hook epi ... OK 2017-05-17 13:38:59,810 INFOCLI centosvm: /opt/pbs/bin/qstat -Qf @centosvm 2017-05-17 13:38:59,822 INFO manager on centosvm: delete queue workq 2017-05-17 13:38:59,822 INFOCLI centosvm: /opt/pbs/bin/qmgr -c delete queue workq 2017-05-17 13:38:59,835 INFOCLI centosvm: /opt/pbs/bin/qstat -Qf workq@centosvm 2017-05-17 13:38:59,849 INFO expect on server centosvm: unset queue workq ... OK 2017-05-17 13:38:59,849 INFO manager on centosvm: create queue workq {'started': 'True', 'queue_type': 'Execution', 'enabled': 'True'} 2017-05-17 13:38:59,849 INFOCLI centosvm: /opt/pbs/bin/qmgr -c create queue workq started=True,queue_type=Execution,enabled=True 2017-05-17 13:38:59,868 INFOCLI centosvm: /opt/pbs/bin/qstat -Qf workq@centosvm 2017-05-17 13:38:59,880 INFO expect on server centosvm: started set True || queue_type set Execution || enabled set True queue workq ... OK 2017-05-17 13:38:59,881 INFO manager on centosvm: set server {'default_queue': 'workq'} 2017-05-17 13:38:59,881 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server default_queue=workq 2017-05-17 13:38:59,894 INFO status on centosvm: resource 2017-05-17 13:38:59,895 INFO manager on centosvm: list resource 2017-05-17 13:38:59,895 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource 2017-05-17 13:38:59,908 INFO manager on centosvm: delete resource foo_f,foo_i,foo_str2,foo_str3,foo_str4,foo_str,stra 2017-05-17 13:38:59,908 INFOCLI centosvm: /opt/pbs/bin/qmgr -c delete resource foo_f,foo_i,foo_str2,foo_str3,foo_str4,foo_str,stra 2017-05-17 13:39:00,160 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_f 2017-05-17 13:39:00,181 INFO expect on server centosvm: unset resource foo_f ... OK 2017-05-17 13:39:00,181 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_i 2017-05-17 13:39:00,208 INFO expect on server centosvm: unset resource foo_i ... OK 2017-05-17 13:39:00,208 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_str2 2017-05-17 13:39:00,230 INFO expect on server centosvm: unset resource foo_str2 ... OK 2017-05-17 13:39:00,230 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_str3 2017-05-17 13:39:00,251 INFO expect on server centosvm: unset resource foo_str3 ... OK 2017-05-17 13:39:00,252 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_str4 2017-05-17 13:39:00,279 INFO expect on server centosvm: unset resource foo_str4 ... OK 2017-05-17 13:39:00,280 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_str 2017-05-17 13:39:00,302 INFO expect on server centosvm: unset resource foo_str ... OK 2017-05-17 13:39:00,303 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource stra 2017-05-17 13:39:00,323 INFO expect on server centosvm: unset resource stra ... OK 2017-05-17 13:39:00,324 INFOCLI status on centosvm: server license_count 2017-05-17 13:39:00,325 INFOCLI centosvm: /opt/pbs/bin/qstat -Bf centosvm 2017-05-17 13:39:00,353 INFO server: centosvm licensed 2017-05-17 13:39:00,386 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/server_priv/comm.lock 2017-05-17 13:39:00,421 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-05-17 13:39:00,432 INFO scheduler centosvm: reverting configuration to defaults 2017-05-17 13:39:00,432 INFO manager on centosvm: list sched 2017-05-17 13:39:00,432 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list sched 2017-05-17 13:39:00,449 INFO manager on centosvm: unset sched ['sched_cycle_length'] 2017-05-17 13:39:00,450 INFOCLI centosvm: /opt/pbs/bin/qmgr -c unset sched sched_cycle_length 2017-05-17 13:39:00,464 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/dedicated_time 2017-05-17 13:39:00,475 INFOCLI2 centosvm: cmp /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-05-17 13:39:00,485 ERROR err: ['cmp: /var/spool/pbs/sched_priv/resource_group: Permission denied'] 2017-05-17 13:39:00,485 INFOCLI2 centosvm: sudo -H /usr/bin/cp -p /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-05-17 13:39:00,498 INFOCLI2 centosvm: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-05-17 13:39:00,510 INFO scheduler centosvm: reverting holidays file to default 2017-05-17 13:39:00,510 INFOCLI2 centosvm: cmp /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-05-17 13:39:00,513 ERROR err: ['cmp: /var/spool/pbs/sched_priv/holidays: Permission denied'] 2017-05-17 13:39:00,513 INFOCLI2 centosvm: sudo -H /usr/bin/cp -p /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-05-17 13:39:00,525 INFOCLI2 centosvm: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/holidays 2017-05-17 13:39:00,540 INFO scheduler centosvm: Parsing holidays file 2017-05-17 13:39:00,541 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2017-05-17 13:39:00,552 INFOCLI2 centosvm: cmp /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-05-17 13:39:00,555 ERROR err: ['cmp: /var/spool/pbs/sched_priv/sched_config: Permission denied'] 2017-05-17 13:39:00,555 INFOCLI2 centosvm: sudo -H /usr/bin/cp -p /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-05-17 13:39:00,574 INFOCLI2 centosvm: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-05-17 13:39:00,585 INFO scheduler centosvm: sent signal -HUP 2017-05-17 13:39:00,586 INFOCLI2 centosvm: sudo -H kill -HUP 84281 2017-05-17 13:39:00,599 INFOCLI2 centosvm: sudo -H /usr/bin/rm /var/spool/pbs/sched_priv/usage 2017-05-17 13:39:00,611 ERROR err: ['/usr/bin/rm: cannot remove \xe2\x80\x98/var/spool/pbs/sched_priv/usage\xe2\x80\x99: No such file or directory'] 2017-05-17 13:39:00,645 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-05-17 13:39:00,657 INFOCLI2 centosvm: sudo -H cmp /opt/pbs/sbin/pbs_mom /opt/pbs/sbin/pbs_mom.cpuset 2017-05-17 13:39:00,695 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:39:00,714 INFO mom centosvm: reverting configuration to defaults 2017-05-17 13:39:00,715 INFOCLI2 centosvm: sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2017-05-17 13:39:00,725 INFOCLI2 centosvm: sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2017-05-17 13:39:00,736 INFOCLI2 centosvm: sudo -H ls /var/spool/pbs/mom_priv/config.d 2017-05-17 13:39:00,748 INFO mom centosvm: sent signal -HUP 2017-05-17 13:39:00,748 INFOCLI2 centosvm: sudo -H kill -HUP 84278 2017-05-17 13:39:00,776 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:39:00,789 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v -a 2017-05-17 13:39:00,803 INFO status on centosvm: node centosvm 2017-05-17 13:39:00,803 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:39:00,816 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:39:00,827 INFO expect on server centosvm: state = free node centosvm ... OK 2017-05-17 13:39:00,828 INFOCLI2 centosvm2: ssh centosvm2 sudo -H cmp /opt/pbs/sbin/pbs_mom /opt/pbs/sbin/pbs_mom.cpuset 2017-05-17 13:39:01,996 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:39:02,856 INFO mom centosvm2: reverting configuration to defaults 2017-05-17 13:39:02,857 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2017-05-17 13:39:03,361 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2017-05-17 13:39:03,772 INFOCLI2 centosvm2: ssh centosvm2 sudo -H ls /var/spool/pbs/mom_priv/config.d 2017-05-17 13:39:04,141 INFO mom centosvm2: sent signal -HUP 2017-05-17 13:39:04,141 INFOCLI2 centosvm2: ssh centosvm2 sudo -H kill -HUP 9078 2017-05-17 13:39:05,216 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:39:05,847 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v -a 2017-05-17 13:39:05,877 INFO status on centosvm: node centosvm2 2017-05-17 13:39:05,878 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm2 2017-05-17 13:39:05,900 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm2 2017-05-17 13:39:05,920 INFO expect on server centosvm: state = free node centosvm2 ... OK 2017-05-17 13:39:05,920 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H cmp /opt/pbs/sbin/pbs_mom /opt/pbs/sbin/pbs_mom.cpuset 2017-05-17 13:39:06,910 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:39:07,328 INFO mom centosvm_nfs: reverting configuration to defaults 2017-05-17 13:39:07,329 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2017-05-17 13:39:07,896 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2017-05-17 13:39:08,407 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H ls /var/spool/pbs/mom_priv/config.d 2017-05-17 13:39:08,969 INFO mom centosvm_nfs: sent signal -HUP 2017-05-17 13:39:08,971 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H kill -HUP 76746 2017-05-17 13:39:10,133 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:39:10,693 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v -a 2017-05-17 13:39:10,707 INFO status on centosvm: node centosvm_nfs 2017-05-17 13:39:10,708 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm_nfs 2017-05-17 13:39:10,719 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm_nfs 2017-05-17 13:39:10,734 INFO expect on server centosvm: state = free node centosvm_nfs ... OK 2017-05-17 13:39:10,735 INFO ========================================= 2017-05-17 13:39:10,735 INFO Completed TestPbsAccumulateRescUsed setUp 2017-05-17 13:39:10,735 INFO ========================================= 2017-05-17 13:39:10,735 INFO len moms = 3 2017-05-17 13:39:10,735 INFOCLI2 centosvm: sudo -H /usr/bin/rm -rf /var/spool/pbs/mom_priv/config.d 2017-05-17 13:39:10,750 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/rm -rf /var/spool/pbs/mom_priv/config.d 2017-05-17 13:39:11,128 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H /usr/bin/rm -rf /var/spool/pbs/mom_priv/config.d 2017-05-17 13:39:11,644 INFO manager on centosvm: delete node 2017-05-17 13:39:11,645 INFOCLI centosvm: /opt/pbs/bin/qmgr -c delete node @default 2017-05-17 13:39:11,672 INFO manager on centosvm: create node centosvm 2017-05-17 13:39:11,673 INFOCLI centosvm: /opt/pbs/bin/qmgr -c create node centosvm 2017-05-17 13:39:11,689 INFO manager on centosvm: create node centosvm2 2017-05-17 13:39:11,689 INFOCLI centosvm: /opt/pbs/bin/qmgr -c create node centosvm2 2017-05-17 13:39:11,704 INFO manager on centosvm: create node centosvm_nfs 2017-05-17 13:39:11,705 INFOCLI centosvm: /opt/pbs/bin/qmgr -c create node centosvm_nfs 2017-05-17 13:39:11,720 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:39:11,733 INFO expect on server centosvm: state = free node centosvm got: state = state-unknown,down 2017-05-17 13:39:12,235 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:39:12,283 INFO expect on server centosvm: state = free node centosvm attempt: 2 got: state = state-unknown,down 2017-05-17 13:39:12,786 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:39:12,839 INFO expect on server centosvm: state = free node centosvm attempt: 3 got: state = state-unknown,down 2017-05-17 13:39:13,341 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:39:13,385 INFO expect on server centosvm: state = free node centosvm attempt: 4 ... OK 2017-05-17 13:39:13,386 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm2 2017-05-17 13:39:13,424 INFO expect on server centosvm: state = free node centosvm2 got: state = initializing 2017-05-17 13:39:13,926 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm2 2017-05-17 13:39:13,977 INFO expect on server centosvm: state = free node centosvm2 attempt: 2 ... OK 2017-05-17 13:39:13,979 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm_nfs 2017-05-17 13:39:14,023 INFO expect on server centosvm: state = free node centosvm_nfs ... OK 2017-05-17 13:39:14,024 INFO status on centosvm: hook 2017-05-17 13:39:14,024 INFO manager on centosvm: list hook 2017-05-17 13:39:14,024 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-05-17 13:39:14,079 INFO status on centosvm: hook 2017-05-17 13:39:14,080 INFO manager on centosvm: list hook 2017-05-17 13:39:14,080 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-05-17 13:39:14,142 INFO manager on centosvm: create hook start 2017-05-17 13:39:14,142 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c create hook start 2017-05-17 13:39:14,203 INFO manager on centosvm: set hook start {'enabled': 'True', 'event': 'exechost_startup'} 2017-05-17 13:39:14,207 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c set hook start enabled=True,event=exechost_startup 2017-05-17 13:39:14,251 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook start 2017-05-17 13:39:14,288 INFO expect on server centosvm: enabled set True || event set exechost_startup hook start ... OK 2017-05-17 13:39:14,289 INFO manager on centosvm: import hook start {'content-type': 'application/x-python', 'input-file': '/tmp/PtlPbsO5Yv2B', 'content-encoding': 'default'} 2017-05-17 13:39:14,290 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c import hook start application/x-python default /tmp/PtlPbsO5Yv2B 2017-05-17 13:39:14,324 INFO server centosvm: imported hook body --- import pbs e=pbs.event() localnode=pbs.get_local_nodename() e.vnode_list[localnode].resources_available['foo_i'] = 7 e.vnode_list[localnode].resources_available['foo_f'] = 5.0 e.vnode_list[localnode].resources_available['foo_str'] = "seventyseven" --- 2017-05-17 13:39:14,325 INFO mom centosvm: sent signal -HUP 2017-05-17 13:39:14,325 INFOCLI2 centosvm: sudo -H kill -HUP 84278 2017-05-17 13:39:14,339 INFO mom centosvm2: sent signal -HUP 2017-05-17 13:39:14,339 INFOCLI2 centosvm2: ssh centosvm2 sudo -H kill -HUP 9078 2017-05-17 13:39:14,892 INFO mom centosvm_nfs: sent signal -HUP 2017-05-17 13:39:14,894 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H kill -HUP 76746 2017-05-17 13:39:15,382 INFO manager on centosvm: set server {'job_history_enable': 'True'} 2017-05-17 13:39:15,383 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server job_history_enable=True 2017-05-17 13:39:15,404 INFO manager on centosvm as root: create resource foo_str2 {'flag': 'h', 'type': 'string'} 2017-05-17 13:39:15,405 INFOCLI centosvm: sudo -H -u root /opt/pbs/bin/qmgr -c create resource foo_str2 flag=h,type=string 2017-05-17 13:39:15,453 INFO mom centosvm log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:39:15,851 INFO mom centosvm2 log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:39:16,314 INFO mom centosvm_nfs log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:39:16,314 INFO manager on centosvm as root: create resource foo_str3 {'flag': 'h', 'type': 'string'} 2017-05-17 13:39:16,314 INFOCLI centosvm: sudo -H -u root /opt/pbs/bin/qmgr -c create resource foo_str3 flag=h,type=string 2017-05-17 13:39:16,367 INFO mom centosvm log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:39:16,784 INFO mom centosvm2 log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:39:17,195 INFO mom centosvm_nfs log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:39:17,195 INFO manager on centosvm as root: create resource foo_str4 {'flag': 'h', 'type': 'string'} 2017-05-17 13:39:17,196 INFOCLI centosvm: sudo -H -u root /opt/pbs/bin/qmgr -c create resource foo_str4 flag=h,type=string 2017-05-17 13:39:17,232 INFO mom centosvm log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:39:17,653 INFO mom centosvm2 log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:39:18,084 INFO mom centosvm_nfs log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:39:18,085 INFO manager on centosvm as root: create resource stra {'flag': 'h', 'type': 'string_array'} 2017-05-17 13:39:18,085 INFOCLI centosvm: sudo -H -u root /opt/pbs/bin/qmgr -c create resource stra flag=h,type=string_array 2017-05-17 13:39:18,145 INFO mom centosvm log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:39:18,619 INFO mom centosvm2 log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:39:19,116 INFO mom centosvm_nfs log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:39:19,117 INFO test_epilogue 2017-05-17 13:39:19,117 INFO status on centosvm: hook 2017-05-17 13:39:19,117 INFO manager on centosvm: list hook 2017-05-17 13:39:19,118 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-05-17 13:39:19,170 INFO status on centosvm: hook 2017-05-17 13:39:19,171 INFO manager on centosvm: list hook 2017-05-17 13:39:19,171 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-05-17 13:39:19,214 INFO manager on centosvm: create hook epi 2017-05-17 13:39:19,214 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c create hook epi 2017-05-17 13:39:19,238 INFO manager on centosvm: set hook epi {'enabled': 'True', 'event': 'execjob_epilogue'} 2017-05-17 13:39:19,239 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c set hook epi enabled=True,event=execjob_epilogue 2017-05-17 13:39:19,269 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook epi 2017-05-17 13:39:19,290 INFO expect on server centosvm: enabled set True || event set execjob_epilogue hook epi ... OK 2017-05-17 13:39:19,291 INFO manager on centosvm: import hook epi {'content-type': 'application/x-python', 'input-file': '/tmp/PtlPbsk3ss8M', 'content-encoding': 'default'} 2017-05-17 13:39:19,291 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c import hook epi application/x-python default /tmp/PtlPbsk3ss8M 2017-05-17 13:39:19,319 INFO server centosvm: imported hook body --- import pbs e=pbs.event() pbs.logmsg(pbs.LOG_DEBUG, "executed epilogue hook") if e.job.in_ms_mom(): e.job.resources_used["vmem"] = pbs.size("9gb") e.job.resources_used["foo_i"] = 9 e.job.resources_used["foo_f"] = 0.09 e.job.resources_used["foo_str"] = '{"seven":7}' e.job.resources_used["cput"] = 10 e.job.resources_used["stra"] = '"glad,elated","happy"' e.job.resources_used["foo_str3"] = """{"a":6,"b":"some value #$%^&*@","c":54.4,"d":"32.5gb"}""" e.job.resources_used["foo_str2"] = "seven" e.job.resources_used["foo_str4"] = "eight" else: e.job.resources_used["vmem"] = pbs.size("10gb") e.job.resources_used["foo_i"] = 10 e.job.resources_used["foo_f"] = 0.10 e.job.resources_used["foo_str"] = '{"eight":8,"nine":9}' e.job.resources_used["foo_str2"] = '{"seven":7}' e.job.resources_used["cput"] = 20 e.job.resources_used["stra"] = '"cucumbers,bananas"' e.job.resources_used["foo_str3"] = """"vn1":4,"vn2":5,"vn3":6""" --- 2017-05-17 13:39:19,320 INFO job: executable set to /bin/sleep with arguments: 100 2017-05-17 13:39:19,320 INFO job: executable set to /bin/sleep with arguments: 10 2017-05-17 13:39:19,320 INFOCLI centosvm: sudo -H -u pbsuser /opt/pbs/bin/qsub -l place=scatter -l walltime=10 -l select=3:ncpus=1 -- /bin/sleep 10 2017-05-17 13:39:19,335 INFO submit to centosvm as pbsuser: job 2.centosvm OrderedDict([('Resource_List.place', 'scatter'), ('Resource_List.walltime', 10), ('Resource_List.select', '3:ncpus=1')]) 2017-05-17 13:39:19,335 INFO server centosvm: expect offset set to 10 2017-05-17 13:39:29,344 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 2.centosvm 2017-05-17 13:39:29,372 INFO expect on server centosvm: resources_used.stra = "glad,elated","happy" || resources_used.cput = 00:00:50 || resources_used.foo_str4 = eight || resources_used.foo_i = 29 || resources_used.ncpus = 3 || job_state = F || resources_used.foo_f = 0.29 || resources_used.vmem = 29gb job 2.centosvm 2017-05-17 13:39:29,874 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:39:29,921 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 2.centosvm 2017-05-17 13:39:29,952 INFO expect on server centosvm: resources_used.stra = "glad,elated","happy" || resources_used.cput = 00:00:50 || resources_used.foo_str4 = eight || resources_used.foo_i = 29 || resources_used.ncpus = 3 || job_state = F || resources_used.foo_f = 0.29 || resources_used.vmem = 29gb job 2.centosvm attempt: 2 2017-05-17 13:39:30,455 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:39:30,507 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 2.centosvm 2017-05-17 13:39:30,537 INFO expect on server centosvm: resources_used.stra = "glad,elated","happy" || resources_used.cput = 00:00:50 || resources_used.foo_str4 = eight || resources_used.foo_i = 29 || resources_used.ncpus = 3 || job_state = F || resources_used.foo_f = 0.29 || resources_used.vmem = 29gb job 2.centosvm attempt: 3 2017-05-17 13:39:31,040 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:39:31,096 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 2.centosvm 2017-05-17 13:39:31,130 INFO expect on server centosvm: resources_used.stra = "glad,elated","happy" || resources_used.cput = 00:00:50 || resources_used.foo_str4 = eight || resources_used.foo_i = 29 || resources_used.ncpus = 3 || job_state = F || resources_used.foo_f = 0.29 || resources_used.vmem = 29gb job 2.centosvm attempt: 4 ... OK 2017-05-17 13:39:31,130 INFO status on centosvm: job 2.centosvm resources_used.foo_str 2017-05-17 13:39:31,131 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 2.centosvm 2017-05-17 13:39:31,163 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 2.centosvm 2017-05-17 13:39:31,185 INFO expect on server centosvm: resources_used.foo_str3 unset job 2.centosvm ... OK 2017-05-17 13:39:31,186 INFO mom centosvm log match: searching for "Job 2.centosvm resources_used.foo_str3 cannot be accumulated: value '"vn1":4,"vn2":5,"vn3":6' from mom centosvm2 not JSON-format"... OK 2017-05-17 13:39:31,186 INFOCLI centosvm: /opt/pbs/bin/qstat -f 2.centosvm 2017-05-17 13:39:31,210 INFO expect on server centosvm: resources_used.foo_str2 unset job 2.centosvm ... OK 2017-05-17 13:39:31,210 INFO mom centosvm log match: searching for "Job 2.centosvm resources_used.foo_str2 cannot be accumulated: value 'seven' from mom centosvm not JSON-format"... OK 2017-05-17 13:39:31,231 INFO centosvm log match: searching for "E;2.centosvm;.*resources_used.foo_f=0.29.*" - using regular expression ... OK 2017-05-17 13:39:31,250 INFO centosvm log match: searching for "E;2.centosvm;.*resources_used.foo_i=29.*" - using regular expression ... OK 2017-05-17 13:39:31,272 INFO centosvm log match: searching for "E;2.centosvm;.*resources_used.foo_str='{"eight": 8, "seven": 7, "nine": 9}'.*" - using regular expression ... OK 2017-05-17 13:39:31,288 INFO centosvm log match: searching for "E;2.centosvm;.*resources_used.vmem=29gb.*" - using regular expression ... OK 2017-05-17 13:39:31,307 INFO centosvm log match: searching for "E;2.centosvm;.*resources_used.cput=00:00:50.*" - using regular expression ... OK 2017-05-17 13:39:31,318 INFO centosvm log match: searching for "E;2.centosvm;.*resources_used.foo_str2=.*" - using regular expression - No match 2017-05-17 13:39:32,355 INFO centosvm log match: searching for "E;2.centosvm;.*resources_used.foo_str4=eight.*" - using regular expression ... OK 2017-05-17 13:39:32,391 INFO centosvm log match: searching for "E;2.centosvm;.*resources_used.ncpus=3.*" - using regular expression ... OK 2017-05-17 13:39:32,417 INFO centosvm log match: searching for "E;2.centosvm;.*('resources_used.foo_str3=',).*" - using regular expression - No match 2017-05-17 13:39:33,430 INFO centosvm log match: searching for "E;2.centosvm;.*resources_used.stra="glad\,elated"\,"happy".*" - using regular expression ... OK 2017-05-17 13:39:33,430 INFO =========================================== 2017-05-17 13:39:33,430 INFO Entered TestPbsAccumulateRescUsed tearDown 2017-05-17 13:39:33,430 INFO =========================================== 2017-05-17 13:39:33,430 INFO ============================================ 2017-05-17 13:39:33,430 INFO Completed TestPbsAccumulateRescUsed tearDown 2017-05-17 13:39:33,430 INFO ============================================ 2017-05-17 13:39:33,431 INFO ok 2017-05-17 13:39:33,431 INFO test name: test_job_array (tests.functional.pbs_accumulate_resc_used.TestPbsAccumulateRescUsed)... 2017-05-17 13:39:33,431 INFO test start time: Wed May 17 13:39:33 2017 2017-05-17 13:39:33,431 INFO test docstring: Test that resource accumulation for subjobs also work 2017-05-17 13:39:33,431 INFO ======================================== 2017-05-17 13:39:33,432 INFO Entered TestPbsAccumulateRescUsed setUp 2017-05-17 13:39:33,432 INFO ======================================== 2017-05-17 13:39:33,432 INFOCLI centosvm: /opt/pbs/bin/qstat -Bf centosvm 2017-05-17 13:39:33,444 INFO status on centosvm: server 2017-05-17 13:39:33,444 INFOCLI centosvm: /opt/pbs/bin/qstat -Bf centosvm 2017-05-17 13:39:33,456 INFO manager on centosvm: set server {'managers': (3, 'dilipk@*')} 2017-05-17 13:39:33,456 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c set server managers-=dilipk@* 2017-05-17 13:39:33,479 INFO manager on centosvm: set server {'managers': (2, 'dilipk@*')} 2017-05-17 13:39:33,479 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c set server managers+=dilipk@* 2017-05-17 13:39:33,505 INFO server centosvm: reverting configuration to defaults 2017-05-17 13:39:33,505 INFOCLI centosvm: /opt/pbs/bin/qstat -Bf centosvm 2017-05-17 13:39:33,519 INFO select on centosvm: __ALL__ 2017-05-17 13:39:33,520 INFOCLI centosvm: /opt/pbs/bin/qselect -x 2017-05-17 13:39:33,532 INFO delete job on centosvm: 2.centosvm 2017-05-17 13:39:33,532 INFOCLI centosvm: /opt/pbs/bin/qdel -W force -x 2.centosvm 2017-05-17 13:39:33,545 INFOCLI centosvm: /opt/pbs/bin/qstat -f 2.centosvm 2017-05-17 13:39:33,562 INFOCLI centosvm: /opt/pbs/bin/qstat -f @centosvm 2017-05-17 13:39:33,573 INFO expect on server centosvm: job_state set 0 job ... OK 2017-05-17 13:39:33,573 INFOCLI centosvm: /opt/pbs/bin/pbs_rstat -f 2017-05-17 13:39:33,589 INFO manager on centosvm: unset server ['job_history_enable'] 2017-05-17 13:39:33,589 INFOCLI centosvm: /opt/pbs/bin/qmgr -c unset server job_history_enable 2017-05-17 13:39:33,604 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-05-17 13:39:33,631 INFO manager on centosvm: delete hook start,epi 2017-05-17 13:39:33,631 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c delete hook start,epi 2017-05-17 13:39:33,656 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook start 2017-05-17 13:39:33,677 INFO expect on server centosvm: unset hook start ... OK 2017-05-17 13:39:33,677 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook epi 2017-05-17 13:39:33,699 INFO expect on server centosvm: unset hook epi ... OK 2017-05-17 13:39:33,699 INFOCLI centosvm: /opt/pbs/bin/qstat -Qf @centosvm 2017-05-17 13:39:33,714 INFO manager on centosvm: delete queue workq 2017-05-17 13:39:33,714 INFOCLI centosvm: /opt/pbs/bin/qmgr -c delete queue workq 2017-05-17 13:39:33,728 INFOCLI centosvm: /opt/pbs/bin/qstat -Qf workq@centosvm 2017-05-17 13:39:33,740 INFO expect on server centosvm: unset queue workq ... OK 2017-05-17 13:39:33,740 INFO manager on centosvm: create queue workq {'started': 'True', 'queue_type': 'Execution', 'enabled': 'True'} 2017-05-17 13:39:33,741 INFOCLI centosvm: /opt/pbs/bin/qmgr -c create queue workq started=True,queue_type=Execution,enabled=True 2017-05-17 13:39:33,762 INFOCLI centosvm: /opt/pbs/bin/qstat -Qf workq@centosvm 2017-05-17 13:39:33,774 INFO expect on server centosvm: started set True || queue_type set Execution || enabled set True queue workq ... OK 2017-05-17 13:39:33,774 INFO manager on centosvm: set server {'default_queue': 'workq'} 2017-05-17 13:39:33,774 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server default_queue=workq 2017-05-17 13:39:33,786 INFO status on centosvm: resource 2017-05-17 13:39:33,786 INFO manager on centosvm: list resource 2017-05-17 13:39:33,786 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource 2017-05-17 13:39:33,800 INFO manager on centosvm: delete resource foo_f,foo_i,foo_str2,foo_str3,foo_str4,foo_str,stra 2017-05-17 13:39:33,801 INFOCLI centosvm: /opt/pbs/bin/qmgr -c delete resource foo_f,foo_i,foo_str2,foo_str3,foo_str4,foo_str,stra 2017-05-17 13:39:33,963 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_f 2017-05-17 13:39:33,974 INFO expect on server centosvm: unset resource foo_f ... OK 2017-05-17 13:39:33,974 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_i 2017-05-17 13:39:33,986 INFO expect on server centosvm: unset resource foo_i ... OK 2017-05-17 13:39:33,987 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_str2 2017-05-17 13:39:33,999 INFO expect on server centosvm: unset resource foo_str2 ... OK 2017-05-17 13:39:33,999 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_str3 2017-05-17 13:39:34,010 INFO expect on server centosvm: unset resource foo_str3 ... OK 2017-05-17 13:39:34,010 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_str4 2017-05-17 13:39:34,022 INFO expect on server centosvm: unset resource foo_str4 ... OK 2017-05-17 13:39:34,022 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_str 2017-05-17 13:39:34,033 INFO expect on server centosvm: unset resource foo_str ... OK 2017-05-17 13:39:34,033 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource stra 2017-05-17 13:39:34,049 INFO expect on server centosvm: unset resource stra ... OK 2017-05-17 13:39:34,049 INFOCLI status on centosvm: server license_count 2017-05-17 13:39:34,050 INFOCLI centosvm: /opt/pbs/bin/qstat -Bf centosvm 2017-05-17 13:39:34,061 INFO server: centosvm licensed 2017-05-17 13:39:34,076 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/server_priv/comm.lock 2017-05-17 13:39:34,100 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-05-17 13:39:34,114 INFO scheduler centosvm: reverting configuration to defaults 2017-05-17 13:39:34,114 INFO manager on centosvm: list sched 2017-05-17 13:39:34,114 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list sched 2017-05-17 13:39:34,126 INFO manager on centosvm: unset sched ['sched_cycle_length'] 2017-05-17 13:39:34,126 INFOCLI centosvm: /opt/pbs/bin/qmgr -c unset sched sched_cycle_length 2017-05-17 13:39:34,140 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/dedicated_time 2017-05-17 13:39:34,151 INFOCLI2 centosvm: cmp /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-05-17 13:39:34,155 ERROR err: ['cmp: /var/spool/pbs/sched_priv/resource_group: Permission denied'] 2017-05-17 13:39:34,155 INFOCLI2 centosvm: sudo -H /usr/bin/cp -p /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-05-17 13:39:34,166 INFOCLI2 centosvm: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-05-17 13:39:34,178 INFO scheduler centosvm: reverting holidays file to default 2017-05-17 13:39:34,178 INFOCLI2 centosvm: cmp /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-05-17 13:39:34,181 ERROR err: ['cmp: /var/spool/pbs/sched_priv/holidays: Permission denied'] 2017-05-17 13:39:34,181 INFOCLI2 centosvm: sudo -H /usr/bin/cp -p /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-05-17 13:39:34,197 INFOCLI2 centosvm: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/holidays 2017-05-17 13:39:34,209 INFO scheduler centosvm: Parsing holidays file 2017-05-17 13:39:34,209 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2017-05-17 13:39:34,221 INFOCLI2 centosvm: cmp /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-05-17 13:39:34,223 ERROR err: ['cmp: /var/spool/pbs/sched_priv/sched_config: Permission denied'] 2017-05-17 13:39:34,224 INFOCLI2 centosvm: sudo -H /usr/bin/cp -p /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-05-17 13:39:34,238 INFOCLI2 centosvm: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-05-17 13:39:34,249 INFO scheduler centosvm: sent signal -HUP 2017-05-17 13:39:34,249 INFOCLI2 centosvm: sudo -H kill -HUP 84281 2017-05-17 13:39:34,263 INFOCLI2 centosvm: sudo -H /usr/bin/rm /var/spool/pbs/sched_priv/usage 2017-05-17 13:39:34,274 ERROR err: ['/usr/bin/rm: cannot remove \xe2\x80\x98/var/spool/pbs/sched_priv/usage\xe2\x80\x99: No such file or directory'] 2017-05-17 13:39:34,305 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-05-17 13:39:34,316 INFOCLI2 centosvm: sudo -H cmp /opt/pbs/sbin/pbs_mom /opt/pbs/sbin/pbs_mom.cpuset 2017-05-17 13:39:34,341 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:39:34,352 INFO mom centosvm: reverting configuration to defaults 2017-05-17 13:39:34,352 INFOCLI2 centosvm: sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2017-05-17 13:39:34,364 INFOCLI2 centosvm: sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2017-05-17 13:39:34,375 INFOCLI2 centosvm: sudo -H ls /var/spool/pbs/mom_priv/config.d 2017-05-17 13:39:34,388 INFO mom centosvm: sent signal -HUP 2017-05-17 13:39:34,388 INFOCLI2 centosvm: sudo -H kill -HUP 84278 2017-05-17 13:39:34,412 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:39:34,424 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v -a 2017-05-17 13:39:34,440 INFO status on centosvm: node centosvm 2017-05-17 13:39:34,440 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:39:34,454 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:39:34,472 INFO expect on server centosvm: state = free node centosvm ... OK 2017-05-17 13:39:34,472 INFOCLI2 centosvm2: ssh centosvm2 sudo -H cmp /opt/pbs/sbin/pbs_mom /opt/pbs/sbin/pbs_mom.cpuset 2017-05-17 13:39:35,706 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:39:36,264 INFO mom centosvm2: reverting configuration to defaults 2017-05-17 13:39:36,265 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2017-05-17 13:39:36,842 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2017-05-17 13:39:37,318 INFOCLI2 centosvm2: ssh centosvm2 sudo -H ls /var/spool/pbs/mom_priv/config.d 2017-05-17 13:39:37,962 INFO mom centosvm2: sent signal -HUP 2017-05-17 13:39:37,964 INFOCLI2 centosvm2: ssh centosvm2 sudo -H kill -HUP 9078 2017-05-17 13:39:39,184 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:39:39,735 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v -a 2017-05-17 13:39:39,752 INFO status on centosvm: node centosvm2 2017-05-17 13:39:39,753 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm2 2017-05-17 13:39:39,768 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm2 2017-05-17 13:39:39,784 INFO expect on server centosvm: state = free node centosvm2 ... OK 2017-05-17 13:39:39,785 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H cmp /opt/pbs/sbin/pbs_mom /opt/pbs/sbin/pbs_mom.cpuset 2017-05-17 13:39:40,706 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:39:41,131 INFO mom centosvm_nfs: reverting configuration to defaults 2017-05-17 13:39:41,132 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2017-05-17 13:39:41,598 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2017-05-17 13:39:42,046 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H ls /var/spool/pbs/mom_priv/config.d 2017-05-17 13:39:42,459 INFO mom centosvm_nfs: sent signal -HUP 2017-05-17 13:39:42,459 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H kill -HUP 76746 2017-05-17 13:39:43,376 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:39:43,897 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v -a 2017-05-17 13:39:43,921 INFO status on centosvm: node centosvm_nfs 2017-05-17 13:39:43,921 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm_nfs 2017-05-17 13:39:43,965 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm_nfs 2017-05-17 13:39:43,981 INFO expect on server centosvm: state = free node centosvm_nfs ... OK 2017-05-17 13:39:43,981 INFO ========================================= 2017-05-17 13:39:43,982 INFO Completed TestPbsAccumulateRescUsed setUp 2017-05-17 13:39:43,982 INFO ========================================= 2017-05-17 13:39:43,982 INFO len moms = 3 2017-05-17 13:39:43,982 INFOCLI2 centosvm: sudo -H /usr/bin/rm -rf /var/spool/pbs/mom_priv/config.d 2017-05-17 13:39:44,008 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/rm -rf /var/spool/pbs/mom_priv/config.d 2017-05-17 13:39:44,411 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H /usr/bin/rm -rf /var/spool/pbs/mom_priv/config.d 2017-05-17 13:39:44,892 INFO manager on centosvm: delete node 2017-05-17 13:39:44,892 INFOCLI centosvm: /opt/pbs/bin/qmgr -c delete node @default 2017-05-17 13:39:44,916 INFO manager on centosvm: create node centosvm 2017-05-17 13:39:44,916 INFOCLI centosvm: /opt/pbs/bin/qmgr -c create node centosvm 2017-05-17 13:39:44,941 INFO manager on centosvm: create node centosvm2 2017-05-17 13:39:44,941 INFOCLI centosvm: /opt/pbs/bin/qmgr -c create node centosvm2 2017-05-17 13:39:44,957 INFO manager on centosvm: create node centosvm_nfs 2017-05-17 13:39:44,958 INFOCLI centosvm: /opt/pbs/bin/qmgr -c create node centosvm_nfs 2017-05-17 13:39:44,981 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:39:44,996 INFO expect on server centosvm: state = free node centosvm got: state = state-unknown,down 2017-05-17 13:39:45,498 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:39:45,537 INFO expect on server centosvm: state = free node centosvm attempt: 2 got: state = state-unknown,down 2017-05-17 13:39:46,039 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:39:46,058 INFO expect on server centosvm: state = free node centosvm attempt: 3 ... OK 2017-05-17 13:39:46,058 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm2 2017-05-17 13:39:46,073 INFO expect on server centosvm: state = free node centosvm2 got: state = initializing 2017-05-17 13:39:46,575 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm2 2017-05-17 13:39:46,618 INFO expect on server centosvm: state = free node centosvm2 attempt: 2 ... OK 2017-05-17 13:39:46,619 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm_nfs 2017-05-17 13:39:46,661 INFO expect on server centosvm: state = free node centosvm_nfs ... OK 2017-05-17 13:39:46,662 INFO status on centosvm: hook 2017-05-17 13:39:46,662 INFO manager on centosvm: list hook 2017-05-17 13:39:46,663 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-05-17 13:39:46,732 INFO status on centosvm: hook 2017-05-17 13:39:46,732 INFO manager on centosvm: list hook 2017-05-17 13:39:46,733 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-05-17 13:39:46,777 INFO manager on centosvm: create hook start 2017-05-17 13:39:46,778 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c create hook start 2017-05-17 13:39:46,814 INFO manager on centosvm: set hook start {'enabled': 'True', 'event': 'exechost_startup'} 2017-05-17 13:39:46,814 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c set hook start enabled=True,event=exechost_startup 2017-05-17 13:39:46,878 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook start 2017-05-17 13:39:46,904 INFO expect on server centosvm: enabled set True || event set exechost_startup hook start ... OK 2017-05-17 13:39:46,904 INFO manager on centosvm: import hook start {'content-type': 'application/x-python', 'input-file': '/tmp/PtlPbsK3ajPN', 'content-encoding': 'default'} 2017-05-17 13:39:46,905 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c import hook start application/x-python default /tmp/PtlPbsK3ajPN 2017-05-17 13:39:46,927 INFO server centosvm: imported hook body --- import pbs e=pbs.event() localnode=pbs.get_local_nodename() e.vnode_list[localnode].resources_available['foo_i'] = 7 e.vnode_list[localnode].resources_available['foo_f'] = 5.0 e.vnode_list[localnode].resources_available['foo_str'] = "seventyseven" --- 2017-05-17 13:39:46,927 INFO mom centosvm: sent signal -HUP 2017-05-17 13:39:46,928 INFOCLI2 centosvm: sudo -H kill -HUP 84278 2017-05-17 13:39:46,939 INFO mom centosvm2: sent signal -HUP 2017-05-17 13:39:46,939 INFOCLI2 centosvm2: ssh centosvm2 sudo -H kill -HUP 9078 2017-05-17 13:39:47,395 INFO mom centosvm_nfs: sent signal -HUP 2017-05-17 13:39:47,396 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H kill -HUP 76746 2017-05-17 13:39:48,010 INFO manager on centosvm: set server {'job_history_enable': 'True'} 2017-05-17 13:39:48,010 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server job_history_enable=True 2017-05-17 13:39:48,037 INFO manager on centosvm as root: create resource foo_str2 {'flag': 'h', 'type': 'string'} 2017-05-17 13:39:48,037 INFOCLI centosvm: sudo -H -u root /opt/pbs/bin/qmgr -c create resource foo_str2 flag=h,type=string 2017-05-17 13:39:48,122 INFO mom centosvm log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:39:48,541 INFO mom centosvm2 log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:39:49,090 INFO mom centosvm_nfs log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:39:49,092 INFO manager on centosvm as root: create resource foo_str3 {'flag': 'h', 'type': 'string'} 2017-05-17 13:39:49,093 INFOCLI centosvm: sudo -H -u root /opt/pbs/bin/qmgr -c create resource foo_str3 flag=h,type=string 2017-05-17 13:39:49,170 INFO mom centosvm log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:39:49,576 INFO mom centosvm2 log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:39:49,939 INFO mom centosvm_nfs log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:39:49,939 INFO manager on centosvm as root: create resource foo_str4 {'flag': 'h', 'type': 'string'} 2017-05-17 13:39:49,940 INFOCLI centosvm: sudo -H -u root /opt/pbs/bin/qmgr -c create resource foo_str4 flag=h,type=string 2017-05-17 13:39:49,976 INFO mom centosvm log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:39:50,458 INFO mom centosvm2 log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:39:50,941 INFO mom centosvm_nfs log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:39:50,942 INFO manager on centosvm as root: create resource stra {'flag': 'h', 'type': 'string_array'} 2017-05-17 13:39:50,943 INFOCLI centosvm: sudo -H -u root /opt/pbs/bin/qmgr -c create resource stra flag=h,type=string_array 2017-05-17 13:39:51,030 INFO mom centosvm log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:39:51,413 INFO mom centosvm2 log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:39:51,813 INFO mom centosvm_nfs log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:39:51,814 INFO status on centosvm: hook 2017-05-17 13:39:51,814 INFO manager on centosvm: list hook 2017-05-17 13:39:51,815 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-05-17 13:39:51,876 INFO status on centosvm: hook 2017-05-17 13:39:51,877 INFO manager on centosvm: list hook 2017-05-17 13:39:51,877 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-05-17 13:39:51,919 INFO manager on centosvm: create hook test 2017-05-17 13:39:51,919 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c create hook test 2017-05-17 13:39:51,971 INFO manager on centosvm: set hook test {'enabled': 'True', 'event': 'execjob_epilogue'} 2017-05-17 13:39:51,972 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c set hook test enabled=True,event=execjob_epilogue 2017-05-17 13:39:51,999 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook test 2017-05-17 13:39:52,029 INFO expect on server centosvm: enabled set True || event set execjob_epilogue hook test ... OK 2017-05-17 13:39:52,029 INFO manager on centosvm: import hook test {'content-type': 'application/x-python', 'input-file': '/tmp/PtlPbsu4eSl5', 'content-encoding': 'default'} 2017-05-17 13:39:52,029 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c import hook test application/x-python default /tmp/PtlPbsu4eSl5 2017-05-17 13:39:52,054 INFO server centosvm: imported hook body --- import pbs e=pbs.event() pbs.logmsg(pbs.LOG_DEBUG, "executed epilogue hook") if e.job.in_ms_mom(): e.job.resources_used["vmem"] = pbs.size("9gb") e.job.resources_used["foo_i"] = 9 e.job.resources_used["foo_f"] = 0.09 e.job.resources_used["foo_str"] = '{"seven":7}' e.job.resources_used["cput"] = 10 e.job.resources_used["stra"] = '"glad,elated","happy"' else: e.job.resources_used["vmem"] = pbs.size("10gb") e.job.resources_used["foo_i"] = 10 e.job.resources_used["foo_f"] = 0.10 e.job.resources_used["foo_str"] = '{"eight":8,"nine":9}' e.job.resources_used["cput"] = 20 e.job.resources_used["stra"] = '"cucumbers,bananas"' --- 2017-05-17 13:39:52,054 INFO job: executable set to /bin/sleep with arguments: 100 2017-05-17 13:39:52,055 INFO job: executable set to /bin/sleep with arguments: 5 2017-05-17 13:39:52,055 INFOCLI centosvm: sudo -H -u pbsuser /opt/pbs/bin/qsub -J 1-2 -l place=scatter -l walltime=10 -l select=3:ncpus=1 -- /bin/sleep 5 2017-05-17 13:39:52,082 INFO submit to centosvm as pbsuser: job 3[].centosvm OrderedDict([('array_indices_submitted', '1-2'), ('Resource_List.place', 'scatter'), ('Resource_List.walltime', 10), ('Resource_List.select', '3:ncpus=1')]) 2017-05-17 13:39:52,082 INFO =========================================== 2017-05-17 13:39:52,082 INFO Entered TestPbsAccumulateRescUsed tearDown 2017-05-17 13:39:52,082 INFO =========================================== 2017-05-17 13:39:52,083 INFO ============================================ 2017-05-17 13:39:52,083 INFO Completed TestPbsAccumulateRescUsed tearDown 2017-05-17 13:39:52,083 INFO ============================================ 2017-05-17 13:39:52,083 INFO ok 2017-05-17 13:39:52,083 INFO test name: test_job_rerun (tests.functional.pbs_accumulate_resc_used.TestPbsAccumulateRescUsed)... 2017-05-17 13:39:52,083 INFO test start time: Wed May 17 13:39:52 2017 2017-05-17 13:39:52,083 INFO test docstring: Test that resource accumulates once when job is rerun 2017-05-17 13:39:52,083 INFO ======================================== 2017-05-17 13:39:52,083 INFO Entered TestPbsAccumulateRescUsed setUp 2017-05-17 13:39:52,084 INFO ======================================== 2017-05-17 13:39:52,084 INFOCLI centosvm: /opt/pbs/bin/qstat -Bf centosvm 2017-05-17 13:39:52,094 INFO status on centosvm: server 2017-05-17 13:39:52,094 INFOCLI centosvm: /opt/pbs/bin/qstat -Bf centosvm 2017-05-17 13:39:52,109 INFO manager on centosvm: set server {'managers': (3, 'dilipk@*')} 2017-05-17 13:39:52,109 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c set server managers-=dilipk@* 2017-05-17 13:39:52,130 INFO manager on centosvm: set server {'managers': (2, 'dilipk@*')} 2017-05-17 13:39:52,131 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c set server managers+=dilipk@* 2017-05-17 13:39:52,153 INFO server centosvm: reverting configuration to defaults 2017-05-17 13:39:52,153 INFOCLI centosvm: /opt/pbs/bin/qstat -Bf centosvm 2017-05-17 13:39:52,164 INFO select on centosvm: __ALL__ 2017-05-17 13:39:52,164 INFOCLI centosvm: /opt/pbs/bin/qselect -x 2017-05-17 13:39:52,173 INFO delete job on centosvm: 3[].centosvm 2017-05-17 13:39:52,174 INFOCLI centosvm: /opt/pbs/bin/qdel -W force -x 3[].centosvm 2017-05-17 13:39:52,191 INFOCLI centosvm: /opt/pbs/bin/qstat -f 3[].centosvm 2017-05-17 13:39:52,206 INFOCLI centosvm: /opt/pbs/bin/qstat -f @centosvm 2017-05-17 13:39:52,223 INFO expect on server centosvm: job_state set 0 job ... OK 2017-05-17 13:39:52,223 INFOCLI centosvm: /opt/pbs/bin/pbs_rstat -f 2017-05-17 13:39:52,234 INFO manager on centosvm: unset server ['job_history_enable'] 2017-05-17 13:39:52,234 INFOCLI centosvm: /opt/pbs/bin/qmgr -c unset server job_history_enable 2017-05-17 13:39:52,249 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-05-17 13:39:52,275 INFO manager on centosvm: delete hook test,start 2017-05-17 13:39:52,275 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c delete hook test,start 2017-05-17 13:39:52,299 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook test 2017-05-17 13:39:52,320 INFO expect on server centosvm: unset hook test ... OK 2017-05-17 13:39:52,321 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook start 2017-05-17 13:39:52,342 INFO expect on server centosvm: unset hook start ... OK 2017-05-17 13:39:52,342 INFOCLI centosvm: /opt/pbs/bin/qstat -Qf @centosvm 2017-05-17 13:39:52,359 INFO manager on centosvm: delete queue workq 2017-05-17 13:39:52,359 INFOCLI centosvm: /opt/pbs/bin/qmgr -c delete queue workq 2017-05-17 13:39:52,371 INFOCLI centosvm: /opt/pbs/bin/qstat -Qf workq@centosvm 2017-05-17 13:39:52,383 INFO expect on server centosvm: unset queue workq ... OK 2017-05-17 13:39:52,383 INFO manager on centosvm: create queue workq {'started': 'True', 'queue_type': 'Execution', 'enabled': 'True'} 2017-05-17 13:39:52,383 INFOCLI centosvm: /opt/pbs/bin/qmgr -c create queue workq started=True,queue_type=Execution,enabled=True 2017-05-17 13:39:52,398 INFOCLI centosvm: /opt/pbs/bin/qstat -Qf workq@centosvm 2017-05-17 13:39:52,409 INFO expect on server centosvm: started set True || queue_type set Execution || enabled set True queue workq ... OK 2017-05-17 13:39:52,409 INFO manager on centosvm: set server {'default_queue': 'workq'} 2017-05-17 13:39:52,409 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server default_queue=workq 2017-05-17 13:39:52,423 INFO status on centosvm: resource 2017-05-17 13:39:52,423 INFO manager on centosvm: list resource 2017-05-17 13:39:52,424 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource 2017-05-17 13:39:52,435 INFO manager on centosvm: delete resource foo_f,foo_i,foo_str2,foo_str3,foo_str4,foo_str,stra 2017-05-17 13:39:52,435 INFOCLI centosvm: /opt/pbs/bin/qmgr -c delete resource foo_f,foo_i,foo_str2,foo_str3,foo_str4,foo_str,stra 2017-05-17 13:39:52,603 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_f 2017-05-17 13:39:52,614 INFO expect on server centosvm: unset resource foo_f ... OK 2017-05-17 13:39:52,614 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_i 2017-05-17 13:39:52,626 INFO expect on server centosvm: unset resource foo_i ... OK 2017-05-17 13:39:52,626 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_str2 2017-05-17 13:39:52,638 INFO expect on server centosvm: unset resource foo_str2 ... OK 2017-05-17 13:39:52,639 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_str3 2017-05-17 13:39:52,650 INFO expect on server centosvm: unset resource foo_str3 ... OK 2017-05-17 13:39:52,651 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_str4 2017-05-17 13:39:52,661 INFO expect on server centosvm: unset resource foo_str4 ... OK 2017-05-17 13:39:52,661 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_str 2017-05-17 13:39:52,680 INFO expect on server centosvm: unset resource foo_str ... OK 2017-05-17 13:39:52,680 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource stra 2017-05-17 13:39:52,695 INFO expect on server centosvm: unset resource stra ... OK 2017-05-17 13:39:52,695 INFOCLI status on centosvm: server license_count 2017-05-17 13:39:52,695 INFOCLI centosvm: /opt/pbs/bin/qstat -Bf centosvm 2017-05-17 13:39:52,706 INFO server: centosvm licensed 2017-05-17 13:39:52,740 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/server_priv/comm.lock 2017-05-17 13:39:52,770 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-05-17 13:39:52,780 INFO scheduler centosvm: reverting configuration to defaults 2017-05-17 13:39:52,780 INFO manager on centosvm: list sched 2017-05-17 13:39:52,780 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list sched 2017-05-17 13:39:52,797 INFO manager on centosvm: unset sched ['sched_cycle_length'] 2017-05-17 13:39:52,797 INFOCLI centosvm: /opt/pbs/bin/qmgr -c unset sched sched_cycle_length 2017-05-17 13:39:52,811 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/dedicated_time 2017-05-17 13:39:52,824 INFOCLI2 centosvm: cmp /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-05-17 13:39:52,827 ERROR err: ['cmp: /var/spool/pbs/sched_priv/resource_group: Permission denied'] 2017-05-17 13:39:52,827 INFOCLI2 centosvm: sudo -H /usr/bin/cp -p /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-05-17 13:39:52,837 INFOCLI2 centosvm: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-05-17 13:39:52,849 INFO scheduler centosvm: reverting holidays file to default 2017-05-17 13:39:52,849 INFOCLI2 centosvm: cmp /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-05-17 13:39:52,852 ERROR err: ['cmp: /var/spool/pbs/sched_priv/holidays: Permission denied'] 2017-05-17 13:39:52,852 INFOCLI2 centosvm: sudo -H /usr/bin/cp -p /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-05-17 13:39:52,864 INFOCLI2 centosvm: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/holidays 2017-05-17 13:39:52,875 INFO scheduler centosvm: Parsing holidays file 2017-05-17 13:39:52,875 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2017-05-17 13:39:52,888 INFOCLI2 centosvm: cmp /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-05-17 13:39:52,892 ERROR err: ['cmp: /var/spool/pbs/sched_priv/sched_config: Permission denied'] 2017-05-17 13:39:52,892 INFOCLI2 centosvm: sudo -H /usr/bin/cp -p /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-05-17 13:39:52,903 INFOCLI2 centosvm: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-05-17 13:39:52,913 INFO scheduler centosvm: sent signal -HUP 2017-05-17 13:39:52,913 INFOCLI2 centosvm: sudo -H kill -HUP 84281 2017-05-17 13:39:52,925 INFOCLI2 centosvm: sudo -H /usr/bin/rm /var/spool/pbs/sched_priv/usage 2017-05-17 13:39:52,939 ERROR err: ['/usr/bin/rm: cannot remove \xe2\x80\x98/var/spool/pbs/sched_priv/usage\xe2\x80\x99: No such file or directory'] 2017-05-17 13:39:52,967 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-05-17 13:39:52,978 INFOCLI2 centosvm: sudo -H cmp /opt/pbs/sbin/pbs_mom /opt/pbs/sbin/pbs_mom.cpuset 2017-05-17 13:39:53,005 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:39:53,020 INFO mom centosvm: reverting configuration to defaults 2017-05-17 13:39:53,021 INFOCLI2 centosvm: sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2017-05-17 13:39:53,034 INFOCLI2 centosvm: sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2017-05-17 13:39:53,045 INFOCLI2 centosvm: sudo -H ls /var/spool/pbs/mom_priv/config.d 2017-05-17 13:39:53,056 INFO mom centosvm: sent signal -HUP 2017-05-17 13:39:53,056 INFOCLI2 centosvm: sudo -H kill -HUP 84278 2017-05-17 13:39:53,089 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:39:53,101 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v -a 2017-05-17 13:39:53,125 INFO status on centosvm: node centosvm 2017-05-17 13:39:53,125 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:39:53,136 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:39:53,147 INFO expect on server centosvm: state = free node centosvm ... OK 2017-05-17 13:39:53,147 INFOCLI2 centosvm2: ssh centosvm2 sudo -H cmp /opt/pbs/sbin/pbs_mom /opt/pbs/sbin/pbs_mom.cpuset 2017-05-17 13:39:54,189 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:39:54,989 INFO mom centosvm2: reverting configuration to defaults 2017-05-17 13:39:54,990 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2017-05-17 13:39:55,481 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2017-05-17 13:39:56,130 INFOCLI2 centosvm2: ssh centosvm2 sudo -H ls /var/spool/pbs/mom_priv/config.d 2017-05-17 13:39:56,612 INFO mom centosvm2: sent signal -HUP 2017-05-17 13:39:56,613 INFOCLI2 centosvm2: ssh centosvm2 sudo -H kill -HUP 9078 2017-05-17 13:39:57,579 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:39:58,014 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v -a 2017-05-17 13:39:58,027 INFO status on centosvm: node centosvm2 2017-05-17 13:39:58,027 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm2 2017-05-17 13:39:58,045 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm2 2017-05-17 13:39:58,061 INFO expect on server centosvm: state = free node centosvm2 ... OK 2017-05-17 13:39:58,062 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H cmp /opt/pbs/sbin/pbs_mom /opt/pbs/sbin/pbs_mom.cpuset 2017-05-17 13:39:59,154 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:39:59,546 INFO mom centosvm_nfs: reverting configuration to defaults 2017-05-17 13:39:59,546 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2017-05-17 13:40:00,011 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2017-05-17 13:40:00,404 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H ls /var/spool/pbs/mom_priv/config.d 2017-05-17 13:40:00,882 INFO mom centosvm_nfs: sent signal -HUP 2017-05-17 13:40:00,882 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H kill -HUP 76746 2017-05-17 13:40:02,197 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:40:02,702 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v -a 2017-05-17 13:40:02,717 INFO status on centosvm: node centosvm_nfs 2017-05-17 13:40:02,717 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm_nfs 2017-05-17 13:40:02,732 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm_nfs 2017-05-17 13:40:02,748 INFO expect on server centosvm: state = free node centosvm_nfs ... OK 2017-05-17 13:40:02,748 INFO ========================================= 2017-05-17 13:40:02,748 INFO Completed TestPbsAccumulateRescUsed setUp 2017-05-17 13:40:02,748 INFO ========================================= 2017-05-17 13:40:02,749 INFO len moms = 3 2017-05-17 13:40:02,749 INFOCLI2 centosvm: sudo -H /usr/bin/rm -rf /var/spool/pbs/mom_priv/config.d 2017-05-17 13:40:02,764 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/rm -rf /var/spool/pbs/mom_priv/config.d 2017-05-17 13:40:03,270 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H /usr/bin/rm -rf /var/spool/pbs/mom_priv/config.d 2017-05-17 13:40:03,703 INFO manager on centosvm: delete node 2017-05-17 13:40:03,703 INFOCLI centosvm: /opt/pbs/bin/qmgr -c delete node @default 2017-05-17 13:40:03,727 INFO manager on centosvm: create node centosvm 2017-05-17 13:40:03,727 INFOCLI centosvm: /opt/pbs/bin/qmgr -c create node centosvm 2017-05-17 13:40:03,747 INFO manager on centosvm: create node centosvm2 2017-05-17 13:40:03,748 INFOCLI centosvm: /opt/pbs/bin/qmgr -c create node centosvm2 2017-05-17 13:40:03,764 INFO manager on centosvm: create node centosvm_nfs 2017-05-17 13:40:03,764 INFOCLI centosvm: /opt/pbs/bin/qmgr -c create node centosvm_nfs 2017-05-17 13:40:03,779 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:40:03,792 INFO expect on server centosvm: state = free node centosvm got: state = state-unknown,down 2017-05-17 13:40:04,293 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:40:04,306 INFO expect on server centosvm: state = free node centosvm attempt: 2 got: state = state-unknown,down 2017-05-17 13:40:04,808 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:40:04,850 INFO expect on server centosvm: state = free node centosvm attempt: 3 got: state = state-unknown,down 2017-05-17 13:40:05,352 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:40:05,379 INFO expect on server centosvm: state = free node centosvm attempt: 4 ... OK 2017-05-17 13:40:05,379 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm2 2017-05-17 13:40:05,398 INFO expect on server centosvm: state = free node centosvm2 got: state = initializing 2017-05-17 13:40:05,899 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm2 2017-05-17 13:40:05,941 INFO expect on server centosvm: state = free node centosvm2 attempt: 2 ... OK 2017-05-17 13:40:05,942 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm_nfs 2017-05-17 13:40:05,981 INFO expect on server centosvm: state = free node centosvm_nfs ... OK 2017-05-17 13:40:05,981 INFO status on centosvm: hook 2017-05-17 13:40:05,982 INFO manager on centosvm: list hook 2017-05-17 13:40:05,982 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-05-17 13:40:06,060 INFO status on centosvm: hook 2017-05-17 13:40:06,061 INFO manager on centosvm: list hook 2017-05-17 13:40:06,062 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-05-17 13:40:06,143 INFO manager on centosvm: create hook start 2017-05-17 13:40:06,144 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c create hook start 2017-05-17 13:40:06,234 INFO manager on centosvm: set hook start {'enabled': 'True', 'event': 'exechost_startup'} 2017-05-17 13:40:06,234 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c set hook start enabled=True,event=exechost_startup 2017-05-17 13:40:06,278 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook start 2017-05-17 13:40:06,318 INFO expect on server centosvm: enabled set True || event set exechost_startup hook start ... OK 2017-05-17 13:40:06,319 INFO manager on centosvm: import hook start {'content-type': 'application/x-python', 'input-file': '/tmp/PtlPbsAjLLlN', 'content-encoding': 'default'} 2017-05-17 13:40:06,320 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c import hook start application/x-python default /tmp/PtlPbsAjLLlN 2017-05-17 13:40:06,351 INFO server centosvm: imported hook body --- import pbs e=pbs.event() localnode=pbs.get_local_nodename() e.vnode_list[localnode].resources_available['foo_i'] = 7 e.vnode_list[localnode].resources_available['foo_f'] = 5.0 e.vnode_list[localnode].resources_available['foo_str'] = "seventyseven" --- 2017-05-17 13:40:06,351 INFO mom centosvm: sent signal -HUP 2017-05-17 13:40:06,351 INFOCLI2 centosvm: sudo -H kill -HUP 84278 2017-05-17 13:40:06,365 INFO mom centosvm2: sent signal -HUP 2017-05-17 13:40:06,366 INFOCLI2 centosvm2: ssh centosvm2 sudo -H kill -HUP 9078 2017-05-17 13:40:06,974 INFO mom centosvm_nfs: sent signal -HUP 2017-05-17 13:40:06,975 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H kill -HUP 76746 2017-05-17 13:40:07,446 INFO manager on centosvm: set server {'job_history_enable': 'True'} 2017-05-17 13:40:07,447 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server job_history_enable=True 2017-05-17 13:40:07,501 INFO manager on centosvm as root: create resource foo_str2 {'flag': 'h', 'type': 'string'} 2017-05-17 13:40:07,501 INFOCLI centosvm: sudo -H -u root /opt/pbs/bin/qmgr -c create resource foo_str2 flag=h,type=string 2017-05-17 13:40:07,554 INFO mom centosvm log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:40:08,023 INFO mom centosvm2 log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:40:08,475 INFO mom centosvm_nfs log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:40:08,476 INFO manager on centosvm as root: create resource foo_str3 {'flag': 'h', 'type': 'string'} 2017-05-17 13:40:08,477 INFOCLI centosvm: sudo -H -u root /opt/pbs/bin/qmgr -c create resource foo_str3 flag=h,type=string 2017-05-17 13:40:08,562 INFO mom centosvm log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:40:09,005 INFO mom centosvm2 log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:40:09,583 INFO mom centosvm_nfs log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:40:09,584 INFO manager on centosvm as root: create resource foo_str4 {'flag': 'h', 'type': 'string'} 2017-05-17 13:40:09,585 INFOCLI centosvm: sudo -H -u root /opt/pbs/bin/qmgr -c create resource foo_str4 flag=h,type=string 2017-05-17 13:40:09,662 INFO mom centosvm log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:40:10,091 INFO mom centosvm2 log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:40:10,640 INFO mom centosvm_nfs log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:40:10,640 INFO manager on centosvm as root: create resource stra {'flag': 'h', 'type': 'string_array'} 2017-05-17 13:40:10,640 INFOCLI centosvm: sudo -H -u root /opt/pbs/bin/qmgr -c create resource stra flag=h,type=string_array 2017-05-17 13:40:10,698 INFO mom centosvm log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:40:11,153 INFO mom centosvm2 log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:40:11,565 INFO mom centosvm_nfs log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:40:11,566 INFO status on centosvm: hook 2017-05-17 13:40:11,566 INFO manager on centosvm: list hook 2017-05-17 13:40:11,567 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-05-17 13:40:11,627 INFO status on centosvm: hook 2017-05-17 13:40:11,628 INFO manager on centosvm: list hook 2017-05-17 13:40:11,628 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-05-17 13:40:11,660 INFO manager on centosvm: create hook period 2017-05-17 13:40:11,660 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c create hook period 2017-05-17 13:40:11,688 INFO manager on centosvm: set hook period {'freq': 10, 'enabled': 'True', 'event': 'exechost_periodic'} 2017-05-17 13:40:11,689 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c set hook period freq=10,enabled=True,event=exechost_periodic 2017-05-17 13:40:11,720 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook period 2017-05-17 13:40:11,743 INFO expect on server centosvm: freq set 10 || enabled set True || event set exechost_periodic hook period ... OK 2017-05-17 13:40:11,744 INFO manager on centosvm: import hook period {'content-type': 'application/x-python', 'input-file': '/tmp/PtlPbs9UOI4I', 'content-encoding': 'default'} 2017-05-17 13:40:11,744 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c import hook period application/x-python default /tmp/PtlPbs9UOI4I 2017-05-17 13:40:11,766 INFO server centosvm: imported hook body --- import pbs e = pbs.event() pbs.logmsg(pbs.LOG_DEBUG, "executed periodic hook") for jj in e.job_list.keys(): e.job_list[jj].resources_used["foo_f"] = 1.01 e.job_list[jj].resources_used["cput"] = 10 --- 2017-05-17 13:40:11,767 INFO job: executable set to /bin/sleep with arguments: 100 2017-05-17 13:40:11,767 INFOCLI centosvm: sudo -H -u pbsuser /opt/pbs/bin/qsub -l place=scatter -l select=3:ncpus=1 -- /bin/sleep 100 2017-05-17 13:40:11,782 INFO submit to centosvm as pbsuser: job 4.centosvm OrderedDict([('Resource_List.place', 'scatter'), ('Resource_List.select', '3:ncpus=1')]) 2017-05-17 13:40:11,782 INFOCLI centosvm: /opt/pbs/bin/qstat -f 4.centosvm 2017-05-17 13:40:11,802 INFO expect on server centosvm: job_state = R || substate = 42 job 4.centosvm got: job_state = Q 2017-05-17 13:40:12,304 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:40:12,330 INFOCLI centosvm: /opt/pbs/bin/qstat -f 4.centosvm 2017-05-17 13:40:12,356 INFO expect on server centosvm: job_state = R || substate = 42 job 4.centosvm attempt: 2 got: substate = 41 2017-05-17 13:40:12,857 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:40:12,893 INFOCLI centosvm: /opt/pbs/bin/qstat -f 4.centosvm 2017-05-17 13:40:12,916 INFO expect on server centosvm: job_state = R || substate = 42 job 4.centosvm attempt: 3 ... OK 2017-05-17 13:40:12,917 INFO server centosvm: expect offset set to 10 2017-05-17 13:40:22,928 INFOCLI centosvm: /opt/pbs/bin/qstat -f 4.centosvm 2017-05-17 13:40:22,961 INFO expect on server centosvm: resources_used.cput = 30 || resources_used.foo_f = 3.03 job 4.centosvm got: resources_used.cput = 0 2017-05-17 13:40:23,966 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:40:24,012 INFOCLI centosvm: /opt/pbs/bin/qstat -f 4.centosvm 2017-05-17 13:40:24,044 INFO expect on server centosvm: resources_used.cput = 30 || resources_used.foo_f = 3.03 job 4.centosvm attempt: 2 got: resources_used.cput = 10 2017-05-17 13:40:25,049 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:40:25,094 INFOCLI centosvm: /opt/pbs/bin/qstat -f 4.centosvm 2017-05-17 13:40:25,130 INFO expect on server centosvm: resources_used.cput = 30 || resources_used.foo_f = 3.03 job 4.centosvm attempt: 3 got: resources_used.cput = 10 2017-05-17 13:40:26,135 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:40:26,245 INFOCLI centosvm: /opt/pbs/bin/qstat -f 4.centosvm 2017-05-17 13:40:26,261 INFO expect on server centosvm: resources_used.cput = 30 || resources_used.foo_f = 3.03 job 4.centosvm attempt: 4 got: resources_used.cput = 10 2017-05-17 13:40:27,265 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:40:27,307 INFOCLI centosvm: /opt/pbs/bin/qstat -f 4.centosvm 2017-05-17 13:40:27,348 INFO expect on server centosvm: resources_used.cput = 30 || resources_used.foo_f = 3.03 job 4.centosvm attempt: 5 got: resources_used.cput = 10 2017-05-17 13:40:28,352 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:40:28,397 INFOCLI centosvm: /opt/pbs/bin/qstat -f 4.centosvm 2017-05-17 13:40:28,437 INFO expect on server centosvm: resources_used.cput = 30 || resources_used.foo_f = 3.03 job 4.centosvm attempt: 6 got: resources_used.cput = 10 2017-05-17 13:40:29,441 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:40:29,491 INFOCLI centosvm: /opt/pbs/bin/qstat -f 4.centosvm 2017-05-17 13:40:29,527 INFO expect on server centosvm: resources_used.cput = 30 || resources_used.foo_f = 3.03 job 4.centosvm attempt: 7 got: resources_used.cput = 10 2017-05-17 13:40:30,532 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:40:30,576 INFOCLI centosvm: /opt/pbs/bin/qstat -f 4.centosvm 2017-05-17 13:40:30,606 INFO expect on server centosvm: resources_used.cput = 30 || resources_used.foo_f = 3.03 job 4.centosvm attempt: 8 got: resources_used.cput = 10 2017-05-17 13:40:31,611 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:40:31,661 INFOCLI centosvm: /opt/pbs/bin/qstat -f 4.centosvm 2017-05-17 13:40:31,694 INFO expect on server centosvm: resources_used.cput = 30 || resources_used.foo_f = 3.03 job 4.centosvm attempt: 9 got: resources_used.cput = 10 2017-05-17 13:40:32,697 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:40:32,742 INFOCLI centosvm: /opt/pbs/bin/qstat -f 4.centosvm 2017-05-17 13:40:32,772 INFO expect on server centosvm: resources_used.cput = 30 || resources_used.foo_f = 3.03 job 4.centosvm attempt: 10 got: resources_used.cput = 10 2017-05-17 13:40:33,774 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:40:33,820 INFOCLI centosvm: /opt/pbs/bin/qstat -f 4.centosvm 2017-05-17 13:40:33,862 INFO expect on server centosvm: resources_used.cput = 30 || resources_used.foo_f = 3.03 job 4.centosvm attempt: 11 got: resources_used.cput = 10 2017-05-17 13:40:34,867 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:40:34,911 INFOCLI centosvm: /opt/pbs/bin/qstat -f 4.centosvm 2017-05-17 13:40:34,942 INFO expect on server centosvm: resources_used.cput = 30 || resources_used.foo_f = 3.03 job 4.centosvm attempt: 12 got: resources_used.cput = 10 2017-05-17 13:40:35,946 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:40:35,994 INFOCLI centosvm: /opt/pbs/bin/qstat -f 4.centosvm 2017-05-17 13:40:36,034 INFO expect on server centosvm: resources_used.cput = 30 || resources_used.foo_f = 3.03 job 4.centosvm attempt: 13 got: resources_used.cput = 10 2017-05-17 13:40:37,037 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:40:37,109 INFOCLI centosvm: /opt/pbs/bin/qstat -f 4.centosvm 2017-05-17 13:40:37,132 INFO expect on server centosvm: resources_used.cput = 30 || resources_used.foo_f = 3.03 job 4.centosvm attempt: 14 got: resources_used.cput = 10 2017-05-17 13:40:38,137 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:40:38,186 INFOCLI centosvm: /opt/pbs/bin/qstat -f 4.centosvm 2017-05-17 13:40:38,215 INFO expect on server centosvm: resources_used.cput = 30 || resources_used.foo_f = 3.03 job 4.centosvm attempt: 15 got: resources_used.cput = 10 2017-05-17 13:40:39,218 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:40:39,264 INFOCLI centosvm: /opt/pbs/bin/qstat -f 4.centosvm 2017-05-17 13:40:39,306 INFO expect on server centosvm: resources_used.cput = 30 || resources_used.foo_f = 3.03 job 4.centosvm attempt: 16 got: resources_used.cput = 10 2017-05-17 13:40:40,312 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:40:40,360 INFOCLI centosvm: /opt/pbs/bin/qstat -f 4.centosvm 2017-05-17 13:40:40,405 INFO expect on server centosvm: resources_used.cput = 30 || resources_used.foo_f = 3.03 job 4.centosvm attempt: 17 got: resources_used.cput = 10 2017-05-17 13:40:41,408 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:40:41,455 INFOCLI centosvm: /opt/pbs/bin/qstat -f 4.centosvm 2017-05-17 13:40:41,475 INFO expect on server centosvm: resources_used.cput = 30 || resources_used.foo_f = 3.03 job 4.centosvm attempt: 18 got: resources_used.cput = 10 2017-05-17 13:40:42,478 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:40:42,523 INFOCLI centosvm: /opt/pbs/bin/qstat -f 4.centosvm 2017-05-17 13:40:42,566 INFO expect on server centosvm: resources_used.cput = 30 || resources_used.foo_f = 3.03 job 4.centosvm attempt: 19 got: resources_used.cput = 10 2017-05-17 13:40:43,568 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:40:43,619 INFOCLI centosvm: /opt/pbs/bin/qstat -f 4.centosvm 2017-05-17 13:40:43,666 INFO expect on server centosvm: resources_used.cput = 30 || resources_used.foo_f = 3.03 job 4.centosvm attempt: 20 got: resources_used.cput = 10 2017-05-17 13:40:44,668 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:40:44,717 INFOCLI centosvm: /opt/pbs/bin/qstat -f 4.centosvm 2017-05-17 13:40:44,745 INFO expect on server centosvm: resources_used.cput = 30 || resources_used.foo_f = 3.03 job 4.centosvm attempt: 21 got: resources_used.cput = 10 2017-05-17 13:40:45,747 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:40:45,798 INFOCLI centosvm: /opt/pbs/bin/qstat -f 4.centosvm 2017-05-17 13:40:45,834 INFO expect on server centosvm: resources_used.cput = 30 || resources_used.foo_f = 3.03 job 4.centosvm attempt: 22 got: resources_used.cput = 10 2017-05-17 13:40:46,838 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:40:46,887 INFOCLI centosvm: /opt/pbs/bin/qstat -f 4.centosvm 2017-05-17 13:40:46,913 INFO expect on server centosvm: resources_used.cput = 30 || resources_used.foo_f = 3.03 job 4.centosvm attempt: 23 got: resources_used.cput = 10 2017-05-17 13:40:47,918 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:40:47,964 INFOCLI centosvm: /opt/pbs/bin/qstat -f 4.centosvm 2017-05-17 13:40:48,002 INFO expect on server centosvm: resources_used.cput = 30 || resources_used.foo_f = 3.03 job 4.centosvm attempt: 24 got: resources_used.cput = 10 2017-05-17 13:40:49,003 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:40:49,032 INFOCLI centosvm: /opt/pbs/bin/qstat -f 4.centosvm 2017-05-17 13:40:49,053 INFO expect on server centosvm: resources_used.cput = 30 || resources_used.foo_f = 3.03 job 4.centosvm attempt: 25 got: resources_used.cput = 10 2017-05-17 13:40:50,055 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:40:50,092 INFOCLI centosvm: /opt/pbs/bin/qstat -f 4.centosvm 2017-05-17 13:40:50,114 INFO expect on server centosvm: resources_used.cput = 30 || resources_used.foo_f = 3.03 job 4.centosvm attempt: 26 got: resources_used.cput = 10 2017-05-17 13:40:51,117 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:40:51,134 INFOCLI centosvm: /opt/pbs/bin/qstat -f 4.centosvm 2017-05-17 13:40:51,150 INFO expect on server centosvm: resources_used.cput = 30 || resources_used.foo_f = 3.03 job 4.centosvm attempt: 27 got: resources_used.cput = 10 2017-05-17 13:40:52,154 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:40:52,183 INFOCLI centosvm: /opt/pbs/bin/qstat -f 4.centosvm 2017-05-17 13:40:52,204 INFO expect on server centosvm: resources_used.cput = 30 || resources_used.foo_f = 3.03 job 4.centosvm attempt: 28 got: resources_used.cput = 10 2017-05-17 13:40:53,205 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:40:53,231 INFOCLI centosvm: /opt/pbs/bin/qstat -f 4.centosvm 2017-05-17 13:40:53,256 INFO expect on server centosvm: resources_used.cput = 30 || resources_used.foo_f = 3.03 job 4.centosvm attempt: 29 got: resources_used.cput = 10 2017-05-17 13:40:54,257 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:40:54,289 INFOCLI centosvm: /opt/pbs/bin/qstat -f 4.centosvm 2017-05-17 13:40:54,309 INFO expect on server centosvm: resources_used.cput = 30 || resources_used.foo_f = 3.03 job 4.centosvm attempt: 30 got: resources_used.cput = 10 2017-05-17 13:40:55,312 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:40:55,339 INFOCLI centosvm: /opt/pbs/bin/qstat -f 4.centosvm 2017-05-17 13:40:55,361 INFO expect on server centosvm: resources_used.cput = 30 || resources_used.foo_f = 3.03 job 4.centosvm attempt: 31 got: resources_used.cput = 10 2017-05-17 13:40:56,364 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:40:56,428 INFOCLI centosvm: /opt/pbs/bin/qstat -f 4.centosvm 2017-05-17 13:40:56,471 INFO expect on server centosvm: resources_used.cput = 30 || resources_used.foo_f = 3.03 job 4.centosvm attempt: 32 got: resources_used.cput = 10 2017-05-17 13:40:57,474 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:40:57,488 INFOCLI centosvm: /opt/pbs/bin/qstat -f 4.centosvm 2017-05-17 13:40:57,500 INFO expect on server centosvm: resources_used.cput = 30 || resources_used.foo_f = 3.03 job 4.centosvm attempt: 33 got: resources_used.cput = 10 2017-05-17 13:40:58,504 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:40:58,535 INFOCLI centosvm: /opt/pbs/bin/qstat -f 4.centosvm 2017-05-17 13:40:58,563 INFO expect on server centosvm: resources_used.cput = 30 || resources_used.foo_f = 3.03 job 4.centosvm attempt: 34 got: resources_used.cput = 10 2017-05-17 13:40:59,567 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:40:59,614 INFOCLI centosvm: /opt/pbs/bin/qstat -f 4.centosvm 2017-05-17 13:40:59,653 INFO expect on server centosvm: resources_used.cput = 30 || resources_used.foo_f = 3.03 job 4.centosvm attempt: 35 got: resources_used.cput = 10 2017-05-17 13:41:00,655 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:41:00,708 INFOCLI centosvm: /opt/pbs/bin/qstat -f 4.centosvm 2017-05-17 13:41:00,744 INFO expect on server centosvm: resources_used.cput = 30 || resources_used.foo_f = 3.03 job 4.centosvm attempt: 36 got: resources_used.cput = 10 2017-05-17 13:41:01,749 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:41:01,796 INFOCLI centosvm: /opt/pbs/bin/qstat -f 4.centosvm 2017-05-17 13:41:01,836 INFO expect on server centosvm: resources_used.cput = 30 || resources_used.foo_f = 3.03 job 4.centosvm attempt: 37 ... OK 2017-05-17 13:41:01,837 INFO manager on centosvm: set server {'scheduling': 'False'} 2017-05-17 13:41:01,838 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=False 2017-05-17 13:41:01,882 INFO rerun on centosvm as root: 4.centosvm 2017-05-17 13:41:01,883 INFOCLI centosvm: sudo -H -u root /opt/pbs/bin/qrerun 4.centosvm 2017-05-17 13:41:04,136 INFOCLI centosvm: /opt/pbs/bin/qstat -f 4.centosvm 2017-05-17 13:41:04,180 INFO expect on server centosvm: job_state = Q job 4.centosvm ... OK 2017-05-17 13:41:04,182 INFOCLI centosvm: /opt/pbs/bin/qstat -f 4.centosvm 2017-05-17 13:41:04,223 INFO expect on server centosvm: Resource_List.foo_f unset job 4.centosvm ... OK 2017-05-17 13:41:04,224 INFO manager on centosvm: set server {'scheduling': 'True'} 2017-05-17 13:41:04,224 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:41:04,271 INFOCLI centosvm: /opt/pbs/bin/qstat -f 4.centosvm 2017-05-17 13:41:04,296 INFO expect on server centosvm: job_state = R || substate = 42 job 4.centosvm got: job_state = Q 2017-05-17 13:41:04,798 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:41:04,855 INFOCLI centosvm: /opt/pbs/bin/qstat -f 4.centosvm 2017-05-17 13:41:04,886 INFO expect on server centosvm: job_state = R || substate = 42 job 4.centosvm attempt: 2 got: substate = 41 2017-05-17 13:41:05,387 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:41:05,411 INFOCLI centosvm: /opt/pbs/bin/qstat -f 4.centosvm 2017-05-17 13:41:05,438 INFO expect on server centosvm: job_state = R || substate = 42 job 4.centosvm attempt: 3 ... OK 2017-05-17 13:41:05,439 INFO server centosvm: expect offset set to 10 2017-05-17 13:41:15,451 INFOCLI centosvm: /opt/pbs/bin/qstat -f 4.centosvm 2017-05-17 13:41:15,490 INFO expect on server centosvm: resources_used.cput = 30 || resources_used.foo_f = 3.03 job 4.centosvm got: resources_used.cput = 0 2017-05-17 13:41:16,492 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:41:16,547 INFOCLI centosvm: /opt/pbs/bin/qstat -f 4.centosvm 2017-05-17 13:41:16,592 INFO expect on server centosvm: resources_used.cput = 30 || resources_used.foo_f = 3.03 job 4.centosvm attempt: 2 got: resources_used.cput = 10 2017-05-17 13:41:17,595 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:41:17,635 INFOCLI centosvm: /opt/pbs/bin/qstat -f 4.centosvm 2017-05-17 13:41:17,673 INFO expect on server centosvm: resources_used.cput = 30 || resources_used.foo_f = 3.03 job 4.centosvm attempt: 3 got: resources_used.cput = 10 2017-05-17 13:41:18,676 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:41:18,724 INFOCLI centosvm: /opt/pbs/bin/qstat -f 4.centosvm 2017-05-17 13:41:18,750 INFO expect on server centosvm: resources_used.cput = 30 || resources_used.foo_f = 3.03 job 4.centosvm attempt: 4 got: resources_used.cput = 10 2017-05-17 13:41:19,753 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:41:19,780 INFOCLI centosvm: /opt/pbs/bin/qstat -f 4.centosvm 2017-05-17 13:41:19,805 INFO expect on server centosvm: resources_used.cput = 30 || resources_used.foo_f = 3.03 job 4.centosvm attempt: 5 got: resources_used.cput = 10 2017-05-17 13:41:20,810 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:41:20,855 INFOCLI centosvm: /opt/pbs/bin/qstat -f 4.centosvm 2017-05-17 13:41:20,893 INFO expect on server centosvm: resources_used.cput = 30 || resources_used.foo_f = 3.03 job 4.centosvm attempt: 6 got: resources_used.cput = 10 2017-05-17 13:41:21,898 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:41:21,943 INFOCLI centosvm: /opt/pbs/bin/qstat -f 4.centosvm 2017-05-17 13:41:21,969 INFO expect on server centosvm: resources_used.cput = 30 || resources_used.foo_f = 3.03 job 4.centosvm attempt: 7 got: resources_used.cput = 10 2017-05-17 13:41:22,972 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:41:23,019 INFOCLI centosvm: /opt/pbs/bin/qstat -f 4.centosvm 2017-05-17 13:41:23,057 INFO expect on server centosvm: resources_used.cput = 30 || resources_used.foo_f = 3.03 job 4.centosvm attempt: 8 got: resources_used.cput = 10 2017-05-17 13:41:24,062 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:41:24,108 INFOCLI centosvm: /opt/pbs/bin/qstat -f 4.centosvm 2017-05-17 13:41:24,145 INFO expect on server centosvm: resources_used.cput = 30 || resources_used.foo_f = 3.03 job 4.centosvm attempt: 9 got: resources_used.cput = 10 2017-05-17 13:41:25,150 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:41:25,177 INFOCLI centosvm: /opt/pbs/bin/qstat -f 4.centosvm 2017-05-17 13:41:25,200 INFO expect on server centosvm: resources_used.cput = 30 || resources_used.foo_f = 3.03 job 4.centosvm attempt: 10 got: resources_used.cput = 10 2017-05-17 13:41:26,204 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:41:26,252 INFOCLI centosvm: /opt/pbs/bin/qstat -f 4.centosvm 2017-05-17 13:41:26,282 INFO expect on server centosvm: resources_used.cput = 30 || resources_used.foo_f = 3.03 job 4.centosvm attempt: 11 got: resources_used.cput = 10 2017-05-17 13:41:27,285 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:41:27,326 INFOCLI centosvm: /opt/pbs/bin/qstat -f 4.centosvm 2017-05-17 13:41:27,350 INFO expect on server centosvm: resources_used.cput = 30 || resources_used.foo_f = 3.03 job 4.centosvm attempt: 12 got: resources_used.cput = 10 2017-05-17 13:41:28,355 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:41:28,400 INFOCLI centosvm: /opt/pbs/bin/qstat -f 4.centosvm 2017-05-17 13:41:28,432 INFO expect on server centosvm: resources_used.cput = 30 || resources_used.foo_f = 3.03 job 4.centosvm attempt: 13 got: resources_used.cput = 10 2017-05-17 13:41:29,438 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:41:29,485 INFOCLI centosvm: /opt/pbs/bin/qstat -f 4.centosvm 2017-05-17 13:41:29,515 INFO expect on server centosvm: resources_used.cput = 30 || resources_used.foo_f = 3.03 job 4.centosvm attempt: 14 got: resources_used.cput = 10 2017-05-17 13:41:30,518 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:41:30,564 INFOCLI centosvm: /opt/pbs/bin/qstat -f 4.centosvm 2017-05-17 13:41:30,597 INFO expect on server centosvm: resources_used.cput = 30 || resources_used.foo_f = 3.03 job 4.centosvm attempt: 15 got: resources_used.cput = 10 2017-05-17 13:41:31,599 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:41:31,627 INFOCLI centosvm: /opt/pbs/bin/qstat -f 4.centosvm 2017-05-17 13:41:31,653 INFO expect on server centosvm: resources_used.cput = 30 || resources_used.foo_f = 3.03 job 4.centosvm attempt: 16 got: resources_used.cput = 10 2017-05-17 13:41:32,656 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:41:32,706 INFOCLI centosvm: /opt/pbs/bin/qstat -f 4.centosvm 2017-05-17 13:41:32,732 INFO expect on server centosvm: resources_used.cput = 30 || resources_used.foo_f = 3.03 job 4.centosvm attempt: 17 ... OK 2017-05-17 13:41:32,733 INFO =========================================== 2017-05-17 13:41:32,733 INFO Entered TestPbsAccumulateRescUsed tearDown 2017-05-17 13:41:32,734 INFO =========================================== 2017-05-17 13:41:32,734 INFO ============================================ 2017-05-17 13:41:32,734 INFO Completed TestPbsAccumulateRescUsed tearDown 2017-05-17 13:41:32,734 INFO ============================================ 2017-05-17 13:41:32,735 INFO ok 2017-05-17 13:41:32,735 INFO test name: test_mom_down (tests.functional.pbs_accumulate_resc_used.TestPbsAccumulateRescUsed)... 2017-05-17 13:41:32,735 INFO test start time: Wed May 17 13:41:32 2017 2017-05-17 13:41:32,736 INFO test docstring: Test that resource_accumulation is not impacted due to mom restart 2017-05-17 13:41:32,736 INFO ======================================== 2017-05-17 13:41:32,736 INFO Entered TestPbsAccumulateRescUsed setUp 2017-05-17 13:41:32,736 INFO ======================================== 2017-05-17 13:41:32,736 INFOCLI centosvm: /opt/pbs/bin/qstat -Bf centosvm 2017-05-17 13:41:32,758 INFO status on centosvm: server 2017-05-17 13:41:32,758 INFOCLI centosvm: /opt/pbs/bin/qstat -Bf centosvm 2017-05-17 13:41:32,782 INFO manager on centosvm: set server {'managers': (3, 'dilipk@*')} 2017-05-17 13:41:32,782 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c set server managers-=dilipk@* 2017-05-17 13:41:32,825 INFO manager on centosvm: set server {'managers': (2, 'dilipk@*')} 2017-05-17 13:41:32,825 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c set server managers+=dilipk@* 2017-05-17 13:41:32,872 INFO server centosvm: reverting configuration to defaults 2017-05-17 13:41:32,873 INFOCLI centosvm: /opt/pbs/bin/qstat -Bf centosvm 2017-05-17 13:41:32,902 INFO select on centosvm: __ALL__ 2017-05-17 13:41:32,902 INFOCLI centosvm: /opt/pbs/bin/qselect -x 2017-05-17 13:41:32,913 INFO delete job on centosvm: 4.centosvm 2017-05-17 13:41:32,914 INFOCLI centosvm: /opt/pbs/bin/qdel -W force -x 4.centosvm 2017-05-17 13:41:32,931 INFOCLI centosvm: /opt/pbs/bin/qstat -f 4.centosvm 2017-05-17 13:41:32,953 INFOCLI centosvm: /opt/pbs/bin/qstat -f @centosvm 2017-05-17 13:41:32,964 INFO expect on server centosvm: job_state set 0 job ... OK 2017-05-17 13:41:32,964 INFOCLI centosvm: /opt/pbs/bin/pbs_rstat -f 2017-05-17 13:41:32,976 INFO manager on centosvm: unset server ['job_history_enable'] 2017-05-17 13:41:32,977 INFOCLI centosvm: /opt/pbs/bin/qmgr -c unset server job_history_enable 2017-05-17 13:41:32,997 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-05-17 13:41:33,021 INFO manager on centosvm: delete hook start,period 2017-05-17 13:41:33,021 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c delete hook start,period 2017-05-17 13:41:33,050 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook start 2017-05-17 13:41:33,071 INFO expect on server centosvm: unset hook start ... OK 2017-05-17 13:41:33,071 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook period 2017-05-17 13:41:33,101 INFO expect on server centosvm: unset hook period ... OK 2017-05-17 13:41:33,101 INFOCLI centosvm: /opt/pbs/bin/qstat -Qf @centosvm 2017-05-17 13:41:33,115 INFO manager on centosvm: delete queue workq 2017-05-17 13:41:33,116 INFOCLI centosvm: /opt/pbs/bin/qmgr -c delete queue workq 2017-05-17 13:41:33,130 INFOCLI centosvm: /opt/pbs/bin/qstat -Qf workq@centosvm 2017-05-17 13:41:33,141 INFO expect on server centosvm: unset queue workq ... OK 2017-05-17 13:41:33,141 INFO manager on centosvm: create queue workq {'started': 'True', 'queue_type': 'Execution', 'enabled': 'True'} 2017-05-17 13:41:33,141 INFOCLI centosvm: /opt/pbs/bin/qmgr -c create queue workq started=True,queue_type=Execution,enabled=True 2017-05-17 13:41:33,159 INFOCLI centosvm: /opt/pbs/bin/qstat -Qf workq@centosvm 2017-05-17 13:41:33,171 INFO expect on server centosvm: started set True || queue_type set Execution || enabled set True queue workq ... OK 2017-05-17 13:41:33,171 INFO manager on centosvm: set server {'default_queue': 'workq'} 2017-05-17 13:41:33,171 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server default_queue=workq 2017-05-17 13:41:33,186 INFO status on centosvm: resource 2017-05-17 13:41:33,186 INFO manager on centosvm: list resource 2017-05-17 13:41:33,186 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource 2017-05-17 13:41:33,204 INFO manager on centosvm: delete resource foo_f,foo_i,foo_str2,foo_str3,foo_str4,foo_str,stra 2017-05-17 13:41:33,204 INFOCLI centosvm: /opt/pbs/bin/qmgr -c delete resource foo_f,foo_i,foo_str2,foo_str3,foo_str4,foo_str,stra 2017-05-17 13:41:33,424 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_f 2017-05-17 13:41:33,442 INFO expect on server centosvm: unset resource foo_f ... OK 2017-05-17 13:41:33,442 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_i 2017-05-17 13:41:33,453 INFO expect on server centosvm: unset resource foo_i ... OK 2017-05-17 13:41:33,453 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_str2 2017-05-17 13:41:33,465 INFO expect on server centosvm: unset resource foo_str2 ... OK 2017-05-17 13:41:33,466 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_str3 2017-05-17 13:41:33,477 INFO expect on server centosvm: unset resource foo_str3 ... OK 2017-05-17 13:41:33,477 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_str4 2017-05-17 13:41:33,489 INFO expect on server centosvm: unset resource foo_str4 ... OK 2017-05-17 13:41:33,489 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_str 2017-05-17 13:41:33,499 INFO expect on server centosvm: unset resource foo_str ... OK 2017-05-17 13:41:33,500 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource stra 2017-05-17 13:41:33,510 INFO expect on server centosvm: unset resource stra ... OK 2017-05-17 13:41:33,511 INFOCLI status on centosvm: server license_count 2017-05-17 13:41:33,511 INFOCLI centosvm: /opt/pbs/bin/qstat -Bf centosvm 2017-05-17 13:41:33,521 INFO server: centosvm licensed 2017-05-17 13:41:33,547 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/server_priv/comm.lock 2017-05-17 13:41:33,577 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-05-17 13:41:33,588 INFO scheduler centosvm: reverting configuration to defaults 2017-05-17 13:41:33,588 INFO manager on centosvm: list sched 2017-05-17 13:41:33,589 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list sched 2017-05-17 13:41:33,601 INFO manager on centosvm: unset sched ['sched_cycle_length'] 2017-05-17 13:41:33,602 INFOCLI centosvm: /opt/pbs/bin/qmgr -c unset sched sched_cycle_length 2017-05-17 13:41:33,614 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/dedicated_time 2017-05-17 13:41:33,625 INFOCLI2 centosvm: cmp /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-05-17 13:41:33,635 ERROR err: ['cmp: /var/spool/pbs/sched_priv/resource_group: Permission denied'] 2017-05-17 13:41:33,635 INFOCLI2 centosvm: sudo -H /usr/bin/cp -p /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-05-17 13:41:33,646 INFOCLI2 centosvm: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-05-17 13:41:33,656 INFO scheduler centosvm: reverting holidays file to default 2017-05-17 13:41:33,657 INFOCLI2 centosvm: cmp /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-05-17 13:41:33,659 ERROR err: ['cmp: /var/spool/pbs/sched_priv/holidays: Permission denied'] 2017-05-17 13:41:33,659 INFOCLI2 centosvm: sudo -H /usr/bin/cp -p /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-05-17 13:41:33,672 INFOCLI2 centosvm: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/holidays 2017-05-17 13:41:33,684 INFO scheduler centosvm: Parsing holidays file 2017-05-17 13:41:33,684 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2017-05-17 13:41:33,695 INFOCLI2 centosvm: cmp /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-05-17 13:41:33,699 ERROR err: ['cmp: /var/spool/pbs/sched_priv/sched_config: Permission denied'] 2017-05-17 13:41:33,699 INFOCLI2 centosvm: sudo -H /usr/bin/cp -p /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-05-17 13:41:33,712 INFOCLI2 centosvm: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-05-17 13:41:33,726 INFO scheduler centosvm: sent signal -HUP 2017-05-17 13:41:33,726 INFOCLI2 centosvm: sudo -H kill -HUP 84281 2017-05-17 13:41:33,736 INFOCLI2 centosvm: sudo -H /usr/bin/rm /var/spool/pbs/sched_priv/usage 2017-05-17 13:41:33,751 ERROR err: ['/usr/bin/rm: cannot remove \xe2\x80\x98/var/spool/pbs/sched_priv/usage\xe2\x80\x99: No such file or directory'] 2017-05-17 13:41:33,777 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-05-17 13:41:33,788 INFOCLI2 centosvm: sudo -H cmp /opt/pbs/sbin/pbs_mom /opt/pbs/sbin/pbs_mom.cpuset 2017-05-17 13:41:33,817 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:41:33,828 INFO mom centosvm: reverting configuration to defaults 2017-05-17 13:41:33,828 INFOCLI2 centosvm: sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2017-05-17 13:41:33,843 INFOCLI2 centosvm: sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2017-05-17 13:41:33,854 INFOCLI2 centosvm: sudo -H ls /var/spool/pbs/mom_priv/config.d 2017-05-17 13:41:33,865 INFO mom centosvm: sent signal -HUP 2017-05-17 13:41:33,865 INFOCLI2 centosvm: sudo -H kill -HUP 84278 2017-05-17 13:41:33,892 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:41:33,905 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v -a 2017-05-17 13:41:33,920 INFO status on centosvm: node centosvm 2017-05-17 13:41:33,920 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:41:33,932 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:41:33,944 INFO expect on server centosvm: state = free node centosvm ... OK 2017-05-17 13:41:33,944 INFOCLI2 centosvm2: ssh centosvm2 sudo -H cmp /opt/pbs/sbin/pbs_mom /opt/pbs/sbin/pbs_mom.cpuset 2017-05-17 13:41:34,895 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:41:35,439 INFO mom centosvm2: reverting configuration to defaults 2017-05-17 13:41:35,440 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2017-05-17 13:41:35,942 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2017-05-17 13:41:36,367 INFOCLI2 centosvm2: ssh centosvm2 sudo -H ls /var/spool/pbs/mom_priv/config.d 2017-05-17 13:41:36,802 INFO mom centosvm2: sent signal -HUP 2017-05-17 13:41:36,802 INFOCLI2 centosvm2: ssh centosvm2 sudo -H kill -HUP 9078 2017-05-17 13:41:37,920 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:41:38,391 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v -a 2017-05-17 13:41:38,424 INFO status on centosvm: node centosvm2 2017-05-17 13:41:38,425 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm2 2017-05-17 13:41:38,453 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm2 2017-05-17 13:41:38,473 INFO expect on server centosvm: state = free node centosvm2 ... OK 2017-05-17 13:41:38,474 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H cmp /opt/pbs/sbin/pbs_mom /opt/pbs/sbin/pbs_mom.cpuset 2017-05-17 13:41:39,480 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:41:39,915 INFO mom centosvm_nfs: reverting configuration to defaults 2017-05-17 13:41:39,916 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2017-05-17 13:41:40,419 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2017-05-17 13:41:40,852 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H ls /var/spool/pbs/mom_priv/config.d 2017-05-17 13:41:41,233 INFO mom centosvm_nfs: sent signal -HUP 2017-05-17 13:41:41,234 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H kill -HUP 76746 2017-05-17 13:41:42,503 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:41:42,990 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v -a 2017-05-17 13:41:43,038 INFO status on centosvm: node centosvm_nfs 2017-05-17 13:41:43,039 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm_nfs 2017-05-17 13:41:43,065 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm_nfs 2017-05-17 13:41:43,082 INFO expect on server centosvm: state = free node centosvm_nfs ... OK 2017-05-17 13:41:43,083 INFO ========================================= 2017-05-17 13:41:43,083 INFO Completed TestPbsAccumulateRescUsed setUp 2017-05-17 13:41:43,083 INFO ========================================= 2017-05-17 13:41:43,083 INFO len moms = 3 2017-05-17 13:41:43,083 INFOCLI2 centosvm: sudo -H /usr/bin/rm -rf /var/spool/pbs/mom_priv/config.d 2017-05-17 13:41:43,102 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/rm -rf /var/spool/pbs/mom_priv/config.d 2017-05-17 13:41:43,502 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H /usr/bin/rm -rf /var/spool/pbs/mom_priv/config.d 2017-05-17 13:41:43,906 INFO manager on centosvm: delete node 2017-05-17 13:41:43,907 INFOCLI centosvm: /opt/pbs/bin/qmgr -c delete node @default 2017-05-17 13:41:43,941 INFO manager on centosvm: create node centosvm 2017-05-17 13:41:43,941 INFOCLI centosvm: /opt/pbs/bin/qmgr -c create node centosvm 2017-05-17 13:41:43,962 INFO manager on centosvm: create node centosvm2 2017-05-17 13:41:43,963 INFOCLI centosvm: /opt/pbs/bin/qmgr -c create node centosvm2 2017-05-17 13:41:43,980 INFO manager on centosvm: create node centosvm_nfs 2017-05-17 13:41:43,980 INFOCLI centosvm: /opt/pbs/bin/qmgr -c create node centosvm_nfs 2017-05-17 13:41:44,001 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:41:44,012 INFO expect on server centosvm: state = free node centosvm got: state = state-unknown,down 2017-05-17 13:41:44,513 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:41:44,559 INFO expect on server centosvm: state = free node centosvm attempt: 2 got: state = state-unknown,down 2017-05-17 13:41:45,061 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:41:45,074 INFO expect on server centosvm: state = free node centosvm attempt: 3 ... OK 2017-05-17 13:41:45,075 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm2 2017-05-17 13:41:45,086 INFO expect on server centosvm: state = free node centosvm2 got: state = initializing 2017-05-17 13:41:45,588 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm2 2017-05-17 13:41:45,629 INFO expect on server centosvm: state = free node centosvm2 attempt: 2 ... OK 2017-05-17 13:41:45,631 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm_nfs 2017-05-17 13:41:45,658 INFO expect on server centosvm: state = free node centosvm_nfs ... OK 2017-05-17 13:41:45,659 INFO status on centosvm: hook 2017-05-17 13:41:45,659 INFO manager on centosvm: list hook 2017-05-17 13:41:45,659 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-05-17 13:41:45,696 INFO status on centosvm: hook 2017-05-17 13:41:45,696 INFO manager on centosvm: list hook 2017-05-17 13:41:45,697 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-05-17 13:41:45,728 INFO manager on centosvm: create hook start 2017-05-17 13:41:45,728 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c create hook start 2017-05-17 13:41:45,757 INFO manager on centosvm: set hook start {'enabled': 'True', 'event': 'exechost_startup'} 2017-05-17 13:41:45,758 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c set hook start enabled=True,event=exechost_startup 2017-05-17 13:41:45,787 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook start 2017-05-17 13:41:45,821 INFO expect on server centosvm: enabled set True || event set exechost_startup hook start ... OK 2017-05-17 13:41:45,822 INFO manager on centosvm: import hook start {'content-type': 'application/x-python', 'input-file': '/tmp/PtlPbsRwFJPy', 'content-encoding': 'default'} 2017-05-17 13:41:45,822 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c import hook start application/x-python default /tmp/PtlPbsRwFJPy 2017-05-17 13:41:45,856 INFO server centosvm: imported hook body --- import pbs e=pbs.event() localnode=pbs.get_local_nodename() e.vnode_list[localnode].resources_available['foo_i'] = 7 e.vnode_list[localnode].resources_available['foo_f'] = 5.0 e.vnode_list[localnode].resources_available['foo_str'] = "seventyseven" --- 2017-05-17 13:41:45,856 INFO mom centosvm: sent signal -HUP 2017-05-17 13:41:45,856 INFOCLI2 centosvm: sudo -H kill -HUP 84278 2017-05-17 13:41:45,872 INFO mom centosvm2: sent signal -HUP 2017-05-17 13:41:45,872 INFOCLI2 centosvm2: ssh centosvm2 sudo -H kill -HUP 9078 2017-05-17 13:41:46,365 INFO mom centosvm_nfs: sent signal -HUP 2017-05-17 13:41:46,365 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H kill -HUP 76746 2017-05-17 13:41:46,876 INFO manager on centosvm: set server {'job_history_enable': 'True'} 2017-05-17 13:41:46,877 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server job_history_enable=True 2017-05-17 13:41:46,897 INFO manager on centosvm as root: create resource foo_str2 {'flag': 'h', 'type': 'string'} 2017-05-17 13:41:46,898 INFOCLI centosvm: sudo -H -u root /opt/pbs/bin/qmgr -c create resource foo_str2 flag=h,type=string 2017-05-17 13:41:46,945 INFO mom centosvm log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:41:47,346 INFO mom centosvm2 log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:41:47,967 INFO mom centosvm_nfs log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:41:47,969 INFO manager on centosvm as root: create resource foo_str3 {'flag': 'h', 'type': 'string'} 2017-05-17 13:41:47,969 INFOCLI centosvm: sudo -H -u root /opt/pbs/bin/qmgr -c create resource foo_str3 flag=h,type=string 2017-05-17 13:41:48,067 INFO mom centosvm log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:41:48,561 INFO mom centosvm2 log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:41:48,970 INFO mom centosvm_nfs log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:41:48,970 INFO manager on centosvm as root: create resource foo_str4 {'flag': 'h', 'type': 'string'} 2017-05-17 13:41:48,970 INFOCLI centosvm: sudo -H -u root /opt/pbs/bin/qmgr -c create resource foo_str4 flag=h,type=string 2017-05-17 13:41:49,011 INFO mom centosvm log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:41:49,351 INFO mom centosvm2 log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:41:49,688 INFO mom centosvm_nfs log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:41:49,688 INFO manager on centosvm as root: create resource stra {'flag': 'h', 'type': 'string_array'} 2017-05-17 13:41:49,688 INFOCLI centosvm: sudo -H -u root /opt/pbs/bin/qmgr -c create resource stra flag=h,type=string_array 2017-05-17 13:41:49,735 INFO mom centosvm log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:41:50,086 INFO mom centosvm2 log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:41:50,509 INFO mom centosvm_nfs log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:41:50,509 INFO manager on centosvm: set server {'node_fail_requeue': 10} 2017-05-17 13:41:50,509 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server node_fail_requeue=10 2017-05-17 13:41:50,523 INFO status on centosvm: hook 2017-05-17 13:41:50,523 INFO manager on centosvm: list hook 2017-05-17 13:41:50,524 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-05-17 13:41:50,546 INFO status on centosvm: hook 2017-05-17 13:41:50,547 INFO manager on centosvm: list hook 2017-05-17 13:41:50,547 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-05-17 13:41:50,568 INFO manager on centosvm: create hook period 2017-05-17 13:41:50,568 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c create hook period 2017-05-17 13:41:50,592 INFO manager on centosvm: set hook period {'freq': 10, 'enabled': 'True', 'event': 'exechost_periodic'} 2017-05-17 13:41:50,592 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c set hook period freq=10,enabled=True,event=exechost_periodic 2017-05-17 13:41:50,620 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook period 2017-05-17 13:41:50,642 INFO expect on server centosvm: freq set 10 || enabled set True || event set exechost_periodic hook period ... OK 2017-05-17 13:41:50,643 INFO manager on centosvm: import hook period {'content-type': 'application/x-python', 'input-file': '/tmp/PtlPbss8kP80', 'content-encoding': 'default'} 2017-05-17 13:41:50,643 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c import hook period application/x-python default /tmp/PtlPbss8kP80 2017-05-17 13:41:50,668 INFO server centosvm: imported hook body --- import pbs e = pbs.event() pbs.logmsg(pbs.LOG_DEBUG, "executed periodic hook") for jj in e.job_list.keys(): e.job_list[jj].resources_used["foo_i"] = 1 e.job_list[jj].resources_used["foo_str"] = '{"happy":"true"}' e.job_list[jj].resources_used["stra"] = '"one","two"' --- 2017-05-17 13:41:50,669 INFO job: executable set to /bin/sleep with arguments: 100 2017-05-17 13:41:50,670 INFOCLI centosvm: sudo -H -u pbsuser /opt/pbs/bin/qsub -l place=scatter -l select=3:ncpus=1 -- /bin/sleep 100 2017-05-17 13:41:50,700 INFO submit to centosvm as pbsuser: job 5.centosvm OrderedDict([('Resource_List.place', 'scatter'), ('Resource_List.select', '3:ncpus=1')]) 2017-05-17 13:41:50,700 INFOCLI centosvm: sudo -H -u pbsuser /opt/pbs/bin/qsub -l place=scatter -l select=3:ncpus=1 -- /bin/sleep 100 2017-05-17 13:41:50,726 INFO submit to centosvm as pbsuser: job 6.centosvm OrderedDict([('Resource_List.place', 'scatter'), ('Resource_List.select', '3:ncpus=1')]) 2017-05-17 13:41:50,727 INFO server centosvm: expect offset set to 10 2017-05-17 13:42:00,738 INFOCLI centosvm: /opt/pbs/bin/qstat -f 5.centosvm 2017-05-17 13:42:01,797 INFO expect on server centosvm: no data for resources_used.foo_i = 3 job 5.centosvm 2017-05-17 13:42:01,798 INFOCLI centosvm: /opt/pbs/bin/qstat -f 5.centosvm 2017-05-17 13:42:02,818 INFO expect on server centosvm: no data for resources_used.foo_i = 3 job 5.centosvm attempt: 2 2017-05-17 13:42:02,818 INFOCLI centosvm: /opt/pbs/bin/qstat -f 5.centosvm 2017-05-17 13:42:03,835 INFO expect on server centosvm: no data for resources_used.foo_i = 3 job 5.centosvm attempt: 3 2017-05-17 13:42:03,836 INFOCLI centosvm: /opt/pbs/bin/qstat -f 5.centosvm 2017-05-17 13:42:03,847 INFO expect on server centosvm: resources_used.foo_i = 3 job 5.centosvm attempt: 4 got: resources_used.foo_i = 1 2017-05-17 13:42:04,850 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:04,904 INFOCLI centosvm: /opt/pbs/bin/qstat -f 5.centosvm 2017-05-17 13:42:04,933 INFO expect on server centosvm: resources_used.foo_i = 3 job 5.centosvm attempt: 5 got: resources_used.foo_i = 1 2017-05-17 13:42:05,937 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:05,969 INFOCLI centosvm: /opt/pbs/bin/qstat -f 5.centosvm 2017-05-17 13:42:06,006 INFO expect on server centosvm: resources_used.foo_i = 3 job 5.centosvm attempt: 6 got: resources_used.foo_i = 1 2017-05-17 13:42:07,009 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:07,023 INFOCLI centosvm: /opt/pbs/bin/qstat -f 5.centosvm 2017-05-17 13:42:07,035 INFO expect on server centosvm: resources_used.foo_i = 3 job 5.centosvm attempt: 7 got: resources_used.foo_i = 1 2017-05-17 13:42:08,036 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:08,050 INFOCLI centosvm: /opt/pbs/bin/qstat -f 5.centosvm 2017-05-17 13:42:08,062 INFO expect on server centosvm: resources_used.foo_i = 3 job 5.centosvm attempt: 8 got: resources_used.foo_i = 1 2017-05-17 13:42:09,067 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:09,115 INFOCLI centosvm: /opt/pbs/bin/qstat -f 5.centosvm 2017-05-17 13:42:09,163 INFO expect on server centosvm: resources_used.foo_i = 3 job 5.centosvm attempt: 9 got: resources_used.foo_i = 1 2017-05-17 13:42:10,165 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:10,209 INFOCLI centosvm: /opt/pbs/bin/qstat -f 5.centosvm 2017-05-17 13:42:10,248 INFO expect on server centosvm: resources_used.foo_i = 3 job 5.centosvm attempt: 10 got: resources_used.foo_i = 1 2017-05-17 13:42:11,250 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:11,295 INFOCLI centosvm: /opt/pbs/bin/qstat -f 5.centosvm 2017-05-17 13:42:11,343 INFO expect on server centosvm: resources_used.foo_i = 3 job 5.centosvm attempt: 11 got: resources_used.foo_i = 1 2017-05-17 13:42:12,348 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:12,379 INFOCLI centosvm: /opt/pbs/bin/qstat -f 5.centosvm 2017-05-17 13:42:12,398 INFO expect on server centosvm: resources_used.foo_i = 3 job 5.centosvm attempt: 12 got: resources_used.foo_i = 1 2017-05-17 13:42:13,403 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:13,453 INFOCLI centosvm: /opt/pbs/bin/qstat -f 5.centosvm 2017-05-17 13:42:13,480 INFO expect on server centosvm: resources_used.foo_i = 3 job 5.centosvm attempt: 13 got: resources_used.foo_i = 1 2017-05-17 13:42:14,485 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:14,525 INFOCLI centosvm: /opt/pbs/bin/qstat -f 5.centosvm 2017-05-17 13:42:14,542 INFO expect on server centosvm: resources_used.foo_i = 3 job 5.centosvm attempt: 14 got: resources_used.foo_i = 1 2017-05-17 13:42:15,545 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:15,596 INFOCLI centosvm: /opt/pbs/bin/qstat -f 5.centosvm 2017-05-17 13:42:15,630 INFO expect on server centosvm: resources_used.foo_i = 3 job 5.centosvm attempt: 15 got: resources_used.foo_i = 1 2017-05-17 13:42:16,632 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:16,659 INFOCLI centosvm: /opt/pbs/bin/qstat -f 5.centosvm 2017-05-17 13:42:16,682 INFO expect on server centosvm: resources_used.foo_i = 3 job 5.centosvm attempt: 16 got: resources_used.foo_i = 1 2017-05-17 13:42:17,686 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:17,699 INFOCLI centosvm: /opt/pbs/bin/qstat -f 5.centosvm 2017-05-17 13:42:17,711 INFO expect on server centosvm: resources_used.foo_i = 3 job 5.centosvm attempt: 17 got: resources_used.foo_i = 1 2017-05-17 13:42:18,714 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:18,731 INFOCLI centosvm: /opt/pbs/bin/qstat -f 5.centosvm 2017-05-17 13:42:18,744 INFO expect on server centosvm: resources_used.foo_i = 3 job 5.centosvm attempt: 18 got: resources_used.foo_i = 1 2017-05-17 13:42:19,746 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:19,763 INFOCLI centosvm: /opt/pbs/bin/qstat -f 5.centosvm 2017-05-17 13:42:19,775 INFO expect on server centosvm: resources_used.foo_i = 3 job 5.centosvm attempt: 19 got: resources_used.foo_i = 1 2017-05-17 13:42:20,777 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:20,797 INFOCLI centosvm: /opt/pbs/bin/qstat -f 5.centosvm 2017-05-17 13:42:20,813 INFO expect on server centosvm: resources_used.foo_i = 3 job 5.centosvm attempt: 20 got: resources_used.foo_i = 1 2017-05-17 13:42:21,816 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:21,831 INFOCLI centosvm: /opt/pbs/bin/qstat -f 5.centosvm 2017-05-17 13:42:21,843 INFO expect on server centosvm: resources_used.foo_i = 3 job 5.centosvm attempt: 21 ... OK 2017-05-17 13:42:21,844 INFOCLI centosvm: /opt/pbs/bin/qstat -f 6.centosvm 2017-05-17 13:42:21,858 INFO expect on server centosvm: resources_used.foo_i unset job 6.centosvm got: resources_used.foo_i = 3 2017-05-17 13:42:22,360 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:22,390 INFOCLI centosvm: /opt/pbs/bin/qstat -f 6.centosvm 2017-05-17 13:42:22,426 INFO expect on server centosvm: resources_used.foo_i unset job 6.centosvm attempt: 2 got: resources_used.foo_i = 3 2017-05-17 13:42:22,929 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:22,944 INFOCLI centosvm: /opt/pbs/bin/qstat -f 6.centosvm 2017-05-17 13:42:22,958 INFO expect on server centosvm: resources_used.foo_i unset job 6.centosvm attempt: 3 got: resources_used.foo_i = 3 2017-05-17 13:42:23,462 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:23,496 INFOCLI centosvm: /opt/pbs/bin/qstat -f 6.centosvm 2017-05-17 13:42:23,529 INFO expect on server centosvm: resources_used.foo_i unset job 6.centosvm attempt: 4 got: resources_used.foo_i = 3 2017-05-17 13:42:24,034 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:24,070 INFOCLI centosvm: /opt/pbs/bin/qstat -f 6.centosvm 2017-05-17 13:42:24,099 INFO expect on server centosvm: resources_used.foo_i unset job 6.centosvm attempt: 5 got: resources_used.foo_i = 3 2017-05-17 13:42:24,601 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:24,617 INFOCLI centosvm: /opt/pbs/bin/qstat -f 6.centosvm 2017-05-17 13:42:24,630 INFO expect on server centosvm: resources_used.foo_i unset job 6.centosvm attempt: 6 got: resources_used.foo_i = 3 2017-05-17 13:42:25,134 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:25,147 INFOCLI centosvm: /opt/pbs/bin/qstat -f 6.centosvm 2017-05-17 13:42:25,160 INFO expect on server centosvm: resources_used.foo_i unset job 6.centosvm attempt: 7 got: resources_used.foo_i = 3 2017-05-17 13:42:25,664 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:25,681 INFOCLI centosvm: /opt/pbs/bin/qstat -f 6.centosvm 2017-05-17 13:42:25,694 INFO expect on server centosvm: resources_used.foo_i unset job 6.centosvm attempt: 8 got: resources_used.foo_i = 3 2017-05-17 13:42:26,197 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:26,215 INFOCLI centosvm: /opt/pbs/bin/qstat -f 6.centosvm 2017-05-17 13:42:26,231 INFO expect on server centosvm: resources_used.foo_i unset job 6.centosvm attempt: 9 got: resources_used.foo_i = 3 2017-05-17 13:42:26,734 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:26,750 INFOCLI centosvm: /opt/pbs/bin/qstat -f 6.centosvm 2017-05-17 13:42:26,764 INFO expect on server centosvm: resources_used.foo_i unset job 6.centosvm attempt: 10 got: resources_used.foo_i = 3 2017-05-17 13:42:27,265 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:27,281 INFOCLI centosvm: /opt/pbs/bin/qstat -f 6.centosvm 2017-05-17 13:42:27,295 INFO expect on server centosvm: resources_used.foo_i unset job 6.centosvm attempt: 11 got: resources_used.foo_i = 3 2017-05-17 13:42:27,797 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:27,818 INFOCLI centosvm: /opt/pbs/bin/qstat -f 6.centosvm 2017-05-17 13:42:27,838 INFO expect on server centosvm: resources_used.foo_i unset job 6.centosvm attempt: 12 got: resources_used.foo_i = 3 2017-05-17 13:42:28,341 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:28,386 INFOCLI centosvm: /opt/pbs/bin/qstat -f 6.centosvm 2017-05-17 13:42:28,421 INFO expect on server centosvm: resources_used.foo_i unset job 6.centosvm attempt: 13 got: resources_used.foo_i = 3 2017-05-17 13:42:28,924 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:28,970 INFOCLI centosvm: /opt/pbs/bin/qstat -f 6.centosvm 2017-05-17 13:42:29,014 INFO expect on server centosvm: resources_used.foo_i unset job 6.centosvm attempt: 14 got: resources_used.foo_i = 3 2017-05-17 13:42:29,516 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:29,558 INFOCLI centosvm: /opt/pbs/bin/qstat -f 6.centosvm 2017-05-17 13:42:29,585 INFO expect on server centosvm: resources_used.foo_i unset job 6.centosvm attempt: 15 got: resources_used.foo_i = 3 2017-05-17 13:42:30,087 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:30,131 INFOCLI centosvm: /opt/pbs/bin/qstat -f 6.centosvm 2017-05-17 13:42:30,169 INFO expect on server centosvm: resources_used.foo_i unset job 6.centosvm attempt: 16 got: resources_used.foo_i = 3 2017-05-17 13:42:30,672 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:30,714 INFOCLI centosvm: /opt/pbs/bin/qstat -f 6.centosvm 2017-05-17 13:42:30,748 INFO expect on server centosvm: resources_used.foo_i unset job 6.centosvm attempt: 17 got: resources_used.foo_i = 3 2017-05-17 13:42:31,253 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:31,310 INFOCLI centosvm: /opt/pbs/bin/qstat -f 6.centosvm 2017-05-17 13:42:31,344 INFO expect on server centosvm: resources_used.foo_i unset job 6.centosvm attempt: 18 got: resources_used.foo_i = 3 2017-05-17 13:42:31,847 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:31,888 INFOCLI centosvm: /opt/pbs/bin/qstat -f 6.centosvm 2017-05-17 13:42:31,910 INFO expect on server centosvm: resources_used.foo_i unset job 6.centosvm attempt: 19 got: resources_used.foo_i = 3 2017-05-17 13:42:32,413 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:32,462 INFOCLI centosvm: /opt/pbs/bin/qstat -f 6.centosvm 2017-05-17 13:42:32,489 INFO expect on server centosvm: resources_used.foo_i unset job 6.centosvm attempt: 20 got: resources_used.foo_i = 3 2017-05-17 13:42:32,991 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:33,020 INFOCLI centosvm: /opt/pbs/bin/qstat -f 6.centosvm 2017-05-17 13:42:33,055 INFO expect on server centosvm: resources_used.foo_i unset job 6.centosvm attempt: 21 got: resources_used.foo_i = 3 2017-05-17 13:42:33,556 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:33,607 INFOCLI centosvm: /opt/pbs/bin/qstat -f 6.centosvm 2017-05-17 13:42:33,642 INFO expect on server centosvm: resources_used.foo_i unset job 6.centosvm attempt: 22 got: resources_used.foo_i = 3 2017-05-17 13:42:34,146 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:34,161 INFOCLI centosvm: /opt/pbs/bin/qstat -f 6.centosvm 2017-05-17 13:42:34,173 INFO expect on server centosvm: resources_used.foo_i unset job 6.centosvm attempt: 23 got: resources_used.foo_i = 3 2017-05-17 13:42:34,675 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:34,691 INFOCLI centosvm: /opt/pbs/bin/qstat -f 6.centosvm 2017-05-17 13:42:34,706 INFO expect on server centosvm: resources_used.foo_i unset job 6.centosvm attempt: 24 got: resources_used.foo_i = 3 2017-05-17 13:42:35,209 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:35,223 INFOCLI centosvm: /opt/pbs/bin/qstat -f 6.centosvm 2017-05-17 13:42:35,235 INFO expect on server centosvm: resources_used.foo_i unset job 6.centosvm attempt: 25 got: resources_used.foo_i = 3 2017-05-17 13:42:35,736 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:35,751 INFOCLI centosvm: /opt/pbs/bin/qstat -f 6.centosvm 2017-05-17 13:42:35,763 INFO expect on server centosvm: resources_used.foo_i unset job 6.centosvm attempt: 26 got: resources_used.foo_i = 3 2017-05-17 13:42:36,265 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:36,306 INFOCLI centosvm: /opt/pbs/bin/qstat -f 6.centosvm 2017-05-17 13:42:36,327 INFO expect on server centosvm: resources_used.foo_i unset job 6.centosvm attempt: 27 got: resources_used.foo_i = 3 2017-05-17 13:42:36,829 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:36,844 INFOCLI centosvm: /opt/pbs/bin/qstat -f 6.centosvm 2017-05-17 13:42:36,860 INFO expect on server centosvm: resources_used.foo_i unset job 6.centosvm attempt: 28 got: resources_used.foo_i = 3 2017-05-17 13:42:37,361 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:37,412 INFOCLI centosvm: /opt/pbs/bin/qstat -f 6.centosvm 2017-05-17 13:42:37,443 INFO expect on server centosvm: resources_used.foo_i unset job 6.centosvm attempt: 29 got: resources_used.foo_i = 3 2017-05-17 13:42:37,946 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:37,988 INFOCLI centosvm: /opt/pbs/bin/qstat -f 6.centosvm 2017-05-17 13:42:38,017 INFO expect on server centosvm: resources_used.foo_i unset job 6.centosvm attempt: 30 got: resources_used.foo_i = 3 2017-05-17 13:42:38,520 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:38,565 INFOCLI centosvm: /opt/pbs/bin/qstat -f 6.centosvm 2017-05-17 13:42:38,602 INFO expect on server centosvm: resources_used.foo_i unset job 6.centosvm attempt: 31 got: resources_used.foo_i = 3 2017-05-17 13:42:39,106 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:39,163 INFOCLI centosvm: /opt/pbs/bin/qstat -f 6.centosvm 2017-05-17 13:42:39,211 INFO expect on server centosvm: resources_used.foo_i unset job 6.centosvm attempt: 32 got: resources_used.foo_i = 3 2017-05-17 13:42:39,714 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:39,768 INFOCLI centosvm: /opt/pbs/bin/qstat -f 6.centosvm 2017-05-17 13:42:39,807 INFO expect on server centosvm: resources_used.foo_i unset job 6.centosvm attempt: 33 got: resources_used.foo_i = 3 2017-05-17 13:42:40,310 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:40,357 INFOCLI centosvm: /opt/pbs/bin/qstat -f 6.centosvm 2017-05-17 13:42:40,393 INFO expect on server centosvm: resources_used.foo_i unset job 6.centosvm attempt: 34 got: resources_used.foo_i = 3 2017-05-17 13:42:40,895 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:40,931 INFOCLI centosvm: /opt/pbs/bin/qstat -f 6.centosvm 2017-05-17 13:42:40,961 INFO expect on server centosvm: resources_used.foo_i unset job 6.centosvm attempt: 35 got: resources_used.foo_i = 3 2017-05-17 13:42:41,462 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:41,484 INFOCLI centosvm: /opt/pbs/bin/qstat -f 6.centosvm 2017-05-17 13:42:41,503 INFO expect on server centosvm: resources_used.foo_i unset job 6.centosvm attempt: 36 got: resources_used.foo_i = 3 2017-05-17 13:42:42,004 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:42,055 INFOCLI centosvm: /opt/pbs/bin/qstat -f 6.centosvm 2017-05-17 13:42:42,086 INFO expect on server centosvm: resources_used.foo_i unset job 6.centosvm attempt: 37 got: resources_used.foo_i = 3 2017-05-17 13:42:42,588 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:42,615 INFOCLI centosvm: /opt/pbs/bin/qstat -f 6.centosvm 2017-05-17 13:42:42,639 INFO expect on server centosvm: resources_used.foo_i unset job 6.centosvm attempt: 38 got: resources_used.foo_i = 3 2017-05-17 13:42:43,142 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:43,183 INFOCLI centosvm: /opt/pbs/bin/qstat -f 6.centosvm 2017-05-17 13:42:43,213 INFO expect on server centosvm: resources_used.foo_i unset job 6.centosvm attempt: 39 got: resources_used.foo_i = 3 2017-05-17 13:42:43,714 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:43,762 INFOCLI centosvm: /opt/pbs/bin/qstat -f 6.centosvm 2017-05-17 13:42:43,801 INFO expect on server centosvm: resources_used.foo_i unset job 6.centosvm attempt: 40 got: resources_used.foo_i = 3 2017-05-17 13:42:44,303 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:44,354 INFOCLI centosvm: /opt/pbs/bin/qstat -f 6.centosvm 2017-05-17 13:42:44,381 INFO expect on server centosvm: resources_used.foo_i unset job 6.centosvm attempt: 41 got: resources_used.foo_i = 3 2017-05-17 13:42:44,883 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:44,930 INFOCLI centosvm: /opt/pbs/bin/qstat -f 6.centosvm 2017-05-17 13:42:44,957 INFO expect on server centosvm: resources_used.foo_i unset job 6.centosvm attempt: 42 got: resources_used.foo_i = 3 2017-05-17 13:42:45,460 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:45,495 INFOCLI centosvm: /opt/pbs/bin/qstat -f 6.centosvm 2017-05-17 13:42:45,527 INFO expect on server centosvm: resources_used.foo_i unset job 6.centosvm attempt: 43 got: resources_used.foo_i = 3 2017-05-17 13:42:46,030 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:46,067 INFOCLI centosvm: /opt/pbs/bin/qstat -f 6.centosvm 2017-05-17 13:42:46,098 INFO expect on server centosvm: resources_used.foo_i unset job 6.centosvm attempt: 44 got: resources_used.foo_i = 3 2017-05-17 13:42:46,600 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:46,651 INFOCLI centosvm: /opt/pbs/bin/qstat -f 6.centosvm 2017-05-17 13:42:46,681 INFO expect on server centosvm: resources_used.foo_i unset job 6.centosvm attempt: 45 got: resources_used.foo_i = 3 2017-05-17 13:42:47,182 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:47,208 INFOCLI centosvm: /opt/pbs/bin/qstat -f 6.centosvm 2017-05-17 13:42:47,229 INFO expect on server centosvm: resources_used.foo_i unset job 6.centosvm attempt: 46 got: resources_used.foo_i = 3 2017-05-17 13:42:47,730 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:47,778 INFOCLI centosvm: /opt/pbs/bin/qstat -f 6.centosvm 2017-05-17 13:42:47,814 INFO expect on server centosvm: resources_used.foo_i unset job 6.centosvm attempt: 47 got: resources_used.foo_i = 3 2017-05-17 13:42:48,316 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:48,362 INFOCLI centosvm: /opt/pbs/bin/qstat -f 6.centosvm 2017-05-17 13:42:48,392 INFO expect on server centosvm: resources_used.foo_i unset job 6.centosvm attempt: 48 got: resources_used.foo_i = 3 2017-05-17 13:42:48,894 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:48,940 INFOCLI centosvm: /opt/pbs/bin/qstat -f 6.centosvm 2017-05-17 13:42:48,967 INFO expect on server centosvm: resources_used.foo_i unset job 6.centosvm attempt: 49 got: resources_used.foo_i = 3 2017-05-17 13:42:49,470 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:49,514 INFOCLI centosvm: /opt/pbs/bin/qstat -f 6.centosvm 2017-05-17 13:42:49,549 INFO expect on server centosvm: resources_used.foo_i unset job 6.centosvm attempt: 50 got: resources_used.foo_i = 3 2017-05-17 13:42:50,052 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:50,103 INFOCLI centosvm: /opt/pbs/bin/qstat -f 6.centosvm 2017-05-17 13:42:50,145 INFO expect on server centosvm: resources_used.foo_i unset job 6.centosvm attempt: 51 got: resources_used.foo_i = 3 2017-05-17 13:42:50,647 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:50,693 INFOCLI centosvm: /opt/pbs/bin/qstat -f 6.centosvm 2017-05-17 13:42:50,730 INFO expect on server centosvm: resources_used.foo_i unset job 6.centosvm attempt: 52 got: resources_used.foo_i = 3 2017-05-17 13:42:51,234 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:51,282 INFOCLI centosvm: /opt/pbs/bin/qstat -f 6.centosvm 2017-05-17 13:42:51,324 INFO expect on server centosvm: resources_used.foo_i unset job 6.centosvm attempt: 53 got: resources_used.foo_i = 3 2017-05-17 13:42:51,827 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:51,872 INFOCLI centosvm: /opt/pbs/bin/qstat -f 6.centosvm 2017-05-17 13:42:51,902 INFO expect on server centosvm: resources_used.foo_i unset job 6.centosvm attempt: 54 got: resources_used.foo_i = 3 2017-05-17 13:42:52,406 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:52,432 INFOCLI centosvm: /opt/pbs/bin/qstat -f 6.centosvm 2017-05-17 13:42:52,447 INFO expect on server centosvm: resources_used.foo_i unset job 6.centosvm attempt: 55 got: resources_used.foo_i = 3 2017-05-17 13:42:52,950 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:52,995 INFOCLI centosvm: /opt/pbs/bin/qstat -f 6.centosvm 2017-05-17 13:42:53,032 INFO expect on server centosvm: resources_used.foo_i unset job 6.centosvm attempt: 56 got: resources_used.foo_i = 3 2017-05-17 13:42:53,536 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:53,583 INFOCLI centosvm: /opt/pbs/bin/qstat -f 6.centosvm 2017-05-17 13:42:53,613 INFO expect on server centosvm: resources_used.foo_i unset job 6.centosvm attempt: 57 got: resources_used.foo_i = 3 2017-05-17 13:42:54,116 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:54,159 INFOCLI centosvm: /opt/pbs/bin/qstat -f 6.centosvm 2017-05-17 13:42:54,187 INFO expect on server centosvm: resources_used.foo_i unset job 6.centosvm attempt: 58 got: resources_used.foo_i = 3 2017-05-17 13:42:54,690 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:54,718 INFOCLI centosvm: /opt/pbs/bin/qstat -f 6.centosvm 2017-05-17 13:42:54,745 INFO expect on server centosvm: resources_used.foo_i unset job 6.centosvm attempt: 59 got: resources_used.foo_i = 3 2017-05-17 13:42:55,247 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:55,290 INFOCLI centosvm: /opt/pbs/bin/qstat -f 6.centosvm 2017-05-17 13:42:55,315 INFO expect on server centosvm: resources_used.foo_i unset job 6.centosvm attempt: 60 got: resources_used.foo_i = 3 2017-05-17 13:42:55,818 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:42:55,866 INFO ERROR 2017-05-17 13:42:55,867 INFO =========================================== 2017-05-17 13:42:55,867 INFO Entered TestPbsAccumulateRescUsed tearDown 2017-05-17 13:42:55,867 INFO =========================================== 2017-05-17 13:42:55,867 INFO ============================================ 2017-05-17 13:42:55,868 INFO Completed TestPbsAccumulateRescUsed tearDown 2017-05-17 13:42:55,868 INFO ============================================ 2017-05-17 13:42:55,868 INFO test name: test_mom_down2 (tests.functional.pbs_accumulate_resc_used.TestPbsAccumulateRescUsed)... 2017-05-17 13:42:55,869 INFO test start time: Wed May 17 13:42:55 2017 2017-05-17 13:42:55,869 INFO test docstring: Test that when mom is down values are still accumulated for resources 2017-05-17 13:42:55,869 INFO ======================================== 2017-05-17 13:42:55,869 INFO Entered TestPbsAccumulateRescUsed setUp 2017-05-17 13:42:55,869 INFO ======================================== 2017-05-17 13:42:55,870 INFOCLI centosvm: /opt/pbs/bin/qstat -Bf centosvm 2017-05-17 13:42:55,883 INFO status on centosvm: server 2017-05-17 13:42:55,883 INFOCLI centosvm: /opt/pbs/bin/qstat -Bf centosvm 2017-05-17 13:42:55,894 INFO manager on centosvm: set server {'managers': (3, 'dilipk@*')} 2017-05-17 13:42:55,894 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c set server managers-=dilipk@* 2017-05-17 13:42:55,926 INFO manager on centosvm: set server {'managers': (2, 'dilipk@*')} 2017-05-17 13:42:55,926 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c set server managers+=dilipk@* 2017-05-17 13:42:55,952 INFO server centosvm: reverting configuration to defaults 2017-05-17 13:42:55,952 INFOCLI centosvm: /opt/pbs/bin/qstat -Bf centosvm 2017-05-17 13:42:55,964 INFO select on centosvm: __ALL__ 2017-05-17 13:42:55,965 INFOCLI centosvm: /opt/pbs/bin/qselect -x 2017-05-17 13:42:55,974 INFO delete job on centosvm: 5.centosvm, 6.centosvm 2017-05-17 13:42:55,975 INFOCLI centosvm: /opt/pbs/bin/qdel -W force -x 5.centosvm 6.centosvm 2017-05-17 13:42:56,011 INFOCLI centosvm: /opt/pbs/bin/qstat -f 5.centosvm 2017-05-17 13:42:56,052 INFOCLI centosvm: /opt/pbs/bin/qstat -f 6.centosvm 2017-05-17 13:42:56,068 INFOCLI centosvm: /opt/pbs/bin/qstat -f @centosvm 2017-05-17 13:42:56,087 INFO expect on server centosvm: job_state set 0 job ... OK 2017-05-17 13:42:56,088 INFOCLI centosvm: /opt/pbs/bin/pbs_rstat -f 2017-05-17 13:42:56,099 INFO manager on centosvm: unset server ['job_history_enable'] 2017-05-17 13:42:56,100 INFOCLI centosvm: /opt/pbs/bin/qmgr -c unset server job_history_enable 2017-05-17 13:42:56,117 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-05-17 13:42:56,142 INFO manager on centosvm: delete hook start,period 2017-05-17 13:42:56,142 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c delete hook start,period 2017-05-17 13:42:56,174 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook start 2017-05-17 13:42:56,193 INFO expect on server centosvm: unset hook start ... OK 2017-05-17 13:42:56,193 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook period 2017-05-17 13:42:56,221 INFO expect on server centosvm: unset hook period ... OK 2017-05-17 13:42:56,221 INFOCLI centosvm: /opt/pbs/bin/qstat -Qf @centosvm 2017-05-17 13:42:56,231 INFO manager on centosvm: delete queue workq 2017-05-17 13:42:56,231 INFOCLI centosvm: /opt/pbs/bin/qmgr -c delete queue workq 2017-05-17 13:42:56,245 INFOCLI centosvm: /opt/pbs/bin/qstat -Qf workq@centosvm 2017-05-17 13:42:56,258 INFO expect on server centosvm: unset queue workq ... OK 2017-05-17 13:42:56,258 INFO manager on centosvm: create queue workq {'started': 'True', 'queue_type': 'Execution', 'enabled': 'True'} 2017-05-17 13:42:56,258 INFOCLI centosvm: /opt/pbs/bin/qmgr -c create queue workq started=True,queue_type=Execution,enabled=True 2017-05-17 13:42:56,272 INFOCLI centosvm: /opt/pbs/bin/qstat -Qf workq@centosvm 2017-05-17 13:42:56,288 INFO expect on server centosvm: started set True || queue_type set Execution || enabled set True queue workq ... OK 2017-05-17 13:42:56,288 INFO manager on centosvm: set server {'node_fail_requeue': '310', 'default_queue': 'workq'} 2017-05-17 13:42:56,288 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server node_fail_requeue=310,default_queue=workq 2017-05-17 13:42:56,302 INFO status on centosvm: resource 2017-05-17 13:42:56,302 INFO manager on centosvm: list resource 2017-05-17 13:42:56,302 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource 2017-05-17 13:42:56,313 INFO manager on centosvm: delete resource foo_f,foo_i,foo_str2,foo_str3,foo_str4,foo_str,stra 2017-05-17 13:42:56,313 INFOCLI centosvm: /opt/pbs/bin/qmgr -c delete resource foo_f,foo_i,foo_str2,foo_str3,foo_str4,foo_str,stra 2017-05-17 13:42:56,499 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_f 2017-05-17 13:42:56,520 INFO expect on server centosvm: unset resource foo_f ... OK 2017-05-17 13:42:56,521 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_i 2017-05-17 13:42:56,542 INFO expect on server centosvm: unset resource foo_i ... OK 2017-05-17 13:42:56,543 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_str2 2017-05-17 13:42:56,564 INFO expect on server centosvm: unset resource foo_str2 ... OK 2017-05-17 13:42:56,564 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_str3 2017-05-17 13:42:56,585 INFO expect on server centosvm: unset resource foo_str3 ... OK 2017-05-17 13:42:56,586 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_str4 2017-05-17 13:42:56,612 INFO expect on server centosvm: unset resource foo_str4 ... OK 2017-05-17 13:42:56,613 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_str 2017-05-17 13:42:56,637 INFO expect on server centosvm: unset resource foo_str ... OK 2017-05-17 13:42:56,638 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource stra 2017-05-17 13:42:56,659 INFO expect on server centosvm: unset resource stra ... OK 2017-05-17 13:42:56,660 INFOCLI status on centosvm: server license_count 2017-05-17 13:42:56,660 INFOCLI centosvm: /opt/pbs/bin/qstat -Bf centosvm 2017-05-17 13:42:56,681 INFO server: centosvm licensed 2017-05-17 13:42:56,707 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/server_priv/comm.lock 2017-05-17 13:42:56,753 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-05-17 13:42:56,775 INFO scheduler centosvm: reverting configuration to defaults 2017-05-17 13:42:56,776 INFO manager on centosvm: list sched 2017-05-17 13:42:56,776 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list sched 2017-05-17 13:42:56,798 INFO manager on centosvm: unset sched ['sched_cycle_length'] 2017-05-17 13:42:56,798 INFOCLI centosvm: /opt/pbs/bin/qmgr -c unset sched sched_cycle_length 2017-05-17 13:42:56,825 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/dedicated_time 2017-05-17 13:42:56,850 INFOCLI2 centosvm: cmp /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-05-17 13:42:56,856 ERROR err: ['cmp: /var/spool/pbs/sched_priv/resource_group: Permission denied'] 2017-05-17 13:42:56,857 INFOCLI2 centosvm: sudo -H /usr/bin/cp -p /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-05-17 13:42:56,877 INFOCLI2 centosvm: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-05-17 13:42:56,898 INFO scheduler centosvm: reverting holidays file to default 2017-05-17 13:42:56,899 INFOCLI2 centosvm: cmp /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-05-17 13:42:56,905 ERROR err: ['cmp: /var/spool/pbs/sched_priv/holidays: Permission denied'] 2017-05-17 13:42:56,906 INFOCLI2 centosvm: sudo -H /usr/bin/cp -p /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-05-17 13:42:56,927 INFOCLI2 centosvm: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/holidays 2017-05-17 13:42:56,949 INFO scheduler centosvm: Parsing holidays file 2017-05-17 13:42:56,949 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2017-05-17 13:42:56,973 INFOCLI2 centosvm: cmp /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-05-17 13:42:56,979 ERROR err: ['cmp: /var/spool/pbs/sched_priv/sched_config: Permission denied'] 2017-05-17 13:42:56,980 INFOCLI2 centosvm: sudo -H /usr/bin/cp -p /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-05-17 13:42:56,996 INFOCLI2 centosvm: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-05-17 13:42:57,017 INFO scheduler centosvm: sent signal -HUP 2017-05-17 13:42:57,018 INFOCLI2 centosvm: sudo -H kill -HUP 84281 2017-05-17 13:42:57,034 INFOCLI2 centosvm: sudo -H /usr/bin/rm /var/spool/pbs/sched_priv/usage 2017-05-17 13:42:57,050 ERROR err: ['/usr/bin/rm: cannot remove \xe2\x80\x98/var/spool/pbs/sched_priv/usage\xe2\x80\x99: No such file or directory'] 2017-05-17 13:42:57,083 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-05-17 13:42:57,096 INFOCLI2 centosvm: sudo -H cmp /opt/pbs/sbin/pbs_mom /opt/pbs/sbin/pbs_mom.cpuset 2017-05-17 13:42:57,122 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:42:57,134 INFO mom centosvm: reverting configuration to defaults 2017-05-17 13:42:57,134 INFOCLI2 centosvm: sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2017-05-17 13:42:57,146 INFOCLI2 centosvm: sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2017-05-17 13:42:57,159 INFOCLI2 centosvm: sudo -H ls /var/spool/pbs/mom_priv/config.d 2017-05-17 13:42:57,172 INFO mom centosvm: sent signal -HUP 2017-05-17 13:42:57,172 INFOCLI2 centosvm: sudo -H kill -HUP 84278 2017-05-17 13:42:57,204 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:42:57,215 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v -a 2017-05-17 13:42:57,226 INFO status on centosvm: node centosvm 2017-05-17 13:42:57,226 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:42:57,241 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:42:57,252 INFO expect on server centosvm: state = free node centosvm ... OK 2017-05-17 13:42:57,252 INFOCLI2 centosvm2: ssh centosvm2 sudo -H cmp /opt/pbs/sbin/pbs_mom /opt/pbs/sbin/pbs_mom.cpuset 2017-05-17 13:42:58,371 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:42:58,841 INFO mom centosvm2: reverting configuration to defaults 2017-05-17 13:42:58,842 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2017-05-17 13:42:59,231 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2017-05-17 13:42:59,683 INFOCLI2 centosvm2: ssh centosvm2 sudo -H ls /var/spool/pbs/mom_priv/config.d 2017-05-17 13:43:00,045 INFO mom centosvm2: sent signal -HUP 2017-05-17 13:43:00,045 INFOCLI2 centosvm2: ssh centosvm2 sudo -H kill -HUP 9078 2017-05-17 13:43:00,958 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:43:01,448 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v -a 2017-05-17 13:43:01,474 INFO status on centosvm: node centosvm2 2017-05-17 13:43:01,475 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm2 2017-05-17 13:43:01,495 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm2 2017-05-17 13:43:01,510 INFO expect on server centosvm: state = free node centosvm2 ... OK 2017-05-17 13:43:01,510 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H cmp /opt/pbs/sbin/pbs_mom /opt/pbs/sbin/pbs_mom.cpuset 2017-05-17 13:43:02,518 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:43:02,925 INFO mom centosvm_nfs: reverting configuration to defaults 2017-05-17 13:43:02,926 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2017-05-17 13:43:03,361 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2017-05-17 13:43:03,866 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H ls /var/spool/pbs/mom_priv/config.d 2017-05-17 13:43:04,336 INFO mom centosvm_nfs: sent signal -HUP 2017-05-17 13:43:04,337 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H kill -HUP 76746 2017-05-17 13:43:05,311 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:43:05,721 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v -a 2017-05-17 13:43:05,737 INFO status on centosvm: node centosvm_nfs 2017-05-17 13:43:05,738 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm_nfs 2017-05-17 13:43:05,754 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm_nfs 2017-05-17 13:43:05,772 INFO expect on server centosvm: state = free node centosvm_nfs ... OK 2017-05-17 13:43:05,773 INFO ========================================= 2017-05-17 13:43:05,773 INFO Completed TestPbsAccumulateRescUsed setUp 2017-05-17 13:43:05,773 INFO ========================================= 2017-05-17 13:43:05,773 INFO len moms = 3 2017-05-17 13:43:05,773 INFOCLI2 centosvm: sudo -H /usr/bin/rm -rf /var/spool/pbs/mom_priv/config.d 2017-05-17 13:43:05,788 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/rm -rf /var/spool/pbs/mom_priv/config.d 2017-05-17 13:43:06,271 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H /usr/bin/rm -rf /var/spool/pbs/mom_priv/config.d 2017-05-17 13:43:06,677 INFO manager on centosvm: delete node 2017-05-17 13:43:06,678 INFOCLI centosvm: /opt/pbs/bin/qmgr -c delete node @default 2017-05-17 13:43:06,720 INFO manager on centosvm: create node centosvm 2017-05-17 13:43:06,721 INFOCLI centosvm: /opt/pbs/bin/qmgr -c create node centosvm 2017-05-17 13:43:06,751 INFO manager on centosvm: create node centosvm2 2017-05-17 13:43:06,752 INFOCLI centosvm: /opt/pbs/bin/qmgr -c create node centosvm2 2017-05-17 13:43:06,772 INFO manager on centosvm: create node centosvm_nfs 2017-05-17 13:43:06,773 INFOCLI centosvm: /opt/pbs/bin/qmgr -c create node centosvm_nfs 2017-05-17 13:43:06,794 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:43:06,815 INFO expect on server centosvm: state = free node centosvm got: state = state-unknown,down 2017-05-17 13:43:07,316 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:43:07,358 INFO expect on server centosvm: state = free node centosvm attempt: 2 got: state = state-unknown,down 2017-05-17 13:43:07,860 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:43:07,881 INFO expect on server centosvm: state = free node centosvm attempt: 3 got: state = state-unknown,down 2017-05-17 13:43:08,383 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:43:08,427 INFO expect on server centosvm: state = free node centosvm attempt: 4 ... OK 2017-05-17 13:43:08,428 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm2 2017-05-17 13:43:08,450 INFO expect on server centosvm: state = free node centosvm2 ... OK 2017-05-17 13:43:08,450 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm_nfs 2017-05-17 13:43:08,465 INFO expect on server centosvm: state = free node centosvm_nfs ... OK 2017-05-17 13:43:08,466 INFO status on centosvm: hook 2017-05-17 13:43:08,466 INFO manager on centosvm: list hook 2017-05-17 13:43:08,466 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-05-17 13:43:08,494 INFO status on centosvm: hook 2017-05-17 13:43:08,495 INFO manager on centosvm: list hook 2017-05-17 13:43:08,495 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-05-17 13:43:08,520 INFO manager on centosvm: create hook start 2017-05-17 13:43:08,521 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c create hook start 2017-05-17 13:43:08,544 INFO manager on centosvm: set hook start {'enabled': 'True', 'event': 'exechost_startup'} 2017-05-17 13:43:08,545 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c set hook start enabled=True,event=exechost_startup 2017-05-17 13:43:08,572 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook start 2017-05-17 13:43:08,594 INFO expect on server centosvm: enabled set True || event set exechost_startup hook start ... OK 2017-05-17 13:43:08,594 INFO manager on centosvm: import hook start {'content-type': 'application/x-python', 'input-file': '/tmp/PtlPbsKpPaKq', 'content-encoding': 'default'} 2017-05-17 13:43:08,594 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c import hook start application/x-python default /tmp/PtlPbsKpPaKq 2017-05-17 13:43:08,616 INFO server centosvm: imported hook body --- import pbs e=pbs.event() localnode=pbs.get_local_nodename() e.vnode_list[localnode].resources_available['foo_i'] = 7 e.vnode_list[localnode].resources_available['foo_f'] = 5.0 e.vnode_list[localnode].resources_available['foo_str'] = "seventyseven" --- 2017-05-17 13:43:08,616 INFO mom centosvm: sent signal -HUP 2017-05-17 13:43:08,617 INFOCLI2 centosvm: sudo -H kill -HUP 84278 2017-05-17 13:43:08,627 INFO mom centosvm2: sent signal -HUP 2017-05-17 13:43:08,627 INFOCLI2 centosvm2: ssh centosvm2 sudo -H kill -HUP 9078 2017-05-17 13:43:09,052 INFO mom centosvm_nfs: sent signal -HUP 2017-05-17 13:43:09,054 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H kill -HUP 76746 2017-05-17 13:43:09,529 INFO manager on centosvm: set server {'job_history_enable': 'True'} 2017-05-17 13:43:09,530 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server job_history_enable=True 2017-05-17 13:43:09,557 INFO manager on centosvm as root: create resource foo_str2 {'flag': 'h', 'type': 'string'} 2017-05-17 13:43:09,558 INFOCLI centosvm: sudo -H -u root /opt/pbs/bin/qmgr -c create resource foo_str2 flag=h,type=string 2017-05-17 13:43:09,613 INFO mom centosvm log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:43:10,042 INFO mom centosvm2 log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:43:10,390 INFO mom centosvm_nfs log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:43:10,391 INFO manager on centosvm as root: create resource foo_str3 {'flag': 'h', 'type': 'string'} 2017-05-17 13:43:10,391 INFOCLI centosvm: sudo -H -u root /opt/pbs/bin/qmgr -c create resource foo_str3 flag=h,type=string 2017-05-17 13:43:10,428 INFO mom centosvm log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:43:10,808 INFO mom centosvm2 log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:43:11,176 INFO mom centosvm_nfs log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:43:11,176 INFO manager on centosvm as root: create resource foo_str4 {'flag': 'h', 'type': 'string'} 2017-05-17 13:43:11,176 INFOCLI centosvm: sudo -H -u root /opt/pbs/bin/qmgr -c create resource foo_str4 flag=h,type=string 2017-05-17 13:43:11,269 INFO mom centosvm log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:43:11,649 INFO mom centosvm2 log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:43:12,036 INFO mom centosvm_nfs log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:43:12,036 INFO manager on centosvm as root: create resource stra {'flag': 'h', 'type': 'string_array'} 2017-05-17 13:43:12,037 INFOCLI centosvm: sudo -H -u root /opt/pbs/bin/qmgr -c create resource stra flag=h,type=string_array 2017-05-17 13:43:12,109 INFO mom centosvm log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:43:12,512 INFO mom centosvm2 log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:43:12,886 INFO mom centosvm_nfs log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:43:12,887 INFO status on centosvm: hook 2017-05-17 13:43:12,887 INFO manager on centosvm: list hook 2017-05-17 13:43:12,888 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-05-17 13:43:12,929 INFO status on centosvm: hook 2017-05-17 13:43:12,930 INFO manager on centosvm: list hook 2017-05-17 13:43:12,930 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-05-17 13:43:12,959 INFO manager on centosvm: create hook epi 2017-05-17 13:43:12,959 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c create hook epi 2017-05-17 13:43:12,986 INFO manager on centosvm: set hook epi {'enabled': 'True', 'event': 'execjob_epilogue'} 2017-05-17 13:43:12,986 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c set hook epi enabled=True,event=execjob_epilogue 2017-05-17 13:43:13,007 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook epi 2017-05-17 13:43:13,031 INFO expect on server centosvm: enabled set True || event set execjob_epilogue hook epi ... OK 2017-05-17 13:43:13,031 INFO manager on centosvm: import hook epi {'content-type': 'application/x-python', 'input-file': '/tmp/PtlPbseEcSte', 'content-encoding': 'default'} 2017-05-17 13:43:13,031 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c import hook epi application/x-python default /tmp/PtlPbseEcSte 2017-05-17 13:43:13,060 INFO server centosvm: imported hook body --- import pbs e=pbs.event() pbs.logmsg(pbs.LOG_DEBUG, "executed epilogue hook") if e.job.in_ms_mom(): e.job.resources_used["vmem"] = pbs.size("9gb") e.job.resources_used["foo_i"] = 9 e.job.resources_used["foo_f"] = 0.09 e.job.resources_used["foo_str"] = '{"seven":7}' e.job.resources_used["cput"] = 10 e.job.resources_used["stra"] = '"glad,elated","happy"' else: e.job.resources_used["vmem"] = pbs.size("10gb") e.job.resources_used["foo_i"] = 10 e.job.resources_used["foo_f"] = 0.10 e.job.resources_used["foo_str"] = '{"eight":8,"nine":9}' e.job.resources_used["cput"] = 20 e.job.resources_used["stra"] = '"cucumbers,bananas"' --- 2017-05-17 13:43:13,061 INFO job: executable set to /bin/sleep with arguments: 100 2017-05-17 13:43:13,061 INFO job: executable set to /bin/sleep with arguments: 10 2017-05-17 13:43:13,061 INFOCLI centosvm: sudo -H -u pbsuser /opt/pbs/bin/qsub -l place=scatter -l walltime=10 -l select=3:ncpus=1 -- /bin/sleep 10 2017-05-17 13:43:13,085 INFO submit to centosvm as pbsuser: job 7.centosvm OrderedDict([('Resource_List.place', 'scatter'), ('Resource_List.walltime', 10), ('Resource_List.select', '3:ncpus=1')]) 2017-05-17 13:43:13,085 INFOCLI centosvm: /opt/pbs/bin/qstat -f 7.centosvm 2017-05-17 13:43:13,096 INFO expect on server centosvm: job_state = R && substate = 42 job 7.centosvm got: job_state = Q 2017-05-17 13:43:13,598 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:43:13,637 INFOCLI centosvm: /opt/pbs/bin/qstat -f 7.centosvm 2017-05-17 13:43:13,671 INFO expect on server centosvm: job_state = R && substate = 42 job 7.centosvm attempt: 2 got: substate = 41 2017-05-17 13:43:14,173 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:43:14,221 INFOCLI centosvm: /opt/pbs/bin/qstat -f 7.centosvm 2017-05-17 13:43:14,251 INFO expect on server centosvm: job_state = R && substate = 42 job 7.centosvm attempt: 3 got: substate = 41 2017-05-17 13:43:14,753 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:43:14,779 INFOCLI centosvm: /opt/pbs/bin/qstat -f 7.centosvm 2017-05-17 13:43:14,801 INFO expect on server centosvm: job_state = R && substate = 42 job 7.centosvm attempt: 4 ... OK 2017-05-17 13:43:14,801 INFO mom centosvm2: stopping MoM on host centosvm2 2017-05-17 13:43:14,802 INFOCLI2 centosvm2: ssh centosvm2 sudo -H kill -TERM 9078 2017-05-17 13:43:15,202 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:43:15,993 INFO server centosvm: expect offset set to 10 2017-05-17 13:43:25,999 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 7.centosvm 2017-05-17 13:43:26,048 INFO expect on server centosvm: job_state = F || resources_used.foo_i = 19 || resources_used.foo_f = 0.19 || resources_used.foo_str = '{"eight": 8, "seven": 7, "nine": 9}' job 7.centosvm ... OK 2017-05-17 13:43:26,049 INFO service: starting pbs_mom -d /var/spool/pbs 2017-05-17 13:43:26,049 INFOCLI centosvm2: ssh centosvm2 sudo -H /opt/pbs/sbin/pbs_mom -d /var/spool/pbs 2017-05-17 13:43:29,340 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:43:30,807 INFO =========================================== 2017-05-17 13:43:30,808 INFO Entered TestPbsAccumulateRescUsed tearDown 2017-05-17 13:43:30,808 INFO =========================================== 2017-05-17 13:43:30,808 INFO ============================================ 2017-05-17 13:43:30,808 INFO Completed TestPbsAccumulateRescUsed tearDown 2017-05-17 13:43:30,808 INFO ============================================ 2017-05-17 13:43:30,808 INFO ok 2017-05-17 13:43:30,809 INFO test name: test_periodic (tests.functional.pbs_accumulate_resc_used.TestPbsAccumulateRescUsed)... 2017-05-17 13:43:30,809 INFO test start time: Wed May 17 13:43:30 2017 2017-05-17 13:43:30,809 INFO test docstring: Test accumulatinon of resources from an exechost_periodic hook. 2017-05-17 13:43:30,809 INFO ======================================== 2017-05-17 13:43:30,809 INFO Entered TestPbsAccumulateRescUsed setUp 2017-05-17 13:43:30,809 INFO ======================================== 2017-05-17 13:43:30,809 INFOCLI centosvm: /opt/pbs/bin/qstat -Bf centosvm 2017-05-17 13:43:30,822 INFO status on centosvm: server 2017-05-17 13:43:30,822 INFOCLI centosvm: /opt/pbs/bin/qstat -Bf centosvm 2017-05-17 13:43:30,835 INFO manager on centosvm: set server {'managers': (3, 'dilipk@*')} 2017-05-17 13:43:30,835 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c set server managers-=dilipk@* 2017-05-17 13:43:30,861 INFO manager on centosvm: set server {'managers': (2, 'dilipk@*')} 2017-05-17 13:43:30,861 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c set server managers+=dilipk@* 2017-05-17 13:43:30,885 INFO server centosvm: reverting configuration to defaults 2017-05-17 13:43:30,885 INFOCLI centosvm: /opt/pbs/bin/qstat -Bf centosvm 2017-05-17 13:43:30,898 INFO select on centosvm: __ALL__ 2017-05-17 13:43:30,898 INFOCLI centosvm: /opt/pbs/bin/qselect -x 2017-05-17 13:43:30,908 INFO delete job on centosvm: 7.centosvm 2017-05-17 13:43:30,908 INFOCLI centosvm: /opt/pbs/bin/qdel -W force -x 7.centosvm 2017-05-17 13:43:30,922 INFOCLI centosvm: /opt/pbs/bin/qstat -f 7.centosvm 2017-05-17 13:43:30,937 INFOCLI centosvm: /opt/pbs/bin/qstat -f @centosvm 2017-05-17 13:43:30,948 INFO expect on server centosvm: job_state set 0 job ... OK 2017-05-17 13:43:30,948 INFOCLI centosvm: /opt/pbs/bin/pbs_rstat -f 2017-05-17 13:43:30,958 INFO manager on centosvm: unset server ['job_history_enable'] 2017-05-17 13:43:30,958 INFOCLI centosvm: /opt/pbs/bin/qmgr -c unset server job_history_enable 2017-05-17 13:43:30,974 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-05-17 13:43:30,994 INFO manager on centosvm: delete hook start,epi 2017-05-17 13:43:30,994 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c delete hook start,epi 2017-05-17 13:43:31,020 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook start 2017-05-17 13:43:31,041 INFO expect on server centosvm: unset hook start ... OK 2017-05-17 13:43:31,041 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook epi 2017-05-17 13:43:31,071 INFO expect on server centosvm: unset hook epi ... OK 2017-05-17 13:43:31,071 INFOCLI centosvm: /opt/pbs/bin/qstat -Qf @centosvm 2017-05-17 13:43:31,082 INFO manager on centosvm: delete queue workq 2017-05-17 13:43:31,082 INFOCLI centosvm: /opt/pbs/bin/qmgr -c delete queue workq 2017-05-17 13:43:31,097 INFOCLI centosvm: /opt/pbs/bin/qstat -Qf workq@centosvm 2017-05-17 13:43:31,108 INFO expect on server centosvm: unset queue workq ... OK 2017-05-17 13:43:31,108 INFO manager on centosvm: create queue workq {'started': 'True', 'queue_type': 'Execution', 'enabled': 'True'} 2017-05-17 13:43:31,108 INFOCLI centosvm: /opt/pbs/bin/qmgr -c create queue workq started=True,queue_type=Execution,enabled=True 2017-05-17 13:43:31,121 INFOCLI centosvm: /opt/pbs/bin/qstat -Qf workq@centosvm 2017-05-17 13:43:31,135 INFO expect on server centosvm: started set True || queue_type set Execution || enabled set True queue workq ... OK 2017-05-17 13:43:31,135 INFO manager on centosvm: set server {'default_queue': 'workq'} 2017-05-17 13:43:31,135 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server default_queue=workq 2017-05-17 13:43:31,150 INFO status on centosvm: resource 2017-05-17 13:43:31,150 INFO manager on centosvm: list resource 2017-05-17 13:43:31,150 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource 2017-05-17 13:43:31,161 INFO manager on centosvm: delete resource foo_f,foo_i,foo_str2,foo_str3,foo_str4,foo_str,stra 2017-05-17 13:43:31,162 INFOCLI centosvm: /opt/pbs/bin/qmgr -c delete resource foo_f,foo_i,foo_str2,foo_str3,foo_str4,foo_str,stra 2017-05-17 13:43:31,332 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_f 2017-05-17 13:43:31,356 INFO expect on server centosvm: unset resource foo_f ... OK 2017-05-17 13:43:31,358 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_i 2017-05-17 13:43:31,393 INFO expect on server centosvm: unset resource foo_i ... OK 2017-05-17 13:43:31,394 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_str2 2017-05-17 13:43:31,443 INFO expect on server centosvm: unset resource foo_str2 ... OK 2017-05-17 13:43:31,446 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_str3 2017-05-17 13:43:31,478 INFO expect on server centosvm: unset resource foo_str3 ... OK 2017-05-17 13:43:31,479 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_str4 2017-05-17 13:43:31,494 INFO expect on server centosvm: unset resource foo_str4 ... OK 2017-05-17 13:43:31,494 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_str 2017-05-17 13:43:31,507 INFO expect on server centosvm: unset resource foo_str ... OK 2017-05-17 13:43:31,508 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource stra 2017-05-17 13:43:31,521 INFO expect on server centosvm: unset resource stra ... OK 2017-05-17 13:43:31,521 INFOCLI status on centosvm: server license_count 2017-05-17 13:43:31,522 INFOCLI centosvm: /opt/pbs/bin/qstat -Bf centosvm 2017-05-17 13:43:31,534 INFO server: centosvm licensed 2017-05-17 13:43:31,555 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/server_priv/comm.lock 2017-05-17 13:43:31,580 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-05-17 13:43:31,592 INFO scheduler centosvm: reverting configuration to defaults 2017-05-17 13:43:31,592 INFO manager on centosvm: list sched 2017-05-17 13:43:31,592 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list sched 2017-05-17 13:43:31,605 INFO manager on centosvm: unset sched ['sched_cycle_length'] 2017-05-17 13:43:31,605 INFOCLI centosvm: /opt/pbs/bin/qmgr -c unset sched sched_cycle_length 2017-05-17 13:43:31,619 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/dedicated_time 2017-05-17 13:43:31,638 INFOCLI2 centosvm: cmp /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-05-17 13:43:31,642 ERROR err: ['cmp: /var/spool/pbs/sched_priv/resource_group: Permission denied'] 2017-05-17 13:43:31,642 INFOCLI2 centosvm: sudo -H /usr/bin/cp -p /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-05-17 13:43:31,654 INFOCLI2 centosvm: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-05-17 13:43:31,665 INFO scheduler centosvm: reverting holidays file to default 2017-05-17 13:43:31,665 INFOCLI2 centosvm: cmp /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-05-17 13:43:31,672 ERROR err: ['cmp: /var/spool/pbs/sched_priv/holidays: Permission denied'] 2017-05-17 13:43:31,673 INFOCLI2 centosvm: sudo -H /usr/bin/cp -p /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-05-17 13:43:31,685 INFOCLI2 centosvm: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/holidays 2017-05-17 13:43:31,697 INFO scheduler centosvm: Parsing holidays file 2017-05-17 13:43:31,697 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2017-05-17 13:43:31,716 INFOCLI2 centosvm: cmp /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-05-17 13:43:31,719 ERROR err: ['cmp: /var/spool/pbs/sched_priv/sched_config: Permission denied'] 2017-05-17 13:43:31,720 INFOCLI2 centosvm: sudo -H /usr/bin/cp -p /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-05-17 13:43:31,732 INFOCLI2 centosvm: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-05-17 13:43:31,746 INFO scheduler centosvm: sent signal -HUP 2017-05-17 13:43:31,746 INFOCLI2 centosvm: sudo -H kill -HUP 84281 2017-05-17 13:43:31,759 INFOCLI2 centosvm: sudo -H /usr/bin/rm /var/spool/pbs/sched_priv/usage 2017-05-17 13:43:31,775 ERROR err: ['/usr/bin/rm: cannot remove \xe2\x80\x98/var/spool/pbs/sched_priv/usage\xe2\x80\x99: No such file or directory'] 2017-05-17 13:43:31,801 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-05-17 13:43:31,814 INFOCLI2 centosvm: sudo -H cmp /opt/pbs/sbin/pbs_mom /opt/pbs/sbin/pbs_mom.cpuset 2017-05-17 13:43:31,838 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:43:31,850 INFO mom centosvm: reverting configuration to defaults 2017-05-17 13:43:31,850 INFOCLI2 centosvm: sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2017-05-17 13:43:31,860 INFOCLI2 centosvm: sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2017-05-17 13:43:31,870 INFOCLI2 centosvm: sudo -H ls /var/spool/pbs/mom_priv/config.d 2017-05-17 13:43:31,880 INFO mom centosvm: sent signal -HUP 2017-05-17 13:43:31,880 INFOCLI2 centosvm: sudo -H kill -HUP 84278 2017-05-17 13:43:31,905 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:43:31,915 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v -a 2017-05-17 13:43:31,926 INFO status on centosvm: node centosvm 2017-05-17 13:43:31,926 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:43:31,938 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:43:31,951 INFO expect on server centosvm: state = free node centosvm ... OK 2017-05-17 13:43:31,951 INFOCLI2 centosvm2: ssh centosvm2 sudo -H cmp /opt/pbs/sbin/pbs_mom /opt/pbs/sbin/pbs_mom.cpuset 2017-05-17 13:43:32,945 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:43:33,330 INFO mom centosvm2: reverting configuration to defaults 2017-05-17 13:43:33,331 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2017-05-17 13:43:33,754 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2017-05-17 13:43:34,219 INFOCLI2 centosvm2: ssh centosvm2 sudo -H ls /var/spool/pbs/mom_priv/config.d 2017-05-17 13:43:34,619 INFO mom centosvm2: sent signal -HUP 2017-05-17 13:43:34,620 INFOCLI2 centosvm2: ssh centosvm2 sudo -H kill -HUP 15795 2017-05-17 13:43:35,526 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:43:36,078 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v -a 2017-05-17 13:43:36,115 INFO status on centosvm: node centosvm2 2017-05-17 13:43:36,116 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm2 2017-05-17 13:43:36,146 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm2 2017-05-17 13:43:36,167 INFO expect on server centosvm: state = free node centosvm2 ... OK 2017-05-17 13:43:36,167 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H cmp /opt/pbs/sbin/pbs_mom /opt/pbs/sbin/pbs_mom.cpuset 2017-05-17 13:43:37,216 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:43:37,628 INFO mom centosvm_nfs: reverting configuration to defaults 2017-05-17 13:43:37,628 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2017-05-17 13:43:38,026 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2017-05-17 13:43:38,429 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H ls /var/spool/pbs/mom_priv/config.d 2017-05-17 13:43:38,843 INFO mom centosvm_nfs: sent signal -HUP 2017-05-17 13:43:38,844 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H kill -HUP 76746 2017-05-17 13:43:39,830 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:43:40,329 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v -a 2017-05-17 13:43:40,365 INFO status on centosvm: node centosvm_nfs 2017-05-17 13:43:40,366 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm_nfs 2017-05-17 13:43:40,389 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm_nfs 2017-05-17 13:43:40,407 INFO expect on server centosvm: state = free node centosvm_nfs ... OK 2017-05-17 13:43:40,408 INFO ========================================= 2017-05-17 13:43:40,408 INFO Completed TestPbsAccumulateRescUsed setUp 2017-05-17 13:43:40,408 INFO ========================================= 2017-05-17 13:43:40,408 INFO len moms = 3 2017-05-17 13:43:40,409 INFOCLI2 centosvm: sudo -H /usr/bin/rm -rf /var/spool/pbs/mom_priv/config.d 2017-05-17 13:43:40,429 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/rm -rf /var/spool/pbs/mom_priv/config.d 2017-05-17 13:43:40,802 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H /usr/bin/rm -rf /var/spool/pbs/mom_priv/config.d 2017-05-17 13:43:41,176 INFO manager on centosvm: delete node 2017-05-17 13:43:41,177 INFOCLI centosvm: /opt/pbs/bin/qmgr -c delete node @default 2017-05-17 13:43:41,195 INFO manager on centosvm: create node centosvm 2017-05-17 13:43:41,196 INFOCLI centosvm: /opt/pbs/bin/qmgr -c create node centosvm 2017-05-17 13:43:41,218 INFO manager on centosvm: create node centosvm2 2017-05-17 13:43:41,219 INFOCLI centosvm: /opt/pbs/bin/qmgr -c create node centosvm2 2017-05-17 13:43:41,238 INFO manager on centosvm: create node centosvm_nfs 2017-05-17 13:43:41,238 INFOCLI centosvm: /opt/pbs/bin/qmgr -c create node centosvm_nfs 2017-05-17 13:43:41,256 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:43:41,273 INFO expect on server centosvm: state = free node centosvm got: state = state-unknown,down 2017-05-17 13:43:41,774 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:43:41,814 INFO expect on server centosvm: state = free node centosvm attempt: 2 got: state = state-unknown,down 2017-05-17 13:43:42,316 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:43:42,356 INFO expect on server centosvm: state = free node centosvm attempt: 3 got: state = state-unknown,down 2017-05-17 13:43:42,858 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:43:42,903 INFO expect on server centosvm: state = free node centosvm attempt: 4 got: state = state-unknown,down 2017-05-17 13:43:43,405 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:43:43,457 INFO expect on server centosvm: state = free node centosvm attempt: 5 ... OK 2017-05-17 13:43:43,459 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm2 2017-05-17 13:43:43,503 INFO expect on server centosvm: state = free node centosvm2 ... OK 2017-05-17 13:43:43,504 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm_nfs 2017-05-17 13:43:43,534 INFO expect on server centosvm: state = free node centosvm_nfs ... OK 2017-05-17 13:43:43,534 INFO status on centosvm: hook 2017-05-17 13:43:43,534 INFO manager on centosvm: list hook 2017-05-17 13:43:43,535 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-05-17 13:43:43,565 INFO status on centosvm: hook 2017-05-17 13:43:43,565 INFO manager on centosvm: list hook 2017-05-17 13:43:43,566 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-05-17 13:43:43,592 INFO manager on centosvm: create hook start 2017-05-17 13:43:43,592 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c create hook start 2017-05-17 13:43:43,620 INFO manager on centosvm: set hook start {'enabled': 'True', 'event': 'exechost_startup'} 2017-05-17 13:43:43,620 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c set hook start enabled=True,event=exechost_startup 2017-05-17 13:43:43,650 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook start 2017-05-17 13:43:43,682 INFO expect on server centosvm: enabled set True || event set exechost_startup hook start ... OK 2017-05-17 13:43:43,683 INFO manager on centosvm: import hook start {'content-type': 'application/x-python', 'input-file': '/tmp/PtlPbsDESSAA', 'content-encoding': 'default'} 2017-05-17 13:43:43,683 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c import hook start application/x-python default /tmp/PtlPbsDESSAA 2017-05-17 13:43:43,711 INFO server centosvm: imported hook body --- import pbs e=pbs.event() localnode=pbs.get_local_nodename() e.vnode_list[localnode].resources_available['foo_i'] = 7 e.vnode_list[localnode].resources_available['foo_f'] = 5.0 e.vnode_list[localnode].resources_available['foo_str'] = "seventyseven" --- 2017-05-17 13:43:43,711 INFO mom centosvm: sent signal -HUP 2017-05-17 13:43:43,712 INFOCLI2 centosvm: sudo -H kill -HUP 84278 2017-05-17 13:43:43,737 INFO mom centosvm2: sent signal -HUP 2017-05-17 13:43:43,737 INFOCLI2 centosvm2: ssh centosvm2 sudo -H kill -HUP 15795 2017-05-17 13:43:44,143 INFO mom centosvm_nfs: sent signal -HUP 2017-05-17 13:43:44,143 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H kill -HUP 76746 2017-05-17 13:43:44,502 INFO manager on centosvm: set server {'job_history_enable': 'True'} 2017-05-17 13:43:44,503 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server job_history_enable=True 2017-05-17 13:43:44,535 INFO manager on centosvm as root: create resource foo_str2 {'flag': 'h', 'type': 'string'} 2017-05-17 13:43:44,536 INFOCLI centosvm: sudo -H -u root /opt/pbs/bin/qmgr -c create resource foo_str2 flag=h,type=string 2017-05-17 13:43:44,597 INFO mom centosvm log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:43:44,937 INFO mom centosvm2 log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:43:45,530 INFO mom centosvm_nfs log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:43:45,531 INFO manager on centosvm as root: create resource foo_str3 {'flag': 'h', 'type': 'string'} 2017-05-17 13:43:45,531 INFOCLI centosvm: sudo -H -u root /opt/pbs/bin/qmgr -c create resource foo_str3 flag=h,type=string 2017-05-17 13:43:45,596 INFO mom centosvm log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:43:45,949 INFO mom centosvm2 log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:43:46,298 INFO mom centosvm_nfs log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:43:46,299 INFO manager on centosvm as root: create resource foo_str4 {'flag': 'h', 'type': 'string'} 2017-05-17 13:43:46,299 INFOCLI centosvm: sudo -H -u root /opt/pbs/bin/qmgr -c create resource foo_str4 flag=h,type=string 2017-05-17 13:43:46,340 INFO mom centosvm log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:43:46,687 INFO mom centosvm2 log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:43:47,062 INFO mom centosvm_nfs log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:43:47,063 INFO manager on centosvm as root: create resource stra {'flag': 'h', 'type': 'string_array'} 2017-05-17 13:43:47,063 INFOCLI centosvm: sudo -H -u root /opt/pbs/bin/qmgr -c create resource stra flag=h,type=string_array 2017-05-17 13:43:47,134 INFO mom centosvm log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:43:47,538 INFO mom centosvm2 log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:43:48,014 INFO mom centosvm_nfs log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:43:48,015 INFO test_periodic 2017-05-17 13:43:48,015 INFO status on centosvm: hook 2017-05-17 13:43:48,015 INFO manager on centosvm: list hook 2017-05-17 13:43:48,016 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-05-17 13:43:48,057 INFO status on centosvm: hook 2017-05-17 13:43:48,058 INFO manager on centosvm: list hook 2017-05-17 13:43:48,058 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-05-17 13:43:48,087 INFO manager on centosvm: create hook period 2017-05-17 13:43:48,088 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c create hook period 2017-05-17 13:43:48,122 INFO manager on centosvm: set hook period {'freq': 15, 'enabled': 'True', 'event': 'exechost_periodic'} 2017-05-17 13:43:48,122 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c set hook period freq=15,enabled=True,event=exechost_periodic 2017-05-17 13:43:48,149 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook period 2017-05-17 13:43:48,176 INFO expect on server centosvm: freq set 15 || enabled set True || event set exechost_periodic hook period ... OK 2017-05-17 13:43:48,177 INFO manager on centosvm: import hook period {'content-type': 'application/x-python', 'input-file': '/tmp/PtlPbs0oqJ0k', 'content-encoding': 'default'} 2017-05-17 13:43:48,177 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c import hook period application/x-python default /tmp/PtlPbs0oqJ0k 2017-05-17 13:43:48,209 INFO server centosvm: imported hook body --- import pbs e=pbs.event() pbs.logmsg(pbs.LOG_DEBUG, "executed periodic hook") i = 0 l = [] for v in pbs.server().vnodes(): pbs.logmsg(pbs.LOG_DEBUG, "node %s" % (v.name,)) l.append(v.name) local_node=pbs.get_local_nodename() for jk in e.job_list.keys(): if local_node == l[0]: e.job_list[jk].resources_used["vmem"] = pbs.size("11gb") e.job_list[jk].resources_used["foo_i"] = 11 e.job_list[jk].resources_used["foo_f"] = 0.11 e.job_list[jk].resources_used["foo_str"] = '{"seven":7}' e.job_list[jk].resources_used["cput"] = 11 e.job_list[jk].resources_used["stra"] = '"glad,elated","happy"' e.job_list[jk].resources_used["foo_str3"] = """{"a":6,"b":"some value #$%^&*@","c":54.4,"d":"32.5gb"}""" e.job_list[jk].resources_used["foo_str2"] = "seven" elif local_node == l[1]: e.job_list[jk].resources_used["vmem"] = pbs.size("12gb") e.job_list[jk].resources_used["foo_i"] = 12 e.job_list[jk].resources_used["foo_f"] = 0.12 e.job_list[jk].resources_used["foo_str"] = '{"eight":8}' e.job_list[jk].resources_used["cput"] = 12 e.job_list[jk].resources_used["stra"] = '"cucumbers,bananas"' e.job_list[jk].resources_used["foo_str2"] = '{"seven":7}' e.job_list[jk].resources_used["foo_str3"] = """{"vn1":4,"vn2":5,"vn3":6}""" else: e.job_list[jk].resources_used["vmem"] = pbs.size("13gb") e.job_list[jk].resources_used["foo_i"] = 13 e.job_list[jk].resources_used["foo_f"] = 0.13 e.job_list[jk].resources_used["foo_str"] = '{"nine":9}' e.job_list[jk].resources_used["foo_str2"] = '{"seven":7}' e.job_list[jk].resources_used["cput"] = 13 e.job_list[jk].resources_used["stra"] = '"cucumbers,bananas"' e.job_list[jk].resources_used["foo_str3"] = """{"vn1":4,"vn2":5,"vn3":6}""" --- 2017-05-17 13:43:48,209 INFO manager on centosvm: set node centosvm {'resources_available.ncpus': '2'} 2017-05-17 13:43:48,210 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set node centosvm resources_available.ncpus=2 2017-05-17 13:43:48,228 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:43:48,248 INFO expect on server centosvm: resources_available.ncpus set 2 node centosvm ... OK 2017-05-17 13:43:48,248 INFO manager on centosvm: set node centosvm2 {'resources_available.ncpus': '2'} 2017-05-17 13:43:48,248 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set node centosvm2 resources_available.ncpus=2 2017-05-17 13:43:48,273 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm2 2017-05-17 13:43:48,289 INFO expect on server centosvm: resources_available.ncpus set 2 node centosvm2 ... OK 2017-05-17 13:43:48,289 INFO manager on centosvm: set node centosvm_nfs {'resources_available.ncpus': '2'} 2017-05-17 13:43:48,289 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set node centosvm_nfs resources_available.ncpus=2 2017-05-17 13:43:48,311 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm_nfs 2017-05-17 13:43:48,326 INFO expect on server centosvm: resources_available.ncpus set 2 node centosvm_nfs ... OK 2017-05-17 13:43:48,327 INFO job: executable set to /bin/sleep with arguments: 100 2017-05-17 13:43:48,327 INFO job: executable set to /bin/sleep with arguments: 35 2017-05-17 13:43:48,328 INFOCLI centosvm: sudo -H -u pbsuser /opt/pbs/bin/qsub -l place=scatter -l select=3:ncpus=1 -- /bin/sleep 35 2017-05-17 13:43:48,359 INFO submit to centosvm as pbsuser: job 8.centosvm OrderedDict([('Resource_List.place', 'scatter'), ('Resource_List.select', '3:ncpus=1')]) 2017-05-17 13:43:48,359 INFOCLI centosvm: sudo -H -u pbsuser /opt/pbs/bin/qsub -l place=scatter -l select=3:ncpus=1 -- /bin/sleep 35 2017-05-17 13:43:48,374 INFO submit to centosvm as pbsuser: job 9.centosvm OrderedDict([('Resource_List.place', 'scatter'), ('Resource_List.select', '3:ncpus=1')]) 2017-05-17 13:43:48,375 INFO server centosvm: expect offset set to 35 2017-05-17 13:44:23,414 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 8.centosvm 2017-05-17 13:44:23,456 INFO expect on server centosvm: resources_used.cput = 00:00:36 || resources_used.foo_i = 36 || resources_used.ncpus = 3 || job_state = F || resources_used.foo_f = 0.36 || resources_used.vmem = 36gb || resources_used.stra = "glad,elated","happy" job 8.centosvm got: resources_used.cput = 11 2017-05-17 13:44:23,962 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:44:23,991 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 8.centosvm 2017-05-17 13:44:24,030 INFO expect on server centosvm: resources_used.cput = 00:00:36 || resources_used.foo_i = 36 || resources_used.ncpus = 3 || job_state = F || resources_used.foo_f = 0.36 || resources_used.vmem = 36gb || resources_used.stra = "glad,elated","happy" job 8.centosvm attempt: 2 got: resources_used.cput = 11 2017-05-17 13:44:24,533 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:44:24,585 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 8.centosvm 2017-05-17 13:44:24,615 INFO expect on server centosvm: resources_used.cput = 00:00:36 || resources_used.foo_i = 36 || resources_used.ncpus = 3 || job_state = F || resources_used.foo_f = 0.36 || resources_used.vmem = 36gb || resources_used.stra = "glad,elated","happy" job 8.centosvm attempt: 3 got: job_state = E 2017-05-17 13:44:25,120 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:44:25,165 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 8.centosvm 2017-05-17 13:44:25,205 INFO expect on server centosvm: resources_used.cput = 00:00:36 || resources_used.foo_i = 36 || resources_used.ncpus = 3 || job_state = F || resources_used.foo_f = 0.36 || resources_used.vmem = 36gb || resources_used.stra = "glad,elated","happy" job 8.centosvm attempt: 4 ... OK 2017-05-17 13:44:25,206 INFO status on centosvm: job 8.centosvm resources_used.foo_str 2017-05-17 13:44:25,206 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 8.centosvm 2017-05-17 13:44:25,245 INFO status on centosvm: job 8.centosvm resources_used.foo_str3 2017-05-17 13:44:25,245 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 8.centosvm 2017-05-17 13:44:25,281 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 8.centosvm 2017-05-17 13:44:25,320 INFO expect on server centosvm: resources_used.foo_str2 unset job 8.centosvm ... OK 2017-05-17 13:44:25,366 INFO centosvm log match: searching for "E;8.centosvm;.*resources_used.foo_f=0.36.*" - using regular expression ... OK 2017-05-17 13:44:25,382 INFO centosvm log match: searching for "E;8.centosvm;.*resources_used.foo_i=36.*" - using regular expression ... OK 2017-05-17 13:44:25,398 INFO centosvm log match: searching for "E;8.centosvm;.*resources_used.foo_str='{"nine": 9, "seven": 7, "eight": 8}'.*" - using regular expression ... OK 2017-05-17 13:44:25,425 INFO centosvm log match: searching for "E;8.centosvm;.*resources_used.vmem=36gb.*" - using regular expression ... OK 2017-05-17 13:44:25,442 INFO centosvm log match: searching for "E;8.centosvm;.*resources_used.cput=00:00:36.*" - using regular expression ... OK 2017-05-17 13:44:25,459 INFO centosvm log match: searching for "E;8.centosvm;.*('resources_used.foo_str2=',).*" - using regular expression - No match 2017-05-17 13:44:26,500 INFO centosvm log match: searching for "E;8.centosvm;.*resources_used.ncpus=3.*" - using regular expression ... OK 2017-05-17 13:44:26,528 INFO centosvm log match: searching for "E;8.centosvm;.*resources_used.foo_str3='{"a": 6, "c": 54\.4, "b": "some value \#\$\%\^\&\*\@", "vn3": 6, "vn2": 5, "vn1": 4, "d": "32\.5gb"}'.*" - using regular expression ... OK 2017-05-17 13:44:26,554 INFO centosvm log match: searching for "E;8.centosvm;.*resources_used.stra="glad\,elated"\,"happy".*" - using regular expression ... OK 2017-05-17 13:44:26,555 INFO server centosvm: expect offset set to 35 2017-05-17 13:45:01,588 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 9.centosvm 2017-05-17 13:45:01,630 INFO expect on server centosvm: resources_used.cput = 00:00:36 || resources_used.foo_i = 36 || resources_used.ncpus = 3 || job_state = F || resources_used.foo_f = 0.36 || resources_used.vmem = 36gb || resources_used.stra = "glad,elated","happy" job 9.centosvm ... OK 2017-05-17 13:45:01,631 INFO status on centosvm: job 9.centosvm resources_used.foo_str 2017-05-17 13:45:01,632 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 9.centosvm 2017-05-17 13:45:01,670 INFO status on centosvm: job 9.centosvm resources_used.foo_str3 2017-05-17 13:45:01,671 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 9.centosvm 2017-05-17 13:45:01,711 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 9.centosvm 2017-05-17 13:45:01,749 INFO expect on server centosvm: resources_used.foo_str2 unset job 9.centosvm ... OK 2017-05-17 13:45:01,780 INFO centosvm log match: searching for "E;9.centosvm;.*resources_used.foo_f=0.36.*" - using regular expression ... OK 2017-05-17 13:45:01,800 INFO centosvm log match: searching for "E;9.centosvm;.*resources_used.foo_i=36.*" - using regular expression ... OK 2017-05-17 13:45:01,821 INFO centosvm log match: searching for "E;9.centosvm;.*resources_used.foo_str='{"nine": 9, "seven": 7, "eight": 8}'.*" - using regular expression ... OK 2017-05-17 13:45:01,842 INFO centosvm log match: searching for "E;9.centosvm;.*resources_used.vmem=36gb.*" - using regular expression ... OK 2017-05-17 13:45:01,856 INFO centosvm log match: searching for "E;9.centosvm;.*resources_used.cput=00:00:36.*" - using regular expression ... OK 2017-05-17 13:45:01,870 INFO centosvm log match: searching for "E;9.centosvm;.*('resources_used.foo_str2=',).*" - using regular expression - No match 2017-05-17 13:45:02,904 INFO centosvm log match: searching for "E;9.centosvm;.*resources_used.ncpus=3.*" - using regular expression ... OK 2017-05-17 13:45:02,926 INFO centosvm log match: searching for "E;9.centosvm;.*resources_used.foo_str3='{"a": 6, "c": 54\.4, "b": "some value \#\$\%\^\&\*\@", "vn3": 6, "vn2": 5, "vn1": 4, "d": "32\.5gb"}'.*" - using regular expression ... OK 2017-05-17 13:45:02,947 INFO centosvm log match: searching for "E;9.centosvm;.*resources_used.stra="glad\,elated"\,"happy".*" - using regular expression ... OK 2017-05-17 13:45:02,948 INFO =========================================== 2017-05-17 13:45:02,948 INFO Entered TestPbsAccumulateRescUsed tearDown 2017-05-17 13:45:02,948 INFO =========================================== 2017-05-17 13:45:02,948 INFO ============================================ 2017-05-17 13:45:02,948 INFO Completed TestPbsAccumulateRescUsed tearDown 2017-05-17 13:45:02,948 INFO ============================================ 2017-05-17 13:45:02,949 INFO ok 2017-05-17 13:45:02,950 INFO test name: test_prologue (tests.functional.pbs_accumulate_resc_used.TestPbsAccumulateRescUsed)... 2017-05-17 13:45:02,950 INFO test start time: Wed May 17 13:45:02 2017 2017-05-17 13:45:02,950 INFO test docstring: Test accumulatinon of resources of a multinode job from an exechost_prologue hook. 2017-05-17 13:45:02,950 INFO ======================================== 2017-05-17 13:45:02,950 INFO Entered TestPbsAccumulateRescUsed setUp 2017-05-17 13:45:02,951 INFO ======================================== 2017-05-17 13:45:02,951 INFOCLI centosvm: /opt/pbs/bin/qstat -Bf centosvm 2017-05-17 13:45:02,966 INFO status on centosvm: server 2017-05-17 13:45:02,966 INFOCLI centosvm: /opt/pbs/bin/qstat -Bf centosvm 2017-05-17 13:45:02,978 INFO manager on centosvm: set server {'managers': (3, 'dilipk@*')} 2017-05-17 13:45:02,978 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c set server managers-=dilipk@* 2017-05-17 13:45:03,007 INFO manager on centosvm: set server {'managers': (2, 'dilipk@*')} 2017-05-17 13:45:03,008 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c set server managers+=dilipk@* 2017-05-17 13:45:03,032 INFO server centosvm: reverting configuration to defaults 2017-05-17 13:45:03,032 INFOCLI centosvm: /opt/pbs/bin/qstat -Bf centosvm 2017-05-17 13:45:03,049 INFO select on centosvm: __ALL__ 2017-05-17 13:45:03,049 INFOCLI centosvm: /opt/pbs/bin/qselect -x 2017-05-17 13:45:03,062 INFO delete job on centosvm: 8.centosvm, 9.centosvm 2017-05-17 13:45:03,062 INFOCLI centosvm: /opt/pbs/bin/qdel -W force -x 8.centosvm 9.centosvm 2017-05-17 13:45:03,085 INFOCLI centosvm: /opt/pbs/bin/qstat -f 8.centosvm 2017-05-17 13:45:03,102 INFOCLI centosvm: /opt/pbs/bin/qstat -f 9.centosvm 2017-05-17 13:45:03,120 INFOCLI centosvm: /opt/pbs/bin/qstat -f @centosvm 2017-05-17 13:45:03,131 INFO expect on server centosvm: job_state set 0 job ... OK 2017-05-17 13:45:03,131 INFOCLI centosvm: /opt/pbs/bin/pbs_rstat -f 2017-05-17 13:45:03,142 INFO manager on centosvm: unset server ['job_history_enable'] 2017-05-17 13:45:03,142 INFOCLI centosvm: /opt/pbs/bin/qmgr -c unset server job_history_enable 2017-05-17 13:45:03,157 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-05-17 13:45:03,177 INFO manager on centosvm: delete hook start,period 2017-05-17 13:45:03,177 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c delete hook start,period 2017-05-17 13:45:03,201 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook start 2017-05-17 13:45:03,221 INFO expect on server centosvm: unset hook start ... OK 2017-05-17 13:45:03,221 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook period 2017-05-17 13:45:03,238 INFO expect on server centosvm: unset hook period ... OK 2017-05-17 13:45:03,239 INFOCLI centosvm: /opt/pbs/bin/qstat -Qf @centosvm 2017-05-17 13:45:03,249 INFO manager on centosvm: delete queue workq 2017-05-17 13:45:03,250 INFOCLI centosvm: /opt/pbs/bin/qmgr -c delete queue workq 2017-05-17 13:45:03,262 INFOCLI centosvm: /opt/pbs/bin/qstat -Qf workq@centosvm 2017-05-17 13:45:03,272 INFO expect on server centosvm: unset queue workq ... OK 2017-05-17 13:45:03,272 INFO manager on centosvm: create queue workq {'started': 'True', 'queue_type': 'Execution', 'enabled': 'True'} 2017-05-17 13:45:03,272 INFOCLI centosvm: /opt/pbs/bin/qmgr -c create queue workq started=True,queue_type=Execution,enabled=True 2017-05-17 13:45:03,286 INFOCLI centosvm: /opt/pbs/bin/qstat -Qf workq@centosvm 2017-05-17 13:45:03,297 INFO expect on server centosvm: started set True || queue_type set Execution || enabled set True queue workq ... OK 2017-05-17 13:45:03,298 INFO manager on centosvm: set server {'default_queue': 'workq'} 2017-05-17 13:45:03,298 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server default_queue=workq 2017-05-17 13:45:03,310 INFO status on centosvm: resource 2017-05-17 13:45:03,310 INFO manager on centosvm: list resource 2017-05-17 13:45:03,310 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource 2017-05-17 13:45:03,324 INFO manager on centosvm: delete resource foo_f,foo_i,foo_str2,foo_str3,foo_str4,foo_str,stra 2017-05-17 13:45:03,325 INFOCLI centosvm: /opt/pbs/bin/qmgr -c delete resource foo_f,foo_i,foo_str2,foo_str3,foo_str4,foo_str,stra 2017-05-17 13:45:03,641 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_f 2017-05-17 13:45:03,683 INFO expect on server centosvm: unset resource foo_f ... OK 2017-05-17 13:45:03,684 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_i 2017-05-17 13:45:03,706 INFO expect on server centosvm: unset resource foo_i ... OK 2017-05-17 13:45:03,706 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_str2 2017-05-17 13:45:03,727 INFO expect on server centosvm: unset resource foo_str2 ... OK 2017-05-17 13:45:03,728 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_str3 2017-05-17 13:45:03,748 INFO expect on server centosvm: unset resource foo_str3 ... OK 2017-05-17 13:45:03,748 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_str4 2017-05-17 13:45:03,762 INFO expect on server centosvm: unset resource foo_str4 ... OK 2017-05-17 13:45:03,762 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_str 2017-05-17 13:45:03,774 INFO expect on server centosvm: unset resource foo_str ... OK 2017-05-17 13:45:03,775 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource stra 2017-05-17 13:45:03,791 INFO expect on server centosvm: unset resource stra ... OK 2017-05-17 13:45:03,791 INFOCLI status on centosvm: server license_count 2017-05-17 13:45:03,791 INFOCLI centosvm: /opt/pbs/bin/qstat -Bf centosvm 2017-05-17 13:45:03,803 INFO server: centosvm licensed 2017-05-17 13:45:03,819 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/server_priv/comm.lock 2017-05-17 13:45:03,849 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-05-17 13:45:03,862 INFO scheduler centosvm: reverting configuration to defaults 2017-05-17 13:45:03,862 INFO manager on centosvm: list sched 2017-05-17 13:45:03,862 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list sched 2017-05-17 13:45:03,874 INFO manager on centosvm: unset sched ['sched_cycle_length'] 2017-05-17 13:45:03,875 INFOCLI centosvm: /opt/pbs/bin/qmgr -c unset sched sched_cycle_length 2017-05-17 13:45:03,890 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/dedicated_time 2017-05-17 13:45:03,901 INFOCLI2 centosvm: cmp /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-05-17 13:45:03,904 ERROR err: ['cmp: /var/spool/pbs/sched_priv/resource_group: Permission denied'] 2017-05-17 13:45:03,905 INFOCLI2 centosvm: sudo -H /usr/bin/cp -p /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-05-17 13:45:03,919 INFOCLI2 centosvm: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-05-17 13:45:03,930 INFO scheduler centosvm: reverting holidays file to default 2017-05-17 13:45:03,931 INFOCLI2 centosvm: cmp /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-05-17 13:45:03,933 ERROR err: ['cmp: /var/spool/pbs/sched_priv/holidays: Permission denied'] 2017-05-17 13:45:03,934 INFOCLI2 centosvm: sudo -H /usr/bin/cp -p /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-05-17 13:45:03,951 INFOCLI2 centosvm: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/holidays 2017-05-17 13:45:03,963 INFO scheduler centosvm: Parsing holidays file 2017-05-17 13:45:03,963 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2017-05-17 13:45:03,975 INFOCLI2 centosvm: cmp /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-05-17 13:45:03,978 ERROR err: ['cmp: /var/spool/pbs/sched_priv/sched_config: Permission denied'] 2017-05-17 13:45:03,978 INFOCLI2 centosvm: sudo -H /usr/bin/cp -p /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-05-17 13:45:03,994 INFOCLI2 centosvm: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-05-17 13:45:04,006 INFO scheduler centosvm: sent signal -HUP 2017-05-17 13:45:04,007 INFOCLI2 centosvm: sudo -H kill -HUP 84281 2017-05-17 13:45:04,020 INFOCLI2 centosvm: sudo -H /usr/bin/rm /var/spool/pbs/sched_priv/usage 2017-05-17 13:45:04,035 ERROR err: ['/usr/bin/rm: cannot remove \xe2\x80\x98/var/spool/pbs/sched_priv/usage\xe2\x80\x99: No such file or directory'] 2017-05-17 13:45:04,069 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-05-17 13:45:04,083 INFOCLI2 centosvm: sudo -H cmp /opt/pbs/sbin/pbs_mom /opt/pbs/sbin/pbs_mom.cpuset 2017-05-17 13:45:04,107 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:45:04,117 INFO mom centosvm: reverting configuration to defaults 2017-05-17 13:45:04,117 INFOCLI2 centosvm: sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2017-05-17 13:45:04,128 INFOCLI2 centosvm: sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2017-05-17 13:45:04,140 INFOCLI2 centosvm: sudo -H ls /var/spool/pbs/mom_priv/config.d 2017-05-17 13:45:04,151 INFO mom centosvm: sent signal -HUP 2017-05-17 13:45:04,151 INFOCLI2 centosvm: sudo -H kill -HUP 84278 2017-05-17 13:45:04,180 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:45:04,192 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v -a 2017-05-17 13:45:04,203 INFO status on centosvm: node centosvm 2017-05-17 13:45:04,203 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:45:04,214 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:45:04,230 INFO expect on server centosvm: state = free node centosvm ... OK 2017-05-17 13:45:04,231 INFOCLI2 centosvm2: ssh centosvm2 sudo -H cmp /opt/pbs/sbin/pbs_mom /opt/pbs/sbin/pbs_mom.cpuset 2017-05-17 13:45:05,290 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:45:05,701 INFO mom centosvm2: reverting configuration to defaults 2017-05-17 13:45:05,702 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2017-05-17 13:45:06,208 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2017-05-17 13:45:06,713 INFOCLI2 centosvm2: ssh centosvm2 sudo -H ls /var/spool/pbs/mom_priv/config.d 2017-05-17 13:45:07,096 INFO mom centosvm2: sent signal -HUP 2017-05-17 13:45:07,097 INFOCLI2 centosvm2: ssh centosvm2 sudo -H kill -HUP 15795 2017-05-17 13:45:08,112 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:45:08,526 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v -a 2017-05-17 13:45:08,537 INFO status on centosvm: node centosvm2 2017-05-17 13:45:08,538 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm2 2017-05-17 13:45:08,551 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm2 2017-05-17 13:45:08,565 INFO expect on server centosvm: state = free node centosvm2 ... OK 2017-05-17 13:45:08,565 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H cmp /opt/pbs/sbin/pbs_mom /opt/pbs/sbin/pbs_mom.cpuset 2017-05-17 13:45:09,535 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:45:09,993 INFO mom centosvm_nfs: reverting configuration to defaults 2017-05-17 13:45:09,994 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2017-05-17 13:45:10,432 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2017-05-17 13:45:11,037 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H ls /var/spool/pbs/mom_priv/config.d 2017-05-17 13:45:11,607 INFO mom centosvm_nfs: sent signal -HUP 2017-05-17 13:45:11,608 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H kill -HUP 76746 2017-05-17 13:45:12,775 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:45:13,353 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v -a 2017-05-17 13:45:13,365 INFO status on centosvm: node centosvm_nfs 2017-05-17 13:45:13,366 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm_nfs 2017-05-17 13:45:13,377 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm_nfs 2017-05-17 13:45:13,393 INFO expect on server centosvm: state = free node centosvm_nfs ... OK 2017-05-17 13:45:13,393 INFO ========================================= 2017-05-17 13:45:13,393 INFO Completed TestPbsAccumulateRescUsed setUp 2017-05-17 13:45:13,394 INFO ========================================= 2017-05-17 13:45:13,394 INFO len moms = 3 2017-05-17 13:45:13,394 INFOCLI2 centosvm: sudo -H /usr/bin/rm -rf /var/spool/pbs/mom_priv/config.d 2017-05-17 13:45:13,410 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/rm -rf /var/spool/pbs/mom_priv/config.d 2017-05-17 13:45:13,887 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H /usr/bin/rm -rf /var/spool/pbs/mom_priv/config.d 2017-05-17 13:45:14,364 INFO manager on centosvm: delete node 2017-05-17 13:45:14,365 INFOCLI centosvm: /opt/pbs/bin/qmgr -c delete node @default 2017-05-17 13:45:14,393 INFO manager on centosvm: create node centosvm 2017-05-17 13:45:14,393 INFOCLI centosvm: /opt/pbs/bin/qmgr -c create node centosvm 2017-05-17 13:45:14,421 INFO manager on centosvm: create node centosvm2 2017-05-17 13:45:14,421 INFOCLI centosvm: /opt/pbs/bin/qmgr -c create node centosvm2 2017-05-17 13:45:14,439 INFO manager on centosvm: create node centosvm_nfs 2017-05-17 13:45:14,440 INFOCLI centosvm: /opt/pbs/bin/qmgr -c create node centosvm_nfs 2017-05-17 13:45:14,468 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:45:14,483 INFO expect on server centosvm: state = free node centosvm got: state = state-unknown,down 2017-05-17 13:45:14,985 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:45:15,014 INFO expect on server centosvm: state = free node centosvm attempt: 2 got: state = state-unknown,down 2017-05-17 13:45:15,515 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:45:15,556 INFO expect on server centosvm: state = free node centosvm attempt: 3 got: state = state-unknown,down 2017-05-17 13:45:16,058 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:45:16,100 INFO expect on server centosvm: state = free node centosvm attempt: 4 ... OK 2017-05-17 13:45:16,101 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm2 2017-05-17 13:45:16,141 INFO expect on server centosvm: state = free node centosvm2 ... OK 2017-05-17 13:45:16,142 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm_nfs 2017-05-17 13:45:16,164 INFO expect on server centosvm: state = free node centosvm_nfs ... OK 2017-05-17 13:45:16,164 INFO status on centosvm: hook 2017-05-17 13:45:16,164 INFO manager on centosvm: list hook 2017-05-17 13:45:16,165 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-05-17 13:45:16,210 INFO status on centosvm: hook 2017-05-17 13:45:16,211 INFO manager on centosvm: list hook 2017-05-17 13:45:16,211 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-05-17 13:45:16,249 INFO manager on centosvm: create hook start 2017-05-17 13:45:16,250 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c create hook start 2017-05-17 13:45:16,286 INFO manager on centosvm: set hook start {'enabled': 'True', 'event': 'exechost_startup'} 2017-05-17 13:45:16,286 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c set hook start enabled=True,event=exechost_startup 2017-05-17 13:45:16,317 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook start 2017-05-17 13:45:16,341 INFO expect on server centosvm: enabled set True || event set exechost_startup hook start ... OK 2017-05-17 13:45:16,342 INFO manager on centosvm: import hook start {'content-type': 'application/x-python', 'input-file': '/tmp/PtlPbsAqesKA', 'content-encoding': 'default'} 2017-05-17 13:45:16,342 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c import hook start application/x-python default /tmp/PtlPbsAqesKA 2017-05-17 13:45:16,368 INFO server centosvm: imported hook body --- import pbs e=pbs.event() localnode=pbs.get_local_nodename() e.vnode_list[localnode].resources_available['foo_i'] = 7 e.vnode_list[localnode].resources_available['foo_f'] = 5.0 e.vnode_list[localnode].resources_available['foo_str'] = "seventyseven" --- 2017-05-17 13:45:16,368 INFO mom centosvm: sent signal -HUP 2017-05-17 13:45:16,368 INFOCLI2 centosvm: sudo -H kill -HUP 84278 2017-05-17 13:45:16,381 INFO mom centosvm2: sent signal -HUP 2017-05-17 13:45:16,381 INFOCLI2 centosvm2: ssh centosvm2 sudo -H kill -HUP 15795 2017-05-17 13:45:16,746 INFO mom centosvm_nfs: sent signal -HUP 2017-05-17 13:45:16,746 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H kill -HUP 76746 2017-05-17 13:45:17,114 INFO manager on centosvm: set server {'job_history_enable': 'True'} 2017-05-17 13:45:17,114 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server job_history_enable=True 2017-05-17 13:45:17,130 INFO manager on centosvm as root: create resource foo_str2 {'flag': 'h', 'type': 'string'} 2017-05-17 13:45:17,130 INFOCLI centosvm: sudo -H -u root /opt/pbs/bin/qmgr -c create resource foo_str2 flag=h,type=string 2017-05-17 13:45:17,168 INFO mom centosvm log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:45:17,513 INFO mom centosvm2 log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:45:17,937 INFO mom centosvm_nfs log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:45:17,938 INFO manager on centosvm as root: create resource foo_str3 {'flag': 'h', 'type': 'string'} 2017-05-17 13:45:17,938 INFOCLI centosvm: sudo -H -u root /opt/pbs/bin/qmgr -c create resource foo_str3 flag=h,type=string 2017-05-17 13:45:18,016 INFO mom centosvm log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:45:18,427 INFO mom centosvm2 log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:45:18,953 INFO mom centosvm_nfs log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:45:18,954 INFO manager on centosvm as root: create resource foo_str4 {'flag': 'h', 'type': 'string'} 2017-05-17 13:45:18,954 INFOCLI centosvm: sudo -H -u root /opt/pbs/bin/qmgr -c create resource foo_str4 flag=h,type=string 2017-05-17 13:45:19,017 INFO mom centosvm log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:45:19,453 INFO mom centosvm2 log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:45:19,817 INFO mom centosvm_nfs log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:45:19,818 INFO manager on centosvm as root: create resource stra {'flag': 'h', 'type': 'string_array'} 2017-05-17 13:45:19,819 INFOCLI centosvm: sudo -H -u root /opt/pbs/bin/qmgr -c create resource stra flag=h,type=string_array 2017-05-17 13:45:19,889 INFO mom centosvm log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:45:20,320 INFO mom centosvm2 log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:45:20,772 INFO mom centosvm_nfs log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:45:20,772 INFO test_prologue 2017-05-17 13:45:20,773 INFO status on centosvm: hook 2017-05-17 13:45:20,773 INFO manager on centosvm: list hook 2017-05-17 13:45:20,774 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-05-17 13:45:20,844 INFO status on centosvm: hook 2017-05-17 13:45:20,845 INFO manager on centosvm: list hook 2017-05-17 13:45:20,845 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-05-17 13:45:20,877 INFO manager on centosvm: create hook prolo 2017-05-17 13:45:20,877 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c create hook prolo 2017-05-17 13:45:20,902 INFO manager on centosvm: set hook prolo {'enabled': 'True', 'event': 'execjob_prologue'} 2017-05-17 13:45:20,902 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c set hook prolo enabled=True,event=execjob_prologue 2017-05-17 13:45:20,927 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook prolo 2017-05-17 13:45:20,948 INFO expect on server centosvm: enabled set True || event set execjob_prologue hook prolo ... OK 2017-05-17 13:45:20,949 INFO manager on centosvm: import hook prolo {'content-type': 'application/x-python', 'input-file': '/tmp/PtlPbs1dXWNi', 'content-encoding': 'default'} 2017-05-17 13:45:20,949 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c import hook prolo application/x-python default /tmp/PtlPbs1dXWNi 2017-05-17 13:45:20,975 INFO server centosvm: imported hook body --- import pbs e=pbs.event() pbs.logmsg(pbs.LOG_DEBUG, "executed prologue hook") if e.job.in_ms_mom(): e.job.resources_used["vmem"] = pbs.size("11gb") e.job.resources_used["foo_i"] = 11 e.job.resources_used["foo_f"] = 0.11 e.job.resources_used["foo_str"] = '{"seven":7}' e.job.resources_used["cput"] = 11 e.job.resources_used["stra"] = '"glad,elated","happy"' e.job.resources_used["foo_str3"] = """{"a":6,"b":"some value #$%^&*@","c":54.4,"d":"32.5gb"}""" e.job.resources_used["foo_str2"] = "seven" e.job.resources_used["foo_str4"] = "eight" else: e.job.resources_used["vmem"] = pbs.size("12gb") e.job.resources_used["foo_i"] = 12 e.job.resources_used["foo_f"] = 0.12 e.job.resources_used["foo_str"] = '{"eight":8,"nine":9}' e.job.resources_used["foo_str2"] = '{"seven":7}' e.job.resources_used["cput"] = 12 e.job.resources_used["stra"] = '"cucumbers,bananas"' e.job.resources_used["foo_str3"] = """"vn1":4,"vn2":5,"vn3":6""" --- 2017-05-17 13:45:20,975 INFO job: executable set to /bin/sleep with arguments: 100 2017-05-17 13:45:20,976 INFOCLI2 centosvm: sudo -H /usr/bin/chmod 0755 /tmp/PtlPbsJobScriptVeGm6i 2017-05-17 13:45:20,986 INFOCLI centosvm: sudo -H -u pbsuser /opt/pbs/bin/qsub -l place=scatter -l walltime=10 -l select=3:ncpus=1 /tmp/PtlPbsJobScriptVeGm6i 2017-05-17 13:45:21,009 INFO submit to centosvm as pbsuser: job 10.centosvm OrderedDict([('Resource_List.place', 'scatter'), ('Resource_List.walltime', 10), ('Resource_List.select', '3:ncpus=1')]) 2017-05-17 13:45:21,009 INFOCLI job script /tmp/PtlPbsJobScriptVeGm6i --- pbsdsh -n 1 hostname pbsdsh -n 2 hostname sleep 10 --- 2017-05-17 13:45:21,009 INFO server centosvm: expect offset set to 10 2017-05-17 13:45:31,011 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 10.centosvm 2017-05-17 13:45:31,058 INFO expect on server centosvm: resources_used.stra = "glad,elated","happy" || resources_used.cput = 00:00:35 || resources_used.foo_str4 = eight || resources_used.foo_i = 35 || resources_used.ncpus = 3 || job_state = F || resources_used.foo_f = 0.35 || resources_used.vmem = 35gb job 10.centosvm got: resources_used.cput = 11 2017-05-17 13:45:31,560 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:45:31,608 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 10.centosvm 2017-05-17 13:45:31,648 INFO expect on server centosvm: resources_used.stra = "glad,elated","happy" || resources_used.cput = 00:00:35 || resources_used.foo_str4 = eight || resources_used.foo_i = 35 || resources_used.ncpus = 3 || job_state = F || resources_used.foo_f = 0.35 || resources_used.vmem = 35gb job 10.centosvm attempt: 2 got: resources_used.cput = 11 2017-05-17 13:45:32,153 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:45:32,202 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 10.centosvm 2017-05-17 13:45:32,243 INFO expect on server centosvm: resources_used.stra = "glad,elated","happy" || resources_used.cput = 00:00:35 || resources_used.foo_str4 = eight || resources_used.foo_i = 35 || resources_used.ncpus = 3 || job_state = F || resources_used.foo_f = 0.35 || resources_used.vmem = 35gb job 10.centosvm attempt: 3 got: resources_used.cput = 11 2017-05-17 13:45:32,745 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:45:32,802 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 10.centosvm 2017-05-17 13:45:32,835 INFO expect on server centosvm: resources_used.stra = "glad,elated","happy" || resources_used.cput = 00:00:35 || resources_used.foo_str4 = eight || resources_used.foo_i = 35 || resources_used.ncpus = 3 || job_state = F || resources_used.foo_f = 0.35 || resources_used.vmem = 35gb job 10.centosvm attempt: 4 ... OK 2017-05-17 13:45:32,836 INFO status on centosvm: job 10.centosvm resources_used.foo_str 2017-05-17 13:45:32,837 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 10.centosvm 2017-05-17 13:45:32,869 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 10.centosvm 2017-05-17 13:45:32,902 INFO expect on server centosvm: resources_used.foo_str3 unset job 10.centosvm ... OK 2017-05-17 13:45:32,903 INFO mom centosvm log match: searching for "Job 10.centosvm resources_used.foo_str3 cannot be accumulated: value '"vn1":4,"vn2":5,"vn3":6' from mom centosvm2 not JSON-format"... OK 2017-05-17 13:45:32,904 INFO mom centosvm log match: searching for "Job 10.centosvm resources_used.foo_str3 cannot be accumulated: value '"vn1":4,"vn2":5,"vn3":6' from mom centosvm_nfs not JSON-format" - No match 2017-05-17 13:45:33,906 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 10.centosvm 2017-05-17 13:45:33,933 INFO expect on server centosvm: resources_used.foo_str3 unset job 10.centosvm ... OK 2017-05-17 13:45:33,934 INFOCLI centosvm: /opt/pbs/bin/qstat -f 10.centosvm 2017-05-17 13:45:33,958 INFO expect on server centosvm: resources_used.foo_str2 unset job 10.centosvm ... OK 2017-05-17 13:45:33,959 INFO mom centosvm log match: searching for "Job 10.centosvm resources_used.foo_str2 cannot be accumulated: value 'seven' from mom centosvm not JSON-format"... OK 2017-05-17 13:45:33,984 INFO centosvm log match: searching for "E;10.centosvm;.*resources_used.foo_f=0.35.*" - using regular expression ... OK 2017-05-17 13:45:34,006 INFO centosvm log match: searching for "E;10.centosvm;.*resources_used.foo_i=35.*" - using regular expression ... OK 2017-05-17 13:45:34,019 INFO centosvm log match: searching for "E;10.centosvm;.*resources_used.foo_str='{"eight": 8, "seven": 7, "nine": 9}'.*" - using regular expression ... OK 2017-05-17 13:45:34,030 INFO centosvm log match: searching for "E;10.centosvm;.*resources_used.vmem=35gb.*" - using regular expression ... OK 2017-05-17 13:45:34,041 INFO centosvm log match: searching for "E;10.centosvm;.*resources_used.cput=00:00:35.*" - using regular expression ... OK 2017-05-17 13:45:34,058 INFO centosvm log match: searching for "E;10.centosvm;.*resources_used.foo_str2=.*" - using regular expression - No match 2017-05-17 13:45:35,080 INFO centosvm log match: searching for "E;10.centosvm;.*resources_used.ncpus=3.*" - using regular expression ... OK 2017-05-17 13:45:35,102 INFO centosvm log match: searching for "E;10.centosvm;.*resources_used.foo_str3=.*" - using regular expression - No match 2017-05-17 13:45:36,144 INFO centosvm log match: searching for "E;10.centosvm;.*resources_used.foo_str4=eight.*" - using regular expression ... OK 2017-05-17 13:45:36,186 INFO centosvm log match: searching for "E;10.centosvm;.*resources_used.stra="glad\,elated"\,"happy".*" - using regular expression ... OK 2017-05-17 13:45:36,187 INFO =========================================== 2017-05-17 13:45:36,188 INFO Entered TestPbsAccumulateRescUsed tearDown 2017-05-17 13:45:36,188 INFO =========================================== 2017-05-17 13:45:36,188 INFO ============================================ 2017-05-17 13:45:36,189 INFO Completed TestPbsAccumulateRescUsed tearDown 2017-05-17 13:45:36,189 INFO ============================================ 2017-05-17 13:45:36,190 INFO ok 2017-05-17 13:45:36,191 INFO test name: test_reservation (tests.functional.pbs_accumulate_resc_used.TestPbsAccumulateRescUsed)... 2017-05-17 13:45:36,191 INFO test start time: Wed May 17 13:45:36 2017 2017-05-17 13:45:36,191 INFO test docstring: Test that job inside reservations works same 2017-05-17 13:45:36,192 INFO ======================================== 2017-05-17 13:45:36,192 INFO Entered TestPbsAccumulateRescUsed setUp 2017-05-17 13:45:36,192 INFO ======================================== 2017-05-17 13:45:36,193 INFOCLI centosvm: /opt/pbs/bin/qstat -Bf centosvm 2017-05-17 13:45:36,216 INFO status on centosvm: server 2017-05-17 13:45:36,216 INFOCLI centosvm: /opt/pbs/bin/qstat -Bf centosvm 2017-05-17 13:45:36,238 INFO manager on centosvm: set server {'managers': (3, 'dilipk@*')} 2017-05-17 13:45:36,239 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c set server managers-=dilipk@* 2017-05-17 13:45:36,283 INFO manager on centosvm: set server {'managers': (2, 'dilipk@*')} 2017-05-17 13:45:36,284 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c set server managers+=dilipk@* 2017-05-17 13:45:36,325 INFO server centosvm: reverting configuration to defaults 2017-05-17 13:45:36,326 INFOCLI centosvm: /opt/pbs/bin/qstat -Bf centosvm 2017-05-17 13:45:36,348 INFO select on centosvm: __ALL__ 2017-05-17 13:45:36,349 INFOCLI centosvm: /opt/pbs/bin/qselect -x 2017-05-17 13:45:36,375 INFO delete job on centosvm: 10.centosvm 2017-05-17 13:45:36,375 INFOCLI centosvm: /opt/pbs/bin/qdel -W force -x 10.centosvm 2017-05-17 13:45:36,411 INFOCLI centosvm: /opt/pbs/bin/qstat -f 10.centosvm 2017-05-17 13:45:36,443 INFOCLI centosvm: /opt/pbs/bin/qstat -f @centosvm 2017-05-17 13:45:36,459 INFO expect on server centosvm: job_state set 0 job ... OK 2017-05-17 13:45:36,460 INFOCLI centosvm: /opt/pbs/bin/pbs_rstat -f 2017-05-17 13:45:36,472 INFO manager on centosvm: unset server ['job_history_enable'] 2017-05-17 13:45:36,473 INFOCLI centosvm: /opt/pbs/bin/qmgr -c unset server job_history_enable 2017-05-17 13:45:36,492 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-05-17 13:45:36,512 INFO manager on centosvm: delete hook start,prolo 2017-05-17 13:45:36,512 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c delete hook start,prolo 2017-05-17 13:45:36,542 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook start 2017-05-17 13:45:36,561 INFO expect on server centosvm: unset hook start ... OK 2017-05-17 13:45:36,562 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook prolo 2017-05-17 13:45:36,585 INFO expect on server centosvm: unset hook prolo ... OK 2017-05-17 13:45:36,586 INFOCLI centosvm: /opt/pbs/bin/qstat -Qf @centosvm 2017-05-17 13:45:36,596 INFO manager on centosvm: delete queue workq 2017-05-17 13:45:36,596 INFOCLI centosvm: /opt/pbs/bin/qmgr -c delete queue workq 2017-05-17 13:45:36,611 INFOCLI centosvm: /opt/pbs/bin/qstat -Qf workq@centosvm 2017-05-17 13:45:36,621 INFO expect on server centosvm: unset queue workq ... OK 2017-05-17 13:45:36,622 INFO manager on centosvm: create queue workq {'started': 'True', 'queue_type': 'Execution', 'enabled': 'True'} 2017-05-17 13:45:36,622 INFOCLI centosvm: /opt/pbs/bin/qmgr -c create queue workq started=True,queue_type=Execution,enabled=True 2017-05-17 13:45:36,635 INFOCLI centosvm: /opt/pbs/bin/qstat -Qf workq@centosvm 2017-05-17 13:45:36,650 INFO expect on server centosvm: started set True || queue_type set Execution || enabled set True queue workq ... OK 2017-05-17 13:45:36,650 INFO manager on centosvm: set server {'default_queue': 'workq'} 2017-05-17 13:45:36,650 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server default_queue=workq 2017-05-17 13:45:36,663 INFO status on centosvm: resource 2017-05-17 13:45:36,663 INFO manager on centosvm: list resource 2017-05-17 13:45:36,663 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource 2017-05-17 13:45:36,675 INFO manager on centosvm: delete resource foo_f,foo_i,foo_str2,foo_str3,foo_str4,foo_str,stra 2017-05-17 13:45:36,675 INFOCLI centosvm: /opt/pbs/bin/qmgr -c delete resource foo_f,foo_i,foo_str2,foo_str3,foo_str4,foo_str,stra 2017-05-17 13:45:37,058 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_f 2017-05-17 13:45:37,094 INFO expect on server centosvm: unset resource foo_f ... OK 2017-05-17 13:45:37,095 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_i 2017-05-17 13:45:37,121 INFO expect on server centosvm: unset resource foo_i ... OK 2017-05-17 13:45:37,122 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_str2 2017-05-17 13:45:37,149 INFO expect on server centosvm: unset resource foo_str2 ... OK 2017-05-17 13:45:37,150 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_str3 2017-05-17 13:45:37,167 INFO expect on server centosvm: unset resource foo_str3 ... OK 2017-05-17 13:45:37,168 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_str4 2017-05-17 13:45:37,182 INFO expect on server centosvm: unset resource foo_str4 ... OK 2017-05-17 13:45:37,182 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_str 2017-05-17 13:45:37,193 INFO expect on server centosvm: unset resource foo_str ... OK 2017-05-17 13:45:37,193 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource stra 2017-05-17 13:45:37,207 INFO expect on server centosvm: unset resource stra ... OK 2017-05-17 13:45:37,208 INFOCLI status on centosvm: server license_count 2017-05-17 13:45:37,208 INFOCLI centosvm: /opt/pbs/bin/qstat -Bf centosvm 2017-05-17 13:45:37,218 INFO server: centosvm licensed 2017-05-17 13:45:37,231 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/server_priv/comm.lock 2017-05-17 13:45:37,257 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-05-17 13:45:37,268 INFO scheduler centosvm: reverting configuration to defaults 2017-05-17 13:45:37,268 INFO manager on centosvm: list sched 2017-05-17 13:45:37,268 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list sched 2017-05-17 13:45:37,282 INFO manager on centosvm: unset sched ['sched_cycle_length'] 2017-05-17 13:45:37,282 INFOCLI centosvm: /opt/pbs/bin/qmgr -c unset sched sched_cycle_length 2017-05-17 13:45:37,295 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/dedicated_time 2017-05-17 13:45:37,304 INFOCLI2 centosvm: cmp /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-05-17 13:45:37,307 ERROR err: ['cmp: /var/spool/pbs/sched_priv/resource_group: Permission denied'] 2017-05-17 13:45:37,307 INFOCLI2 centosvm: sudo -H /usr/bin/cp -p /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-05-17 13:45:37,318 INFOCLI2 centosvm: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-05-17 13:45:37,329 INFO scheduler centosvm: reverting holidays file to default 2017-05-17 13:45:37,330 INFOCLI2 centosvm: cmp /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-05-17 13:45:37,333 ERROR err: ['cmp: /var/spool/pbs/sched_priv/holidays: Permission denied'] 2017-05-17 13:45:37,333 INFOCLI2 centosvm: sudo -H /usr/bin/cp -p /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-05-17 13:45:37,345 INFOCLI2 centosvm: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/holidays 2017-05-17 13:45:37,358 INFO scheduler centosvm: Parsing holidays file 2017-05-17 13:45:37,358 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2017-05-17 13:45:37,369 INFOCLI2 centosvm: cmp /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-05-17 13:45:37,372 ERROR err: ['cmp: /var/spool/pbs/sched_priv/sched_config: Permission denied'] 2017-05-17 13:45:37,372 INFOCLI2 centosvm: sudo -H /usr/bin/cp -p /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-05-17 13:45:37,383 INFOCLI2 centosvm: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-05-17 13:45:37,393 INFO scheduler centosvm: sent signal -HUP 2017-05-17 13:45:37,394 INFOCLI2 centosvm: sudo -H kill -HUP 84281 2017-05-17 13:45:37,405 INFOCLI2 centosvm: sudo -H /usr/bin/rm /var/spool/pbs/sched_priv/usage 2017-05-17 13:45:37,416 ERROR err: ['/usr/bin/rm: cannot remove \xe2\x80\x98/var/spool/pbs/sched_priv/usage\xe2\x80\x99: No such file or directory'] 2017-05-17 13:45:37,441 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-05-17 13:45:37,453 INFOCLI2 centosvm: sudo -H cmp /opt/pbs/sbin/pbs_mom /opt/pbs/sbin/pbs_mom.cpuset 2017-05-17 13:45:37,476 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:45:37,486 INFO mom centosvm: reverting configuration to defaults 2017-05-17 13:45:37,487 INFOCLI2 centosvm: sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2017-05-17 13:45:37,497 INFOCLI2 centosvm: sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2017-05-17 13:45:37,509 INFOCLI2 centosvm: sudo -H ls /var/spool/pbs/mom_priv/config.d 2017-05-17 13:45:37,519 INFO mom centosvm: sent signal -HUP 2017-05-17 13:45:37,519 INFOCLI2 centosvm: sudo -H kill -HUP 84278 2017-05-17 13:45:37,545 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:45:37,560 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v -a 2017-05-17 13:45:37,575 INFO status on centosvm: node centosvm 2017-05-17 13:45:37,575 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:45:37,592 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:45:37,608 INFO expect on server centosvm: state = free node centosvm ... OK 2017-05-17 13:45:37,608 INFOCLI2 centosvm2: ssh centosvm2 sudo -H cmp /opt/pbs/sbin/pbs_mom /opt/pbs/sbin/pbs_mom.cpuset 2017-05-17 13:45:38,827 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:45:39,395 INFO mom centosvm2: reverting configuration to defaults 2017-05-17 13:45:39,396 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2017-05-17 13:45:39,818 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2017-05-17 13:45:40,311 INFOCLI2 centosvm2: ssh centosvm2 sudo -H ls /var/spool/pbs/mom_priv/config.d 2017-05-17 13:45:40,815 INFO mom centosvm2: sent signal -HUP 2017-05-17 13:45:40,816 INFOCLI2 centosvm2: ssh centosvm2 sudo -H kill -HUP 15795 2017-05-17 13:45:41,760 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:45:42,255 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v -a 2017-05-17 13:45:42,290 INFO status on centosvm: node centosvm2 2017-05-17 13:45:42,290 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm2 2017-05-17 13:45:42,314 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm2 2017-05-17 13:45:42,328 INFO expect on server centosvm: state = free node centosvm2 ... OK 2017-05-17 13:45:42,329 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H cmp /opt/pbs/sbin/pbs_mom /opt/pbs/sbin/pbs_mom.cpuset 2017-05-17 13:45:43,164 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:45:43,576 INFO mom centosvm_nfs: reverting configuration to defaults 2017-05-17 13:45:43,577 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2017-05-17 13:45:44,132 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2017-05-17 13:45:44,842 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H ls /var/spool/pbs/mom_priv/config.d 2017-05-17 13:45:45,232 INFO mom centosvm_nfs: sent signal -HUP 2017-05-17 13:45:45,232 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H kill -HUP 76746 2017-05-17 13:45:46,169 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:45:46,665 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v -a 2017-05-17 13:45:46,679 INFO status on centosvm: node centosvm_nfs 2017-05-17 13:45:46,679 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm_nfs 2017-05-17 13:45:46,693 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm_nfs 2017-05-17 13:45:46,708 INFO expect on server centosvm: state = free node centosvm_nfs ... OK 2017-05-17 13:45:46,708 INFO ========================================= 2017-05-17 13:45:46,709 INFO Completed TestPbsAccumulateRescUsed setUp 2017-05-17 13:45:46,709 INFO ========================================= 2017-05-17 13:45:46,709 INFO len moms = 3 2017-05-17 13:45:46,709 INFOCLI2 centosvm: sudo -H /usr/bin/rm -rf /var/spool/pbs/mom_priv/config.d 2017-05-17 13:45:46,722 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/rm -rf /var/spool/pbs/mom_priv/config.d 2017-05-17 13:45:47,099 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H /usr/bin/rm -rf /var/spool/pbs/mom_priv/config.d 2017-05-17 13:45:47,452 INFO manager on centosvm: delete node 2017-05-17 13:45:47,453 INFOCLI centosvm: /opt/pbs/bin/qmgr -c delete node @default 2017-05-17 13:45:47,468 INFO manager on centosvm: create node centosvm 2017-05-17 13:45:47,469 INFOCLI centosvm: /opt/pbs/bin/qmgr -c create node centosvm 2017-05-17 13:45:47,486 INFO manager on centosvm: create node centosvm2 2017-05-17 13:45:47,486 INFOCLI centosvm: /opt/pbs/bin/qmgr -c create node centosvm2 2017-05-17 13:45:47,502 INFO manager on centosvm: create node centosvm_nfs 2017-05-17 13:45:47,502 INFOCLI centosvm: /opt/pbs/bin/qmgr -c create node centosvm_nfs 2017-05-17 13:45:47,517 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:45:47,539 INFO expect on server centosvm: state = free node centosvm got: state = state-unknown,down 2017-05-17 13:45:48,041 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:45:48,085 INFO expect on server centosvm: state = free node centosvm attempt: 2 got: state = state-unknown,down 2017-05-17 13:45:48,586 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:45:48,614 INFO expect on server centosvm: state = free node centosvm attempt: 3 got: state = state-unknown,down 2017-05-17 13:45:49,116 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:45:49,164 INFO expect on server centosvm: state = free node centosvm attempt: 4 ... OK 2017-05-17 13:45:49,165 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm2 2017-05-17 13:45:49,192 INFO expect on server centosvm: state = free node centosvm2 got: state = initializing 2017-05-17 13:45:49,694 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm2 2017-05-17 13:45:49,711 INFO expect on server centosvm: state = free node centosvm2 attempt: 2 ... OK 2017-05-17 13:45:49,712 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm_nfs 2017-05-17 13:45:49,724 INFO expect on server centosvm: state = free node centosvm_nfs ... OK 2017-05-17 13:45:49,725 INFO status on centosvm: hook 2017-05-17 13:45:49,725 INFO manager on centosvm: list hook 2017-05-17 13:45:49,725 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-05-17 13:45:49,744 INFO status on centosvm: hook 2017-05-17 13:45:49,745 INFO manager on centosvm: list hook 2017-05-17 13:45:49,745 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-05-17 13:45:49,768 INFO manager on centosvm: create hook start 2017-05-17 13:45:49,768 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c create hook start 2017-05-17 13:45:49,790 INFO manager on centosvm: set hook start {'enabled': 'True', 'event': 'exechost_startup'} 2017-05-17 13:45:49,790 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c set hook start enabled=True,event=exechost_startup 2017-05-17 13:45:49,812 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook start 2017-05-17 13:45:49,831 INFO expect on server centosvm: enabled set True || event set exechost_startup hook start ... OK 2017-05-17 13:45:49,832 INFO manager on centosvm: import hook start {'content-type': 'application/x-python', 'input-file': '/tmp/PtlPbsZJxJJA', 'content-encoding': 'default'} 2017-05-17 13:45:49,832 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c import hook start application/x-python default /tmp/PtlPbsZJxJJA 2017-05-17 13:45:49,852 INFO server centosvm: imported hook body --- import pbs e=pbs.event() localnode=pbs.get_local_nodename() e.vnode_list[localnode].resources_available['foo_i'] = 7 e.vnode_list[localnode].resources_available['foo_f'] = 5.0 e.vnode_list[localnode].resources_available['foo_str'] = "seventyseven" --- 2017-05-17 13:45:49,852 INFO mom centosvm: sent signal -HUP 2017-05-17 13:45:49,852 INFOCLI2 centosvm: sudo -H kill -HUP 84278 2017-05-17 13:45:49,864 INFO mom centosvm2: sent signal -HUP 2017-05-17 13:45:49,865 INFOCLI2 centosvm2: ssh centosvm2 sudo -H kill -HUP 15795 2017-05-17 13:45:50,367 INFO mom centosvm_nfs: sent signal -HUP 2017-05-17 13:45:50,367 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H kill -HUP 76746 2017-05-17 13:45:50,857 INFO manager on centosvm: set server {'job_history_enable': 'True'} 2017-05-17 13:45:50,858 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server job_history_enable=True 2017-05-17 13:45:50,908 INFO manager on centosvm as root: create resource foo_str2 {'flag': 'h', 'type': 'string'} 2017-05-17 13:45:50,909 INFOCLI centosvm: sudo -H -u root /opt/pbs/bin/qmgr -c create resource foo_str2 flag=h,type=string 2017-05-17 13:45:51,022 INFO mom centosvm log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:45:51,413 INFO mom centosvm2 log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:45:51,984 INFO mom centosvm_nfs log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:45:51,984 INFO manager on centosvm as root: create resource foo_str3 {'flag': 'h', 'type': 'string'} 2017-05-17 13:45:51,984 INFOCLI centosvm: sudo -H -u root /opt/pbs/bin/qmgr -c create resource foo_str3 flag=h,type=string 2017-05-17 13:45:52,069 INFO mom centosvm log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:45:52,450 INFO mom centosvm2 log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:45:53,064 INFO mom centosvm_nfs log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:45:53,065 INFO manager on centosvm as root: create resource foo_str4 {'flag': 'h', 'type': 'string'} 2017-05-17 13:45:53,066 INFOCLI centosvm: sudo -H -u root /opt/pbs/bin/qmgr -c create resource foo_str4 flag=h,type=string 2017-05-17 13:45:53,162 INFO mom centosvm log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:45:53,710 INFO mom centosvm2 log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:45:54,111 INFO mom centosvm_nfs log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:45:54,111 INFO manager on centosvm as root: create resource stra {'flag': 'h', 'type': 'string_array'} 2017-05-17 13:45:54,112 INFOCLI centosvm: sudo -H -u root /opt/pbs/bin/qmgr -c create resource stra flag=h,type=string_array 2017-05-17 13:45:54,188 INFO mom centosvm log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:45:54,677 INFO mom centosvm2 log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:45:55,367 INFO mom centosvm_nfs log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:45:55,368 INFO manager on centosvm as root: create resource foo_i2 {'type': 'size'} 2017-05-17 13:45:55,368 INFOCLI centosvm: sudo -H -u root /opt/pbs/bin/qmgr -c create resource foo_i2 type=size 2017-05-17 13:45:55,403 INFO mom centosvm log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:45:55,929 INFO mom centosvm2 log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:45:56,443 INFO mom centosvm_nfs log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:45:56,443 INFO manager on centosvm as root: create resource foo_f2 {'type': 'float'} 2017-05-17 13:45:56,444 INFOCLI centosvm: sudo -H -u root /opt/pbs/bin/qmgr -c create resource foo_f2 type=float 2017-05-17 13:45:56,495 INFO mom centosvm log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:45:57,047 INFO mom centosvm2 log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:45:57,626 INFO mom centosvm_nfs log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:45:57,627 INFO manager on centosvm as root: create resource stra2 {'type': 'string_array'} 2017-05-17 13:45:57,628 INFOCLI centosvm: sudo -H -u root /opt/pbs/bin/qmgr -c create resource stra2 type=string_array 2017-05-17 13:45:57,716 INFO mom centosvm log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:45:58,178 INFO mom centosvm2 log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:45:58,650 INFO mom centosvm_nfs log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:45:58,651 INFO status on centosvm: hook 2017-05-17 13:45:58,651 INFO manager on centosvm: list hook 2017-05-17 13:45:58,652 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-05-17 13:45:58,699 INFO status on centosvm: hook 2017-05-17 13:45:58,699 INFO manager on centosvm: list hook 2017-05-17 13:45:58,699 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-05-17 13:45:58,724 INFO manager on centosvm: create hook epi 2017-05-17 13:45:58,724 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c create hook epi 2017-05-17 13:45:58,752 INFO manager on centosvm: set hook epi {'enabled': 'True', 'event': 'execjob_epilogue'} 2017-05-17 13:45:58,752 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c set hook epi enabled=True,event=execjob_epilogue 2017-05-17 13:45:58,774 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook epi 2017-05-17 13:45:58,795 INFO expect on server centosvm: enabled set True || event set execjob_epilogue hook epi ... OK 2017-05-17 13:45:58,796 INFO manager on centosvm: import hook epi {'content-type': 'application/x-python', 'input-file': '/tmp/PtlPbs0hLshf', 'content-encoding': 'default'} 2017-05-17 13:45:58,796 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c import hook epi application/x-python default /tmp/PtlPbs0hLshf 2017-05-17 13:45:58,817 INFO server centosvm: imported hook body --- import pbs e = pbs.event() j = e.job pbs.logmsg(pbs.LOG_DEBUG, "executed epilogue hook") j.resources_used["foo_i"] = 2 j.resources_used["foo_i2"] = pbs.size(1000) j.resources_used["foo_f"] = 1.02 j.resources_used["foo_f2"] = 2.01 j.resources_used["stra"] = '"happy"' j.resources_used["stra2"] = '"glad"' --- 2017-05-17 13:45:58,817 INFOCLI centosvm: sudo -H -u pbsuser /opt/pbs/bin/pbs_rsub -R 201705171346.08 -l place=scatter -E 201705171346.28 -l select=3:ncpus=1 2017-05-17 13:45:58,854 INFO submit to centosvm as pbsuser: reservation R11.centosvm UNCONFIRMED OrderedDict([('reserve_start', 1495008968.817543), ('Resource_List.place', 'scatter'), ('reserve_end', 1495008988.817545), ('Resource_List.select', '3:ncpus=1')]) 2017-05-17 13:45:58,854 INFOCLI centosvm: /opt/pbs/bin/pbs_rstat -f R11.centosvm 2017-05-17 13:45:58,871 INFO expect on server centosvm: reserve_state ~ RESV_CONFIRMED|2 reservation R11.centosvm got: reserve_state = RESV_UNCONFIRMED 2017-05-17 13:45:59,372 INFOCLI centosvm: /opt/pbs/bin/pbs_rstat -f R11.centosvm 2017-05-17 13:45:59,416 INFO expect on server centosvm: reserve_state ~ RESV_CONFIRMED|2 reservation R11.centosvm attempt: 2 ... OK 2017-05-17 13:45:59,417 INFO job: executable set to /bin/sleep with arguments: 100 2017-05-17 13:45:59,418 INFOCLI centosvm: sudo -H -u pbsuser /opt/pbs/bin/qsub -q R11 -l select=3:ncpus=1 -- /bin/sleep 100 2017-05-17 13:45:59,459 INFO submit to centosvm as pbsuser: job 12.centosvm OrderedDict([('queue', 'R11'), ('Resource_List.select', '3:ncpus=1')]) 2017-05-17 13:45:59,459 INFO server centosvm: expect offset set to 30 2017-05-17 13:46:29,493 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 12.centosvm 2017-05-17 13:46:29,536 INFO expect on server centosvm: resources_used.foo_i2 = 3kb || resources_used.foo_i = 6 || job_state = F || resources_used.foo_f = 3.06 || resources_used.stra2 = "glad" || resources_used.stra = "happy" || resources_used.foo_f2 = 6.03 job 12.centosvm 2017-05-17 13:46:30,540 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:46:30,559 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 12.centosvm 2017-05-17 13:46:30,580 INFO expect on server centosvm: resources_used.foo_i2 = 3kb || resources_used.foo_i = 6 || job_state = F || resources_used.foo_f = 3.06 || resources_used.stra2 = "glad" || resources_used.stra = "happy" || resources_used.foo_f2 = 6.03 job 12.centosvm attempt: 2 2017-05-17 13:46:31,585 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:46:31,630 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 12.centosvm 2017-05-17 13:46:31,668 INFO expect on server centosvm: resources_used.foo_i2 = 3kb || resources_used.foo_i = 6 || job_state = F || resources_used.foo_f = 3.06 || resources_used.stra2 = "glad" || resources_used.stra = "happy" || resources_used.foo_f2 = 6.03 job 12.centosvm attempt: 3 ... OK 2017-05-17 13:46:31,669 INFOCLI centosvm: /opt/pbs/bin/qstat -Bf centosvm 2017-05-17 13:46:31,712 INFO server centosvm: stopping Server on host centosvm 2017-05-17 13:46:31,713 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/server_priv/server.lock 2017-05-17 13:46:31,735 INFOCLI2 centosvm: sudo -H kill -TERM 84707 2017-05-17 13:46:31,756 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/server_priv/server.lock 2017-05-17 13:46:36,980 INFO service: starting pbs_server 2017-05-17 13:46:36,981 INFOCLI centosvm: sudo -H /opt/pbs/sbin/pbs_server 2017-05-17 13:46:43,698 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/server_priv/server.lock 2017-05-17 13:46:43,772 INFOCLI centosvm: /opt/pbs/bin/qstat -Bf centosvm 2017-05-17 13:46:43,791 INFO =========================================== 2017-05-17 13:46:43,792 INFO Entered TestPbsAccumulateRescUsed tearDown 2017-05-17 13:46:43,792 INFO =========================================== 2017-05-17 13:46:43,792 INFO ============================================ 2017-05-17 13:46:43,792 INFO Completed TestPbsAccumulateRescUsed tearDown 2017-05-17 13:46:43,792 INFO ============================================ 2017-05-17 13:46:43,792 INFO ok 2017-05-17 13:46:43,793 INFO test name: test_resource_bool (tests.functional.pbs_accumulate_resc_used.TestPbsAccumulateRescUsed)... 2017-05-17 13:46:43,793 INFO test start time: Wed May 17 13:46:43 2017 2017-05-17 13:46:43,793 INFO test docstring: To test that boolean value are not getting aggregated 2017-05-17 13:46:43,793 INFO ======================================== 2017-05-17 13:46:43,793 INFO Entered TestPbsAccumulateRescUsed setUp 2017-05-17 13:46:43,793 INFO ======================================== 2017-05-17 13:46:43,793 INFOCLI centosvm: /opt/pbs/bin/qstat -Bf centosvm 2017-05-17 13:46:43,805 INFO status on centosvm: server 2017-05-17 13:46:43,805 INFOCLI centosvm: /opt/pbs/bin/qstat -Bf centosvm 2017-05-17 13:46:43,821 INFO manager on centosvm: set server {'managers': (3, 'dilipk@*')} 2017-05-17 13:46:43,821 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c set server managers-=dilipk@* 2017-05-17 13:46:43,850 INFO manager on centosvm: set server {'managers': (2, 'dilipk@*')} 2017-05-17 13:46:43,851 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c set server managers+=dilipk@* 2017-05-17 13:46:43,874 INFO server centosvm: reverting configuration to defaults 2017-05-17 13:46:43,874 INFOCLI centosvm: /opt/pbs/bin/qstat -Bf centosvm 2017-05-17 13:46:43,891 INFO select on centosvm: __ALL__ 2017-05-17 13:46:43,891 INFOCLI centosvm: /opt/pbs/bin/qselect -x 2017-05-17 13:46:43,901 INFOCLI centosvm: /opt/pbs/bin/qstat -f @centosvm 2017-05-17 13:46:43,915 INFO expect on server centosvm: job_state set 0 job ... OK 2017-05-17 13:46:43,915 INFOCLI centosvm: /opt/pbs/bin/pbs_rstat -f 2017-05-17 13:46:43,926 INFO delete resv on centosvm: R11.centosvm 2017-05-17 13:46:43,926 INFOCLI centosvm: /opt/pbs/bin/pbs_rdel R11.centosvm 2017-05-17 13:46:43,939 INFOCLI centosvm: /opt/pbs/bin/pbs_rstat -f 2017-05-17 13:46:43,956 INFO manager on centosvm: unset server ['pbs_license_max', 'pbs_license_min', 'job_history_enable'] 2017-05-17 13:46:43,956 INFOCLI centosvm: /opt/pbs/bin/qmgr -c unset server pbs_license_max,pbs_license_min,job_history_enable 2017-05-17 13:46:43,972 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-05-17 13:46:43,998 INFO manager on centosvm: delete hook start,epi 2017-05-17 13:46:43,998 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c delete hook start,epi 2017-05-17 13:46:44,021 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook start 2017-05-17 13:46:44,043 INFO expect on server centosvm: unset hook start ... OK 2017-05-17 13:46:44,043 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook epi 2017-05-17 13:46:44,063 INFO expect on server centosvm: unset hook epi ... OK 2017-05-17 13:46:44,063 INFOCLI centosvm: /opt/pbs/bin/qstat -Qf @centosvm 2017-05-17 13:46:44,078 INFO manager on centosvm: delete queue workq 2017-05-17 13:46:44,078 INFOCLI centosvm: /opt/pbs/bin/qmgr -c delete queue workq 2017-05-17 13:46:44,091 INFOCLI centosvm: /opt/pbs/bin/qstat -Qf workq@centosvm 2017-05-17 13:46:44,101 INFO expect on server centosvm: unset queue workq ... OK 2017-05-17 13:46:44,101 INFO manager on centosvm: create queue workq {'started': 'True', 'queue_type': 'Execution', 'enabled': 'True'} 2017-05-17 13:46:44,102 INFOCLI centosvm: /opt/pbs/bin/qmgr -c create queue workq started=True,queue_type=Execution,enabled=True 2017-05-17 13:46:44,119 INFOCLI centosvm: /opt/pbs/bin/qstat -Qf workq@centosvm 2017-05-17 13:46:44,130 INFO expect on server centosvm: started set True || queue_type set Execution || enabled set True queue workq ... OK 2017-05-17 13:46:44,130 INFO manager on centosvm: set server {'default_queue': 'workq'} 2017-05-17 13:46:44,130 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server default_queue=workq 2017-05-17 13:46:44,143 INFO status on centosvm: resource 2017-05-17 13:46:44,143 INFO manager on centosvm: list resource 2017-05-17 13:46:44,143 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource 2017-05-17 13:46:44,157 INFO manager on centosvm: delete resource foo_f,foo_i,foo_str2,foo_str3,foo_str4,stra2,foo_i2,foo_str,stra,foo_f2 2017-05-17 13:46:44,158 INFOCLI centosvm: /opt/pbs/bin/qmgr -c delete resource foo_f,foo_i,foo_str2,foo_str3,foo_str4,stra2,foo_i2,foo_str,stra,foo_f2 2017-05-17 13:46:44,368 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_f 2017-05-17 13:46:44,379 INFO expect on server centosvm: unset resource foo_f ... OK 2017-05-17 13:46:44,379 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_i 2017-05-17 13:46:44,390 INFO expect on server centosvm: unset resource foo_i ... OK 2017-05-17 13:46:44,391 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_str2 2017-05-17 13:46:44,403 INFO expect on server centosvm: unset resource foo_str2 ... OK 2017-05-17 13:46:44,404 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_str3 2017-05-17 13:46:44,417 INFO expect on server centosvm: unset resource foo_str3 ... OK 2017-05-17 13:46:44,417 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_str4 2017-05-17 13:46:44,428 INFO expect on server centosvm: unset resource foo_str4 ... OK 2017-05-17 13:46:44,428 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource stra2 2017-05-17 13:46:44,441 INFO expect on server centosvm: unset resource stra2 ... OK 2017-05-17 13:46:44,442 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_i2 2017-05-17 13:46:44,455 INFO expect on server centosvm: unset resource foo_i2 ... OK 2017-05-17 13:46:44,455 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_str 2017-05-17 13:46:44,466 INFO expect on server centosvm: unset resource foo_str ... OK 2017-05-17 13:46:44,466 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource stra 2017-05-17 13:46:44,479 INFO expect on server centosvm: unset resource stra ... OK 2017-05-17 13:46:44,479 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_f2 2017-05-17 13:46:44,490 INFO expect on server centosvm: unset resource foo_f2 ... OK 2017-05-17 13:46:44,490 INFOCLI status on centosvm: server license_count 2017-05-17 13:46:44,490 INFOCLI centosvm: /opt/pbs/bin/qstat -Bf centosvm 2017-05-17 13:46:44,501 INFO server: centosvm licensed 2017-05-17 13:46:44,516 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/server_priv/comm.lock 2017-05-17 13:46:44,539 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-05-17 13:46:44,549 INFO scheduler centosvm: reverting configuration to defaults 2017-05-17 13:46:44,549 INFO manager on centosvm: list sched 2017-05-17 13:46:44,549 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list sched 2017-05-17 13:46:44,560 INFO manager on centosvm: unset sched ['sched_cycle_length'] 2017-05-17 13:46:44,561 INFOCLI centosvm: /opt/pbs/bin/qmgr -c unset sched sched_cycle_length 2017-05-17 13:46:44,574 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/dedicated_time 2017-05-17 13:46:44,584 INFOCLI2 centosvm: cmp /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-05-17 13:46:44,587 ERROR err: ['cmp: /var/spool/pbs/sched_priv/resource_group: Permission denied'] 2017-05-17 13:46:44,587 INFOCLI2 centosvm: sudo -H /usr/bin/cp -p /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-05-17 13:46:44,598 INFOCLI2 centosvm: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-05-17 13:46:44,608 INFO scheduler centosvm: reverting holidays file to default 2017-05-17 13:46:44,609 INFOCLI2 centosvm: cmp /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-05-17 13:46:44,611 ERROR err: ['cmp: /var/spool/pbs/sched_priv/holidays: Permission denied'] 2017-05-17 13:46:44,612 INFOCLI2 centosvm: sudo -H /usr/bin/cp -p /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-05-17 13:46:44,624 INFOCLI2 centosvm: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/holidays 2017-05-17 13:46:44,633 INFO scheduler centosvm: Parsing holidays file 2017-05-17 13:46:44,634 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2017-05-17 13:46:44,646 INFOCLI2 centosvm: cmp /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-05-17 13:46:44,649 ERROR err: ['cmp: /var/spool/pbs/sched_priv/sched_config: Permission denied'] 2017-05-17 13:46:44,649 INFOCLI2 centosvm: sudo -H /usr/bin/cp -p /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-05-17 13:46:44,659 INFOCLI2 centosvm: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-05-17 13:46:44,669 INFO scheduler centosvm: sent signal -HUP 2017-05-17 13:46:44,670 INFOCLI2 centosvm: sudo -H kill -HUP 84281 2017-05-17 13:46:44,679 INFOCLI2 centosvm: sudo -H /usr/bin/rm /var/spool/pbs/sched_priv/usage 2017-05-17 13:46:44,690 ERROR err: ['/usr/bin/rm: cannot remove \xe2\x80\x98/var/spool/pbs/sched_priv/usage\xe2\x80\x99: No such file or directory'] 2017-05-17 13:46:44,715 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-05-17 13:46:44,726 INFOCLI2 centosvm: sudo -H cmp /opt/pbs/sbin/pbs_mom /opt/pbs/sbin/pbs_mom.cpuset 2017-05-17 13:46:44,749 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:46:44,762 INFO mom centosvm: reverting configuration to defaults 2017-05-17 13:46:44,763 INFOCLI2 centosvm: sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2017-05-17 13:46:44,774 INFOCLI2 centosvm: sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2017-05-17 13:46:44,784 INFOCLI2 centosvm: sudo -H ls /var/spool/pbs/mom_priv/config.d 2017-05-17 13:46:44,795 INFO mom centosvm: sent signal -HUP 2017-05-17 13:46:44,795 INFOCLI2 centosvm: sudo -H kill -HUP 84278 2017-05-17 13:46:44,820 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:46:44,832 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v -a 2017-05-17 13:46:44,848 INFO manager on centosvm: delete node 2017-05-17 13:46:44,848 INFOCLI centosvm: /opt/pbs/bin/qmgr -c delete node @default 2017-05-17 13:46:44,863 INFOCLI2 centosvm: sudo -H /usr/bin/rm -rf /var/spool/pbs/mom_priv/config.d 2017-05-17 13:46:44,874 INFO mom centosvm: stopping MoM on host centosvm 2017-05-17 13:46:44,874 INFOCLI2 centosvm: sudo -H kill -TERM 84278 2017-05-17 13:46:44,885 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:46:44,910 INFO service: starting pbs_mom -d /var/spool/pbs 2017-05-17 13:46:44,910 INFOCLI centosvm: sudo -H /opt/pbs/sbin/pbs_mom -d /var/spool/pbs 2017-05-17 13:46:44,947 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:46:44,971 INFO server: no nodes defined, creating one 2017-05-17 13:46:44,971 INFO manager on centosvm: create node centosvm 2017-05-17 13:46:44,971 INFOCLI centosvm: /opt/pbs/bin/qmgr -c create node centosvm 2017-05-17 13:46:44,986 INFO manager on centosvm: set node centosvm {'resources_available.ncpus': 8} 2017-05-17 13:46:44,986 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set node centosvm resources_available.ncpus=8 2017-05-17 13:46:45,000 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:46:45,010 INFO expect on server centosvm: resources_available.ncpus set 8 node centosvm ... OK 2017-05-17 13:46:45,010 INFO status on centosvm: node centosvm 2017-05-17 13:46:45,010 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:46:45,026 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:46:45,038 INFO expect on server centosvm: state = free node centosvm got: state = state-unknown,down 2017-05-17 13:46:46,039 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:46:46,052 INFO expect on server centosvm: state = free node centosvm attempt: 2 ... OK 2017-05-17 13:46:46,052 INFOCLI2 centosvm2: ssh centosvm2 sudo -H cmp /opt/pbs/sbin/pbs_mom /opt/pbs/sbin/pbs_mom.cpuset 2017-05-17 13:46:47,091 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:46:47,518 INFO mom centosvm2: reverting configuration to defaults 2017-05-17 13:46:47,519 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2017-05-17 13:46:47,974 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2017-05-17 13:46:48,449 INFOCLI2 centosvm2: ssh centosvm2 sudo -H ls /var/spool/pbs/mom_priv/config.d 2017-05-17 13:46:48,889 INFO mom centosvm2: sent signal -HUP 2017-05-17 13:46:48,889 INFOCLI2 centosvm2: ssh centosvm2 sudo -H kill -HUP 15795 2017-05-17 13:46:49,815 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:46:50,656 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v -a 2017-05-17 13:46:50,687 INFO status on centosvm: node centosvm2 2017-05-17 13:46:50,688 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm2 2017-05-17 13:46:50,708 ERROR err: ['Node: centosvm2, Error: Unknown node '] 2017-05-17 13:46:50,708 INFO status on centosvm: node centosvm2 2017-05-17 13:46:50,709 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm2 2017-05-17 13:46:50,725 ERROR err: ['Node: centosvm2, Error: Unknown node '] 2017-05-17 13:46:50,725 INFO manager on centosvm: create node centosvm2 2017-05-17 13:46:50,725 INFOCLI centosvm: /opt/pbs/bin/qmgr -c create node centosvm2 2017-05-17 13:46:50,740 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm2 2017-05-17 13:46:50,759 INFO expect on server centosvm: state = free node centosvm2 got: state = state-unknown,down 2017-05-17 13:46:51,760 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm2 2017-05-17 13:46:51,801 INFO expect on server centosvm: state = free node centosvm2 attempt: 2 got: state = state-unknown,down 2017-05-17 13:46:52,803 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm2 2017-05-17 13:46:52,826 INFO expect on server centosvm: state = free node centosvm2 attempt: 3 got: state = state-unknown,down 2017-05-17 13:46:53,828 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm2 2017-05-17 13:46:53,841 INFO expect on server centosvm: state = free node centosvm2 attempt: 4 ... OK 2017-05-17 13:46:53,841 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H cmp /opt/pbs/sbin/pbs_mom /opt/pbs/sbin/pbs_mom.cpuset 2017-05-17 13:46:54,647 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:46:55,014 INFO mom centosvm_nfs: reverting configuration to defaults 2017-05-17 13:46:55,014 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2017-05-17 13:46:55,477 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2017-05-17 13:46:55,862 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H ls /var/spool/pbs/mom_priv/config.d 2017-05-17 13:46:56,496 INFO mom centosvm_nfs: sent signal -HUP 2017-05-17 13:46:56,496 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H kill -HUP 76746 2017-05-17 13:46:57,529 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:46:58,035 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v -a 2017-05-17 13:46:58,054 INFO status on centosvm: node centosvm_nfs 2017-05-17 13:46:58,055 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm_nfs 2017-05-17 13:46:58,079 ERROR err: ['Node: centosvm_nfs, Error: Unknown node '] 2017-05-17 13:46:58,080 INFO status on centosvm: node centosvm_nfs 2017-05-17 13:46:58,080 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm_nfs 2017-05-17 13:46:58,102 ERROR err: ['Node: centosvm_nfs, Error: Unknown node '] 2017-05-17 13:46:58,102 INFO manager on centosvm: create node centosvm_nfs 2017-05-17 13:46:58,103 INFOCLI centosvm: /opt/pbs/bin/qmgr -c create node centosvm_nfs 2017-05-17 13:46:58,124 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm_nfs 2017-05-17 13:46:58,140 INFO expect on server centosvm: state = free node centosvm_nfs got: state = state-unknown,down 2017-05-17 13:46:59,142 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm_nfs 2017-05-17 13:46:59,159 INFO expect on server centosvm: state = free node centosvm_nfs attempt: 2 got: state = state-unknown,down 2017-05-17 13:47:00,160 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm_nfs 2017-05-17 13:47:00,194 INFO expect on server centosvm: state = free node centosvm_nfs attempt: 3 got: state = state-unknown,down 2017-05-17 13:47:01,196 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm_nfs 2017-05-17 13:47:01,232 INFO expect on server centosvm: state = free node centosvm_nfs attempt: 4 ... OK 2017-05-17 13:47:01,233 INFO ========================================= 2017-05-17 13:47:01,233 INFO Completed TestPbsAccumulateRescUsed setUp 2017-05-17 13:47:01,234 INFO ========================================= 2017-05-17 13:47:01,234 INFO len moms = 3 2017-05-17 13:47:01,235 INFOCLI2 centosvm: sudo -H /usr/bin/rm -rf /var/spool/pbs/mom_priv/config.d 2017-05-17 13:47:01,263 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/rm -rf /var/spool/pbs/mom_priv/config.d 2017-05-17 13:47:01,698 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H /usr/bin/rm -rf /var/spool/pbs/mom_priv/config.d 2017-05-17 13:47:02,212 INFO manager on centosvm: delete node 2017-05-17 13:47:02,212 INFOCLI centosvm: /opt/pbs/bin/qmgr -c delete node @default 2017-05-17 13:47:02,257 INFO manager on centosvm: create node centosvm 2017-05-17 13:47:02,258 INFOCLI centosvm: /opt/pbs/bin/qmgr -c create node centosvm 2017-05-17 13:47:02,283 INFO manager on centosvm: create node centosvm2 2017-05-17 13:47:02,284 INFOCLI centosvm: /opt/pbs/bin/qmgr -c create node centosvm2 2017-05-17 13:47:02,303 INFO manager on centosvm: create node centosvm_nfs 2017-05-17 13:47:02,303 INFOCLI centosvm: /opt/pbs/bin/qmgr -c create node centosvm_nfs 2017-05-17 13:47:02,330 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:47:02,346 INFO expect on server centosvm: state = free node centosvm got: state = state-unknown,down 2017-05-17 13:47:02,848 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:47:02,891 INFO expect on server centosvm: state = free node centosvm attempt: 2 got: state = state-unknown,down 2017-05-17 13:47:03,393 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:47:03,462 INFO expect on server centosvm: state = free node centosvm attempt: 3 got: state = state-unknown,down 2017-05-17 13:47:03,964 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:47:04,001 INFO expect on server centosvm: state = free node centosvm attempt: 4 got: state = state-unknown,down 2017-05-17 13:47:04,503 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:47:04,572 INFO expect on server centosvm: state = free node centosvm attempt: 5 ... OK 2017-05-17 13:47:04,574 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm2 2017-05-17 13:47:04,602 INFO expect on server centosvm: state = free node centosvm2 ... OK 2017-05-17 13:47:04,603 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm_nfs 2017-05-17 13:47:04,624 INFO expect on server centosvm: state = free node centosvm_nfs ... OK 2017-05-17 13:47:04,625 INFO status on centosvm: hook 2017-05-17 13:47:04,625 INFO manager on centosvm: list hook 2017-05-17 13:47:04,625 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-05-17 13:47:04,650 INFO status on centosvm: hook 2017-05-17 13:47:04,650 INFO manager on centosvm: list hook 2017-05-17 13:47:04,650 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-05-17 13:47:04,672 INFO manager on centosvm: create hook start 2017-05-17 13:47:04,672 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c create hook start 2017-05-17 13:47:04,697 INFO manager on centosvm: set hook start {'enabled': 'True', 'event': 'exechost_startup'} 2017-05-17 13:47:04,698 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c set hook start enabled=True,event=exechost_startup 2017-05-17 13:47:04,725 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook start 2017-05-17 13:47:04,746 INFO expect on server centosvm: enabled set True || event set exechost_startup hook start ... OK 2017-05-17 13:47:04,746 INFO manager on centosvm: import hook start {'content-type': 'application/x-python', 'input-file': '/tmp/PtlPbsGJ7RLU', 'content-encoding': 'default'} 2017-05-17 13:47:04,746 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c import hook start application/x-python default /tmp/PtlPbsGJ7RLU 2017-05-17 13:47:04,771 INFO server centosvm: imported hook body --- import pbs e=pbs.event() localnode=pbs.get_local_nodename() e.vnode_list[localnode].resources_available['foo_i'] = 7 e.vnode_list[localnode].resources_available['foo_f'] = 5.0 e.vnode_list[localnode].resources_available['foo_str'] = "seventyseven" --- 2017-05-17 13:47:04,771 INFO mom centosvm: sent signal -HUP 2017-05-17 13:47:04,771 INFOCLI2 centosvm: sudo -H kill -HUP 90805 2017-05-17 13:47:04,783 INFO mom centosvm2: sent signal -HUP 2017-05-17 13:47:04,784 INFOCLI2 centosvm2: ssh centosvm2 sudo -H kill -HUP 15795 2017-05-17 13:47:05,208 INFO mom centosvm_nfs: sent signal -HUP 2017-05-17 13:47:05,208 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H kill -HUP 76746 2017-05-17 13:47:05,726 INFO manager on centosvm: set server {'job_history_enable': 'True'} 2017-05-17 13:47:05,727 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server job_history_enable=True 2017-05-17 13:47:05,756 INFO manager on centosvm as root: create resource foo_str2 {'flag': 'h', 'type': 'string'} 2017-05-17 13:47:05,757 INFOCLI centosvm: sudo -H -u root /opt/pbs/bin/qmgr -c create resource foo_str2 flag=h,type=string 2017-05-17 13:47:05,806 INFO mom centosvm log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:47:06,254 INFO mom centosvm2 log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:47:06,637 INFO mom centosvm_nfs log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:47:06,638 INFO manager on centosvm as root: create resource foo_str3 {'flag': 'h', 'type': 'string'} 2017-05-17 13:47:06,638 INFOCLI centosvm: sudo -H -u root /opt/pbs/bin/qmgr -c create resource foo_str3 flag=h,type=string 2017-05-17 13:47:06,696 INFO mom centosvm log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:47:07,061 INFO mom centosvm2 log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:47:07,441 INFO mom centosvm_nfs log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:47:07,442 INFO manager on centosvm as root: create resource foo_str4 {'flag': 'h', 'type': 'string'} 2017-05-17 13:47:07,442 INFOCLI centosvm: sudo -H -u root /opt/pbs/bin/qmgr -c create resource foo_str4 flag=h,type=string 2017-05-17 13:47:07,506 INFO mom centosvm log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:47:07,860 INFO mom centosvm2 log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:47:08,226 INFO mom centosvm_nfs log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:47:08,226 INFO manager on centosvm as root: create resource stra {'flag': 'h', 'type': 'string_array'} 2017-05-17 13:47:08,226 INFOCLI centosvm: sudo -H -u root /opt/pbs/bin/qmgr -c create resource stra flag=h,type=string_array 2017-05-17 13:47:08,271 INFO mom centosvm log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:47:08,630 INFO mom centosvm2 log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:47:08,975 INFO mom centosvm_nfs log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:47:08,975 INFO manager on centosvm as root: create resource foo_bool {'type': 'boolean'} 2017-05-17 13:47:08,975 INFOCLI centosvm: sudo -H -u root /opt/pbs/bin/qmgr -c create resource foo_bool type=boolean 2017-05-17 13:47:09,015 INFO status on centosvm: hook 2017-05-17 13:47:09,015 INFO manager on centosvm: list hook 2017-05-17 13:47:09,016 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-05-17 13:47:09,039 INFO status on centosvm: hook 2017-05-17 13:47:09,040 INFO manager on centosvm: list hook 2017-05-17 13:47:09,040 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-05-17 13:47:09,064 INFO manager on centosvm: create hook epi_bool 2017-05-17 13:47:09,064 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c create hook epi_bool 2017-05-17 13:47:09,087 INFO manager on centosvm: set hook epi_bool {'enabled': 'True', 'event': 'execjob_epilogue'} 2017-05-17 13:47:09,087 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c set hook epi_bool enabled=True,event=execjob_epilogue 2017-05-17 13:47:09,113 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook epi_bool 2017-05-17 13:47:09,137 INFO expect on server centosvm: enabled set True || event set execjob_epilogue hook epi_bool ... OK 2017-05-17 13:47:09,137 INFO manager on centosvm: import hook epi_bool {'content-type': 'application/x-python', 'input-file': '/tmp/PtlPbsvjY7AN', 'content-encoding': 'default'} 2017-05-17 13:47:09,138 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c import hook epi_bool application/x-python default /tmp/PtlPbsvjY7AN 2017-05-17 13:47:09,161 INFO server centosvm: imported hook body --- import pbs e=pbs.event() j=e.job if j.in_ms_mom(): j.resources_used["foo_bool"] = True else: j.resources_used["foo_bool"] = False --- 2017-05-17 13:47:09,161 INFO job: executable set to /bin/sleep with arguments: 100 2017-05-17 13:47:09,162 INFO job: executable set to /bin/sleep with arguments: 5 2017-05-17 13:47:09,162 INFOCLI centosvm: sudo -H -u pbsuser /opt/pbs/bin/qsub -l place=scatter -l walltime=10 -l select=3:ncpus=1 -- /bin/sleep 5 2017-05-17 13:47:09,190 INFO submit to centosvm as pbsuser: job 13.centosvm OrderedDict([('Resource_List.place', 'scatter'), ('Resource_List.walltime', 10), ('Resource_List.select', '3:ncpus=1')]) 2017-05-17 13:47:09,191 INFO server centosvm: expect offset set to 5 2017-05-17 13:47:14,197 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 13.centosvm 2017-05-17 13:47:14,226 INFO expect on server centosvm: resources_used.foo_bool = True || job_state = F job 13.centosvm 2017-05-17 13:47:14,728 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:47:14,778 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 13.centosvm 2017-05-17 13:47:14,800 INFO expect on server centosvm: resources_used.foo_bool = True || job_state = F job 13.centosvm attempt: 2 2017-05-17 13:47:15,302 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:47:15,325 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 13.centosvm 2017-05-17 13:47:15,338 INFO expect on server centosvm: resources_used.foo_bool = True || job_state = F job 13.centosvm attempt: 3 2017-05-17 13:47:15,839 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:47:15,885 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 13.centosvm 2017-05-17 13:47:15,916 INFO expect on server centosvm: resources_used.foo_bool = True || job_state = F job 13.centosvm attempt: 4 ... OK 2017-05-17 13:47:15,917 INFO =========================================== 2017-05-17 13:47:15,917 INFO Entered TestPbsAccumulateRescUsed tearDown 2017-05-17 13:47:15,917 INFO =========================================== 2017-05-17 13:47:15,917 INFO ============================================ 2017-05-17 13:47:15,917 INFO Completed TestPbsAccumulateRescUsed tearDown 2017-05-17 13:47:15,918 INFO ============================================ 2017-05-17 13:47:15,918 INFO ok 2017-05-17 13:47:15,919 INFO test name: test_resource_invisible (tests.functional.pbs_accumulate_resc_used.TestPbsAccumulateRescUsed)... 2017-05-17 13:47:15,919 INFO test start time: Wed May 17 13:47:15 2017 2017-05-17 13:47:15,919 INFO test docstring: Test that value aggregation is same for invisible resources 2017-05-17 13:47:15,919 INFO ======================================== 2017-05-17 13:47:15,920 INFO Entered TestPbsAccumulateRescUsed setUp 2017-05-17 13:47:15,920 INFO ======================================== 2017-05-17 13:47:15,920 INFOCLI centosvm: /opt/pbs/bin/qstat -Bf centosvm 2017-05-17 13:47:15,945 INFO status on centosvm: server 2017-05-17 13:47:15,946 INFOCLI centosvm: /opt/pbs/bin/qstat -Bf centosvm 2017-05-17 13:47:15,982 INFO manager on centosvm: set server {'managers': (3, 'dilipk@*')} 2017-05-17 13:47:15,982 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c set server managers-=dilipk@* 2017-05-17 13:47:16,029 INFO manager on centosvm: set server {'managers': (2, 'dilipk@*')} 2017-05-17 13:47:16,029 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c set server managers+=dilipk@* 2017-05-17 13:47:16,061 INFO server centosvm: reverting configuration to defaults 2017-05-17 13:47:16,061 INFOCLI centosvm: /opt/pbs/bin/qstat -Bf centosvm 2017-05-17 13:47:16,077 INFO select on centosvm: __ALL__ 2017-05-17 13:47:16,078 INFOCLI centosvm: /opt/pbs/bin/qselect -x 2017-05-17 13:47:16,092 INFO delete job on centosvm: 13.centosvm 2017-05-17 13:47:16,093 INFOCLI centosvm: /opt/pbs/bin/qdel -W force -x 13.centosvm 2017-05-17 13:47:16,114 INFOCLI centosvm: /opt/pbs/bin/qstat -f 13.centosvm 2017-05-17 13:47:16,134 INFOCLI centosvm: /opt/pbs/bin/qstat -f @centosvm 2017-05-17 13:47:16,155 INFO expect on server centosvm: job_state set 0 job ... OK 2017-05-17 13:47:16,156 INFOCLI centosvm: /opt/pbs/bin/pbs_rstat -f 2017-05-17 13:47:16,170 INFO manager on centosvm: unset server ['job_history_enable'] 2017-05-17 13:47:16,170 INFOCLI centosvm: /opt/pbs/bin/qmgr -c unset server job_history_enable 2017-05-17 13:47:16,191 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-05-17 13:47:16,213 INFO manager on centosvm: delete hook start,epi_bool 2017-05-17 13:47:16,213 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c delete hook start,epi_bool 2017-05-17 13:47:16,240 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook start 2017-05-17 13:47:16,270 INFO expect on server centosvm: unset hook start ... OK 2017-05-17 13:47:16,270 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook epi_bool 2017-05-17 13:47:16,289 INFO expect on server centosvm: unset hook epi_bool ... OK 2017-05-17 13:47:16,290 INFOCLI centosvm: /opt/pbs/bin/qstat -Qf @centosvm 2017-05-17 13:47:16,305 INFO manager on centosvm: delete queue workq 2017-05-17 13:47:16,306 INFOCLI centosvm: /opt/pbs/bin/qmgr -c delete queue workq 2017-05-17 13:47:16,319 INFOCLI centosvm: /opt/pbs/bin/qstat -Qf workq@centosvm 2017-05-17 13:47:16,329 INFO expect on server centosvm: unset queue workq ... OK 2017-05-17 13:47:16,329 INFO manager on centosvm: create queue workq {'started': 'True', 'queue_type': 'Execution', 'enabled': 'True'} 2017-05-17 13:47:16,330 INFOCLI centosvm: /opt/pbs/bin/qmgr -c create queue workq started=True,queue_type=Execution,enabled=True 2017-05-17 13:47:16,343 INFOCLI centosvm: /opt/pbs/bin/qstat -Qf workq@centosvm 2017-05-17 13:47:16,355 INFO expect on server centosvm: started set True || queue_type set Execution || enabled set True queue workq ... OK 2017-05-17 13:47:16,355 INFO manager on centosvm: set server {'default_queue': 'workq'} 2017-05-17 13:47:16,355 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server default_queue=workq 2017-05-17 13:47:16,369 INFO status on centosvm: resource 2017-05-17 13:47:16,369 INFO manager on centosvm: list resource 2017-05-17 13:47:16,369 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource 2017-05-17 13:47:16,383 INFO manager on centosvm: delete resource foo_f,foo_bool,foo_i,foo_str2,foo_str3,foo_str4,foo_str,stra 2017-05-17 13:47:16,384 INFOCLI centosvm: /opt/pbs/bin/qmgr -c delete resource foo_f,foo_bool,foo_i,foo_str2,foo_str3,foo_str4,foo_str,stra 2017-05-17 13:47:16,570 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_f 2017-05-17 13:47:16,581 INFO expect on server centosvm: unset resource foo_f ... OK 2017-05-17 13:47:16,582 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_bool 2017-05-17 13:47:16,593 INFO expect on server centosvm: unset resource foo_bool ... OK 2017-05-17 13:47:16,593 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_i 2017-05-17 13:47:16,605 INFO expect on server centosvm: unset resource foo_i ... OK 2017-05-17 13:47:16,606 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_str2 2017-05-17 13:47:16,618 INFO expect on server centosvm: unset resource foo_str2 ... OK 2017-05-17 13:47:16,618 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_str3 2017-05-17 13:47:16,628 INFO expect on server centosvm: unset resource foo_str3 ... OK 2017-05-17 13:47:16,629 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_str4 2017-05-17 13:47:16,639 INFO expect on server centosvm: unset resource foo_str4 ... OK 2017-05-17 13:47:16,640 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_str 2017-05-17 13:47:16,650 INFO expect on server centosvm: unset resource foo_str ... OK 2017-05-17 13:47:16,651 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource stra 2017-05-17 13:47:16,662 INFO expect on server centosvm: unset resource stra ... OK 2017-05-17 13:47:16,662 INFOCLI status on centosvm: server license_count 2017-05-17 13:47:16,662 INFOCLI centosvm: /opt/pbs/bin/qstat -Bf centosvm 2017-05-17 13:47:16,673 INFO server: centosvm licensed 2017-05-17 13:47:16,686 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/server_priv/comm.lock 2017-05-17 13:47:16,711 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-05-17 13:47:16,722 INFO scheduler centosvm: reverting configuration to defaults 2017-05-17 13:47:16,722 INFO manager on centosvm: list sched 2017-05-17 13:47:16,722 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list sched 2017-05-17 13:47:16,734 INFO manager on centosvm: unset sched ['sched_cycle_length'] 2017-05-17 13:47:16,734 INFOCLI centosvm: /opt/pbs/bin/qmgr -c unset sched sched_cycle_length 2017-05-17 13:47:16,748 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/dedicated_time 2017-05-17 13:47:16,763 INFOCLI2 centosvm: cmp /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-05-17 13:47:16,767 ERROR err: ['cmp: /var/spool/pbs/sched_priv/resource_group: Permission denied'] 2017-05-17 13:47:16,768 INFOCLI2 centosvm: sudo -H /usr/bin/cp -p /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-05-17 13:47:16,782 INFOCLI2 centosvm: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-05-17 13:47:16,792 INFO scheduler centosvm: reverting holidays file to default 2017-05-17 13:47:16,792 INFOCLI2 centosvm: cmp /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-05-17 13:47:16,795 ERROR err: ['cmp: /var/spool/pbs/sched_priv/holidays: Permission denied'] 2017-05-17 13:47:16,795 INFOCLI2 centosvm: sudo -H /usr/bin/cp -p /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-05-17 13:47:16,806 INFOCLI2 centosvm: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/holidays 2017-05-17 13:47:16,817 INFO scheduler centosvm: Parsing holidays file 2017-05-17 13:47:16,817 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2017-05-17 13:47:16,830 INFOCLI2 centosvm: cmp /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-05-17 13:47:16,833 ERROR err: ['cmp: /var/spool/pbs/sched_priv/sched_config: Permission denied'] 2017-05-17 13:47:16,833 INFOCLI2 centosvm: sudo -H /usr/bin/cp -p /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-05-17 13:47:16,844 INFOCLI2 centosvm: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-05-17 13:47:16,858 INFO scheduler centosvm: sent signal -HUP 2017-05-17 13:47:16,858 INFOCLI2 centosvm: sudo -H kill -HUP 84281 2017-05-17 13:47:16,869 INFOCLI2 centosvm: sudo -H /usr/bin/rm /var/spool/pbs/sched_priv/usage 2017-05-17 13:47:16,881 ERROR err: ['/usr/bin/rm: cannot remove \xe2\x80\x98/var/spool/pbs/sched_priv/usage\xe2\x80\x99: No such file or directory'] 2017-05-17 13:47:16,910 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-05-17 13:47:16,921 INFOCLI2 centosvm: sudo -H cmp /opt/pbs/sbin/pbs_mom /opt/pbs/sbin/pbs_mom.cpuset 2017-05-17 13:47:16,946 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:47:16,959 INFO mom centosvm: reverting configuration to defaults 2017-05-17 13:47:16,959 INFOCLI2 centosvm: sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2017-05-17 13:47:16,969 INFOCLI2 centosvm: sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2017-05-17 13:47:16,981 INFOCLI2 centosvm: sudo -H ls /var/spool/pbs/mom_priv/config.d 2017-05-17 13:47:16,994 INFO mom centosvm: sent signal -HUP 2017-05-17 13:47:16,994 INFOCLI2 centosvm: sudo -H kill -HUP 90805 2017-05-17 13:47:17,021 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:47:17,038 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v -a 2017-05-17 13:47:17,050 INFO status on centosvm: node centosvm 2017-05-17 13:47:17,050 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:47:17,062 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:47:17,081 INFO expect on server centosvm: state = free node centosvm ... OK 2017-05-17 13:47:17,082 INFOCLI2 centosvm2: ssh centosvm2 sudo -H cmp /opt/pbs/sbin/pbs_mom /opt/pbs/sbin/pbs_mom.cpuset 2017-05-17 13:47:18,043 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:47:18,530 INFO mom centosvm2: reverting configuration to defaults 2017-05-17 13:47:18,531 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2017-05-17 13:47:18,968 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2017-05-17 13:47:19,377 INFOCLI2 centosvm2: ssh centosvm2 sudo -H ls /var/spool/pbs/mom_priv/config.d 2017-05-17 13:47:19,882 INFO mom centosvm2: sent signal -HUP 2017-05-17 13:47:19,882 INFOCLI2 centosvm2: ssh centosvm2 sudo -H kill -HUP 15795 2017-05-17 13:47:20,906 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:47:21,371 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v -a 2017-05-17 13:47:21,415 INFO status on centosvm: node centosvm2 2017-05-17 13:47:21,416 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm2 2017-05-17 13:47:21,439 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm2 2017-05-17 13:47:21,453 INFO expect on server centosvm: state = free node centosvm2 ... OK 2017-05-17 13:47:21,454 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H cmp /opt/pbs/sbin/pbs_mom /opt/pbs/sbin/pbs_mom.cpuset 2017-05-17 13:47:22,372 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:47:22,787 INFO mom centosvm_nfs: reverting configuration to defaults 2017-05-17 13:47:22,789 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2017-05-17 13:47:23,335 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2017-05-17 13:47:23,935 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H ls /var/spool/pbs/mom_priv/config.d 2017-05-17 13:47:24,309 INFO mom centosvm_nfs: sent signal -HUP 2017-05-17 13:47:24,309 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H kill -HUP 76746 2017-05-17 13:47:25,325 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:47:25,706 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v -a 2017-05-17 13:47:25,721 INFO status on centosvm: node centosvm_nfs 2017-05-17 13:47:25,722 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm_nfs 2017-05-17 13:47:25,739 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm_nfs 2017-05-17 13:47:25,755 INFO expect on server centosvm: state = free node centosvm_nfs ... OK 2017-05-17 13:47:25,756 INFO ========================================= 2017-05-17 13:47:25,756 INFO Completed TestPbsAccumulateRescUsed setUp 2017-05-17 13:47:25,756 INFO ========================================= 2017-05-17 13:47:25,756 INFO len moms = 3 2017-05-17 13:47:25,757 INFOCLI2 centosvm: sudo -H /usr/bin/rm -rf /var/spool/pbs/mom_priv/config.d 2017-05-17 13:47:25,771 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/rm -rf /var/spool/pbs/mom_priv/config.d 2017-05-17 13:47:26,220 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H /usr/bin/rm -rf /var/spool/pbs/mom_priv/config.d 2017-05-17 13:47:26,588 INFO manager on centosvm: delete node 2017-05-17 13:47:26,588 INFOCLI centosvm: /opt/pbs/bin/qmgr -c delete node @default 2017-05-17 13:47:26,604 INFO manager on centosvm: create node centosvm 2017-05-17 13:47:26,605 INFOCLI centosvm: /opt/pbs/bin/qmgr -c create node centosvm 2017-05-17 13:47:26,623 INFO manager on centosvm: create node centosvm2 2017-05-17 13:47:26,624 INFOCLI centosvm: /opt/pbs/bin/qmgr -c create node centosvm2 2017-05-17 13:47:26,640 INFO manager on centosvm: create node centosvm_nfs 2017-05-17 13:47:26,641 INFOCLI centosvm: /opt/pbs/bin/qmgr -c create node centosvm_nfs 2017-05-17 13:47:26,658 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:47:26,670 INFO expect on server centosvm: state = free node centosvm got: state = state-unknown,down 2017-05-17 13:47:27,172 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:47:27,213 INFO expect on server centosvm: state = free node centosvm attempt: 2 got: state = state-unknown,down 2017-05-17 13:47:27,716 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:47:27,758 INFO expect on server centosvm: state = free node centosvm attempt: 3 got: state = state-unknown,down 2017-05-17 13:47:28,261 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:47:28,307 INFO expect on server centosvm: state = free node centosvm attempt: 4 got: state = initializing 2017-05-17 13:47:28,810 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:47:28,850 INFO expect on server centosvm: state = free node centosvm attempt: 5 ... OK 2017-05-17 13:47:28,851 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm2 2017-05-17 13:47:28,887 INFO expect on server centosvm: state = free node centosvm2 ... OK 2017-05-17 13:47:28,887 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm_nfs 2017-05-17 13:47:28,909 INFO expect on server centosvm: state = free node centosvm_nfs ... OK 2017-05-17 13:47:28,910 INFO status on centosvm: hook 2017-05-17 13:47:28,910 INFO manager on centosvm: list hook 2017-05-17 13:47:28,910 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-05-17 13:47:28,947 INFO status on centosvm: hook 2017-05-17 13:47:28,947 INFO manager on centosvm: list hook 2017-05-17 13:47:28,947 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-05-17 13:47:28,969 INFO manager on centosvm: create hook start 2017-05-17 13:47:28,969 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c create hook start 2017-05-17 13:47:28,994 INFO manager on centosvm: set hook start {'enabled': 'True', 'event': 'exechost_startup'} 2017-05-17 13:47:28,994 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c set hook start enabled=True,event=exechost_startup 2017-05-17 13:47:29,027 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook start 2017-05-17 13:47:29,051 INFO expect on server centosvm: enabled set True || event set exechost_startup hook start ... OK 2017-05-17 13:47:29,051 INFO manager on centosvm: import hook start {'content-type': 'application/x-python', 'input-file': '/tmp/PtlPbs0EZ9kV', 'content-encoding': 'default'} 2017-05-17 13:47:29,051 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c import hook start application/x-python default /tmp/PtlPbs0EZ9kV 2017-05-17 13:47:29,075 INFO server centosvm: imported hook body --- import pbs e=pbs.event() localnode=pbs.get_local_nodename() e.vnode_list[localnode].resources_available['foo_i'] = 7 e.vnode_list[localnode].resources_available['foo_f'] = 5.0 e.vnode_list[localnode].resources_available['foo_str'] = "seventyseven" --- 2017-05-17 13:47:29,076 INFO mom centosvm: sent signal -HUP 2017-05-17 13:47:29,076 INFOCLI2 centosvm: sudo -H kill -HUP 90805 2017-05-17 13:47:29,087 INFO mom centosvm2: sent signal -HUP 2017-05-17 13:47:29,087 INFOCLI2 centosvm2: ssh centosvm2 sudo -H kill -HUP 15795 2017-05-17 13:47:29,539 INFO mom centosvm_nfs: sent signal -HUP 2017-05-17 13:47:29,539 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H kill -HUP 76746 2017-05-17 13:47:29,934 INFO manager on centosvm: set server {'job_history_enable': 'True'} 2017-05-17 13:47:29,935 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server job_history_enable=True 2017-05-17 13:47:29,948 INFO manager on centosvm as root: create resource foo_str2 {'flag': 'h', 'type': 'string'} 2017-05-17 13:47:29,949 INFOCLI centosvm: sudo -H -u root /opt/pbs/bin/qmgr -c create resource foo_str2 flag=h,type=string 2017-05-17 13:47:29,989 INFO mom centosvm log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:47:30,357 INFO mom centosvm2 log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:47:30,731 INFO mom centosvm_nfs log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:47:30,732 INFO manager on centosvm as root: create resource foo_str3 {'flag': 'h', 'type': 'string'} 2017-05-17 13:47:30,733 INFOCLI centosvm: sudo -H -u root /opt/pbs/bin/qmgr -c create resource foo_str3 flag=h,type=string 2017-05-17 13:47:30,809 INFO mom centosvm log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:47:31,187 INFO mom centosvm2 log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:47:31,557 INFO mom centosvm_nfs log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:47:31,558 INFO manager on centosvm as root: create resource foo_str4 {'flag': 'h', 'type': 'string'} 2017-05-17 13:47:31,558 INFOCLI centosvm: sudo -H -u root /opt/pbs/bin/qmgr -c create resource foo_str4 flag=h,type=string 2017-05-17 13:47:31,593 INFO mom centosvm log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:47:31,932 INFO mom centosvm2 log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:47:32,342 INFO mom centosvm_nfs log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:47:32,343 INFO manager on centosvm as root: create resource stra {'flag': 'h', 'type': 'string_array'} 2017-05-17 13:47:32,343 INFOCLI centosvm: sudo -H -u root /opt/pbs/bin/qmgr -c create resource stra flag=h,type=string_array 2017-05-17 13:47:32,418 INFO mom centosvm log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:47:32,869 INFO mom centosvm2 log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:47:33,324 INFO mom centosvm_nfs log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:47:33,324 INFO manager on centosvm as root: set resource foo_f {'flag': 'ih'} 2017-05-17 13:47:33,324 INFOCLI centosvm: sudo -H -u root /opt/pbs/bin/qmgr -c set resource foo_f flag=ih 2017-05-17 13:47:33,375 INFO manager on centosvm as root: set resource foo_str {'flag': 'ih'} 2017-05-17 13:47:33,375 INFOCLI centosvm: sudo -H -u root /opt/pbs/bin/qmgr -c set resource foo_str flag=ih 2017-05-17 13:47:33,407 INFO status on centosvm: hook 2017-05-17 13:47:33,408 INFO manager on centosvm: list hook 2017-05-17 13:47:33,408 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-05-17 13:47:33,427 INFO status on centosvm: hook 2017-05-17 13:47:33,428 INFO manager on centosvm: list hook 2017-05-17 13:47:33,428 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-05-17 13:47:33,450 INFO manager on centosvm: create hook epi_invis 2017-05-17 13:47:33,450 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c create hook epi_invis 2017-05-17 13:47:33,472 INFO manager on centosvm: set hook epi_invis {'enabled': 'True', 'event': 'execjob_epilogue'} 2017-05-17 13:47:33,472 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c set hook epi_invis enabled=True,event=execjob_epilogue 2017-05-17 13:47:33,497 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook epi_invis 2017-05-17 13:47:33,517 INFO expect on server centosvm: enabled set True || event set execjob_epilogue hook epi_invis ... OK 2017-05-17 13:47:33,517 INFO manager on centosvm: import hook epi_invis {'content-type': 'application/x-python', 'input-file': '/tmp/PtlPbspOtiSv', 'content-encoding': 'default'} 2017-05-17 13:47:33,518 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c import hook epi_invis application/x-python default /tmp/PtlPbspOtiSv 2017-05-17 13:47:33,539 INFO server centosvm: imported hook body --- import pbs e=pbs.event() j = e.job if j.in_ms_mom(): j.resources_used["foo_f"] = 2.114 j.resources_used["foo_str"] = '{"one":1,"two":2}' else: j.resources_used["foo_f"] = 3.246 j.resources_used["foo_str"] = '{"two":2, "three":3}' --- 2017-05-17 13:47:33,539 INFO job: executable set to /bin/sleep with arguments: 100 2017-05-17 13:47:33,540 INFO job: executable set to /bin/sleep with arguments: 5 2017-05-17 13:47:33,540 INFOCLI centosvm: sudo -H -u pbsuser /opt/pbs/bin/qsub -l place=scatter -l walltime=10 -l select=3:ncpus=1 -- /bin/sleep 5 2017-05-17 13:47:33,555 INFO submit to centosvm as pbsuser: job 14.centosvm OrderedDict([('Resource_List.place', 'scatter'), ('Resource_List.walltime', 10), ('Resource_List.select', '3:ncpus=1')]) 2017-05-17 13:47:33,555 INFO server centosvm: expect offset set to 5 2017-05-17 13:47:38,561 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 14.centosvm 2017-05-17 13:47:39,102 INFO expect on server centosvm: no data for resources_used.foo_f = 8.606 job 14.centosvm 2017-05-17 13:47:39,104 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 14.centosvm 2017-05-17 13:47:39,647 INFO expect on server centosvm: no data for resources_used.foo_f = 8.606 job 14.centosvm attempt: 2 2017-05-17 13:47:39,649 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 14.centosvm 2017-05-17 13:47:40,188 INFO expect on server centosvm: no data for resources_used.foo_f = 8.606 job 14.centosvm attempt: 3 2017-05-17 13:47:40,190 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 14.centosvm 2017-05-17 13:47:40,231 INFO expect on server centosvm: resources_used.foo_f = 8.606 job 14.centosvm attempt: 4 ... OK 2017-05-17 13:47:40,232 INFO status on centosvm: job 14.centosvm resources_used.foo_str 2017-05-17 13:47:40,233 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 14.centosvm 2017-05-17 13:47:40,260 INFO =========================================== 2017-05-17 13:47:40,261 INFO Entered TestPbsAccumulateRescUsed tearDown 2017-05-17 13:47:40,261 INFO =========================================== 2017-05-17 13:47:40,261 INFO ============================================ 2017-05-17 13:47:40,261 INFO Completed TestPbsAccumulateRescUsed tearDown 2017-05-17 13:47:40,261 INFO ============================================ 2017-05-17 13:47:40,262 INFO ok 2017-05-17 13:47:40,263 INFO test name: test_server_restart (tests.functional.pbs_accumulate_resc_used.TestPbsAccumulateRescUsed)... 2017-05-17 13:47:40,263 INFO test start time: Wed May 17 13:47:40 2017 2017-05-17 13:47:40,263 INFO test docstring: Test that resource accumulation will not get impacted if server is restarted during job execution 2017-05-17 13:47:40,263 INFO ======================================== 2017-05-17 13:47:40,264 INFO Entered TestPbsAccumulateRescUsed setUp 2017-05-17 13:47:40,264 INFO ======================================== 2017-05-17 13:47:40,264 INFOCLI centosvm: /opt/pbs/bin/qstat -Bf centosvm 2017-05-17 13:47:40,284 INFO status on centosvm: server 2017-05-17 13:47:40,284 INFOCLI centosvm: /opt/pbs/bin/qstat -Bf centosvm 2017-05-17 13:47:40,296 INFO manager on centosvm: set server {'managers': (3, 'dilipk@*')} 2017-05-17 13:47:40,296 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c set server managers-=dilipk@* 2017-05-17 13:47:40,319 INFO manager on centosvm: set server {'managers': (2, 'dilipk@*')} 2017-05-17 13:47:40,320 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c set server managers+=dilipk@* 2017-05-17 13:47:40,345 INFO server centosvm: reverting configuration to defaults 2017-05-17 13:47:40,345 INFOCLI centosvm: /opt/pbs/bin/qstat -Bf centosvm 2017-05-17 13:47:40,355 INFO select on centosvm: __ALL__ 2017-05-17 13:47:40,356 INFOCLI centosvm: /opt/pbs/bin/qselect -x 2017-05-17 13:47:40,370 INFO delete job on centosvm: 14.centosvm 2017-05-17 13:47:40,371 INFOCLI centosvm: /opt/pbs/bin/qdel -W force -x 14.centosvm 2017-05-17 13:47:40,385 INFOCLI centosvm: /opt/pbs/bin/qstat -f 14.centosvm 2017-05-17 13:47:40,398 INFOCLI centosvm: /opt/pbs/bin/qstat -f @centosvm 2017-05-17 13:47:40,410 INFO expect on server centosvm: job_state set 0 job ... OK 2017-05-17 13:47:40,410 INFOCLI centosvm: /opt/pbs/bin/pbs_rstat -f 2017-05-17 13:47:40,421 INFO manager on centosvm: unset server ['job_history_enable'] 2017-05-17 13:47:40,421 INFOCLI centosvm: /opt/pbs/bin/qmgr -c unset server job_history_enable 2017-05-17 13:47:40,435 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-05-17 13:47:40,456 INFO manager on centosvm: delete hook start,epi_invis 2017-05-17 13:47:40,456 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c delete hook start,epi_invis 2017-05-17 13:47:40,483 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook start 2017-05-17 13:47:40,510 INFO expect on server centosvm: unset hook start ... OK 2017-05-17 13:47:40,511 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook epi_invis 2017-05-17 13:47:40,542 INFO expect on server centosvm: unset hook epi_invis ... OK 2017-05-17 13:47:40,543 INFOCLI centosvm: /opt/pbs/bin/qstat -Qf @centosvm 2017-05-17 13:47:40,557 INFO manager on centosvm: delete queue workq 2017-05-17 13:47:40,557 INFOCLI centosvm: /opt/pbs/bin/qmgr -c delete queue workq 2017-05-17 13:47:40,579 INFOCLI centosvm: /opt/pbs/bin/qstat -Qf workq@centosvm 2017-05-17 13:47:40,593 INFO expect on server centosvm: unset queue workq ... OK 2017-05-17 13:47:40,593 INFO manager on centosvm: create queue workq {'started': 'True', 'queue_type': 'Execution', 'enabled': 'True'} 2017-05-17 13:47:40,593 INFOCLI centosvm: /opt/pbs/bin/qmgr -c create queue workq started=True,queue_type=Execution,enabled=True 2017-05-17 13:47:40,609 INFOCLI centosvm: /opt/pbs/bin/qstat -Qf workq@centosvm 2017-05-17 13:47:40,623 INFO expect on server centosvm: started set True || queue_type set Execution || enabled set True queue workq ... OK 2017-05-17 13:47:40,623 INFO manager on centosvm: set server {'default_queue': 'workq'} 2017-05-17 13:47:40,624 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server default_queue=workq 2017-05-17 13:47:40,638 INFO status on centosvm: resource 2017-05-17 13:47:40,638 INFO manager on centosvm: list resource 2017-05-17 13:47:40,638 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource 2017-05-17 13:47:40,652 INFO manager on centosvm: delete resource foo_f,foo_i,foo_str2,foo_str3,foo_str4,foo_str,stra 2017-05-17 13:47:40,652 INFOCLI centosvm: /opt/pbs/bin/qmgr -c delete resource foo_f,foo_i,foo_str2,foo_str3,foo_str4,foo_str,stra 2017-05-17 13:47:40,825 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_f 2017-05-17 13:47:40,835 INFO expect on server centosvm: unset resource foo_f ... OK 2017-05-17 13:47:40,836 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_i 2017-05-17 13:47:40,846 INFO expect on server centosvm: unset resource foo_i ... OK 2017-05-17 13:47:40,846 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_str2 2017-05-17 13:47:40,857 INFO expect on server centosvm: unset resource foo_str2 ... OK 2017-05-17 13:47:40,857 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_str3 2017-05-17 13:47:40,868 INFO expect on server centosvm: unset resource foo_str3 ... OK 2017-05-17 13:47:40,868 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_str4 2017-05-17 13:47:40,878 INFO expect on server centosvm: unset resource foo_str4 ... OK 2017-05-17 13:47:40,879 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_str 2017-05-17 13:47:40,890 INFO expect on server centosvm: unset resource foo_str ... OK 2017-05-17 13:47:40,890 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource stra 2017-05-17 13:47:40,901 INFO expect on server centosvm: unset resource stra ... OK 2017-05-17 13:47:40,901 INFOCLI status on centosvm: server license_count 2017-05-17 13:47:40,902 INFOCLI centosvm: /opt/pbs/bin/qstat -Bf centosvm 2017-05-17 13:47:40,913 INFO server: centosvm licensed 2017-05-17 13:47:40,926 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/server_priv/comm.lock 2017-05-17 13:47:40,953 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-05-17 13:47:40,963 INFO scheduler centosvm: reverting configuration to defaults 2017-05-17 13:47:40,963 INFO manager on centosvm: list sched 2017-05-17 13:47:40,963 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list sched 2017-05-17 13:47:40,976 INFO manager on centosvm: unset sched ['sched_cycle_length'] 2017-05-17 13:47:40,977 INFOCLI centosvm: /opt/pbs/bin/qmgr -c unset sched sched_cycle_length 2017-05-17 13:47:40,989 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/dedicated_time 2017-05-17 13:47:41,000 INFOCLI2 centosvm: cmp /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-05-17 13:47:41,003 ERROR err: ['cmp: /var/spool/pbs/sched_priv/resource_group: Permission denied'] 2017-05-17 13:47:41,003 INFOCLI2 centosvm: sudo -H /usr/bin/cp -p /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-05-17 13:47:41,018 INFOCLI2 centosvm: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-05-17 13:47:41,029 INFO scheduler centosvm: reverting holidays file to default 2017-05-17 13:47:41,030 INFOCLI2 centosvm: cmp /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-05-17 13:47:41,033 ERROR err: ['cmp: /var/spool/pbs/sched_priv/holidays: Permission denied'] 2017-05-17 13:47:41,034 INFOCLI2 centosvm: sudo -H /usr/bin/cp -p /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-05-17 13:47:41,045 INFOCLI2 centosvm: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/holidays 2017-05-17 13:47:41,055 INFO scheduler centosvm: Parsing holidays file 2017-05-17 13:47:41,055 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2017-05-17 13:47:41,068 INFOCLI2 centosvm: cmp /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-05-17 13:47:41,070 ERROR err: ['cmp: /var/spool/pbs/sched_priv/sched_config: Permission denied'] 2017-05-17 13:47:41,071 INFOCLI2 centosvm: sudo -H /usr/bin/cp -p /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-05-17 13:47:41,081 INFOCLI2 centosvm: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-05-17 13:47:41,091 INFO scheduler centosvm: sent signal -HUP 2017-05-17 13:47:41,091 INFOCLI2 centosvm: sudo -H kill -HUP 84281 2017-05-17 13:47:41,103 INFOCLI2 centosvm: sudo -H /usr/bin/rm /var/spool/pbs/sched_priv/usage 2017-05-17 13:47:41,114 ERROR err: ['/usr/bin/rm: cannot remove \xe2\x80\x98/var/spool/pbs/sched_priv/usage\xe2\x80\x99: No such file or directory'] 2017-05-17 13:47:41,142 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-05-17 13:47:41,154 INFOCLI2 centosvm: sudo -H cmp /opt/pbs/sbin/pbs_mom /opt/pbs/sbin/pbs_mom.cpuset 2017-05-17 13:47:41,180 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:47:41,191 INFO mom centosvm: reverting configuration to defaults 2017-05-17 13:47:41,191 INFOCLI2 centosvm: sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2017-05-17 13:47:41,202 INFOCLI2 centosvm: sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2017-05-17 13:47:41,214 INFOCLI2 centosvm: sudo -H ls /var/spool/pbs/mom_priv/config.d 2017-05-17 13:47:41,225 INFO mom centosvm: sent signal -HUP 2017-05-17 13:47:41,225 INFOCLI2 centosvm: sudo -H kill -HUP 90805 2017-05-17 13:47:41,254 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:47:41,265 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v -a 2017-05-17 13:47:41,292 INFO status on centosvm: node centosvm 2017-05-17 13:47:41,293 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:47:41,305 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:47:41,318 INFO expect on server centosvm: state = free node centosvm ... OK 2017-05-17 13:47:41,318 INFOCLI2 centosvm2: ssh centosvm2 sudo -H cmp /opt/pbs/sbin/pbs_mom /opt/pbs/sbin/pbs_mom.cpuset 2017-05-17 13:47:42,267 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:47:42,646 INFO mom centosvm2: reverting configuration to defaults 2017-05-17 13:47:42,646 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2017-05-17 13:47:43,034 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2017-05-17 13:47:43,453 INFOCLI2 centosvm2: ssh centosvm2 sudo -H ls /var/spool/pbs/mom_priv/config.d 2017-05-17 13:47:43,856 INFO mom centosvm2: sent signal -HUP 2017-05-17 13:47:43,856 INFOCLI2 centosvm2: ssh centosvm2 sudo -H kill -HUP 15795 2017-05-17 13:47:44,790 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:47:45,411 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v -a 2017-05-17 13:47:45,429 INFO status on centosvm: node centosvm2 2017-05-17 13:47:45,429 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm2 2017-05-17 13:47:45,453 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm2 2017-05-17 13:47:45,469 INFO expect on server centosvm: state = free node centosvm2 ... OK 2017-05-17 13:47:45,470 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H cmp /opt/pbs/sbin/pbs_mom /opt/pbs/sbin/pbs_mom.cpuset 2017-05-17 13:47:46,445 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:47:46,858 INFO mom centosvm_nfs: reverting configuration to defaults 2017-05-17 13:47:46,858 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2017-05-17 13:47:47,312 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2017-05-17 13:47:47,782 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H ls /var/spool/pbs/mom_priv/config.d 2017-05-17 13:47:48,235 INFO mom centosvm_nfs: sent signal -HUP 2017-05-17 13:47:48,236 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H kill -HUP 76746 2017-05-17 13:47:49,079 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:47:49,697 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v -a 2017-05-17 13:47:49,715 INFO status on centosvm: node centosvm_nfs 2017-05-17 13:47:49,716 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm_nfs 2017-05-17 13:47:49,736 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm_nfs 2017-05-17 13:47:49,754 INFO expect on server centosvm: state = free node centosvm_nfs ... OK 2017-05-17 13:47:49,754 INFO ========================================= 2017-05-17 13:47:49,755 INFO Completed TestPbsAccumulateRescUsed setUp 2017-05-17 13:47:49,755 INFO ========================================= 2017-05-17 13:47:49,755 INFO len moms = 3 2017-05-17 13:47:49,755 INFOCLI2 centosvm: sudo -H /usr/bin/rm -rf /var/spool/pbs/mom_priv/config.d 2017-05-17 13:47:49,774 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/rm -rf /var/spool/pbs/mom_priv/config.d 2017-05-17 13:47:50,174 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H /usr/bin/rm -rf /var/spool/pbs/mom_priv/config.d 2017-05-17 13:47:50,597 INFO manager on centosvm: delete node 2017-05-17 13:47:50,598 INFOCLI centosvm: /opt/pbs/bin/qmgr -c delete node @default 2017-05-17 13:47:50,624 INFO manager on centosvm: create node centosvm 2017-05-17 13:47:50,624 INFOCLI centosvm: /opt/pbs/bin/qmgr -c create node centosvm 2017-05-17 13:47:50,640 INFO manager on centosvm: create node centosvm2 2017-05-17 13:47:50,640 INFOCLI centosvm: /opt/pbs/bin/qmgr -c create node centosvm2 2017-05-17 13:47:50,654 INFO manager on centosvm: create node centosvm_nfs 2017-05-17 13:47:50,654 INFOCLI centosvm: /opt/pbs/bin/qmgr -c create node centosvm_nfs 2017-05-17 13:47:50,670 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:47:50,681 INFO expect on server centosvm: state = free node centosvm got: state = state-unknown,down 2017-05-17 13:47:51,183 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:47:51,221 INFO expect on server centosvm: state = free node centosvm attempt: 2 got: state = state-unknown,down 2017-05-17 13:47:51,723 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:47:51,761 INFO expect on server centosvm: state = free node centosvm attempt: 3 got: state = state-unknown,down 2017-05-17 13:47:52,263 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:47:52,342 INFO expect on server centosvm: state = free node centosvm attempt: 4 ... OK 2017-05-17 13:47:52,343 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm2 2017-05-17 13:47:52,371 INFO expect on server centosvm: state = free node centosvm2 ... OK 2017-05-17 13:47:52,371 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm_nfs 2017-05-17 13:47:52,383 INFO expect on server centosvm: state = free node centosvm_nfs ... OK 2017-05-17 13:47:52,383 INFO status on centosvm: hook 2017-05-17 13:47:52,383 INFO manager on centosvm: list hook 2017-05-17 13:47:52,383 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-05-17 13:47:52,408 INFO status on centosvm: hook 2017-05-17 13:47:52,408 INFO manager on centosvm: list hook 2017-05-17 13:47:52,409 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-05-17 13:47:52,431 INFO manager on centosvm: create hook start 2017-05-17 13:47:52,431 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c create hook start 2017-05-17 13:47:52,453 INFO manager on centosvm: set hook start {'enabled': 'True', 'event': 'exechost_startup'} 2017-05-17 13:47:52,454 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c set hook start enabled=True,event=exechost_startup 2017-05-17 13:47:52,476 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook start 2017-05-17 13:47:52,496 INFO expect on server centosvm: enabled set True || event set exechost_startup hook start ... OK 2017-05-17 13:47:52,496 INFO manager on centosvm: import hook start {'content-type': 'application/x-python', 'input-file': '/tmp/PtlPbsCNi1t2', 'content-encoding': 'default'} 2017-05-17 13:47:52,497 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c import hook start application/x-python default /tmp/PtlPbsCNi1t2 2017-05-17 13:47:52,521 INFO server centosvm: imported hook body --- import pbs e=pbs.event() localnode=pbs.get_local_nodename() e.vnode_list[localnode].resources_available['foo_i'] = 7 e.vnode_list[localnode].resources_available['foo_f'] = 5.0 e.vnode_list[localnode].resources_available['foo_str'] = "seventyseven" --- 2017-05-17 13:47:52,522 INFO mom centosvm: sent signal -HUP 2017-05-17 13:47:52,522 INFOCLI2 centosvm: sudo -H kill -HUP 90805 2017-05-17 13:47:52,533 INFO mom centosvm2: sent signal -HUP 2017-05-17 13:47:52,534 INFOCLI2 centosvm2: ssh centosvm2 sudo -H kill -HUP 15795 2017-05-17 13:47:52,894 INFO mom centosvm_nfs: sent signal -HUP 2017-05-17 13:47:52,895 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H kill -HUP 76746 2017-05-17 13:47:53,327 INFO manager on centosvm: set server {'job_history_enable': 'True'} 2017-05-17 13:47:53,328 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server job_history_enable=True 2017-05-17 13:47:53,375 INFO manager on centosvm as root: create resource foo_str2 {'flag': 'h', 'type': 'string'} 2017-05-17 13:47:53,376 INFOCLI centosvm: sudo -H -u root /opt/pbs/bin/qmgr -c create resource foo_str2 flag=h,type=string 2017-05-17 13:47:53,428 INFO mom centosvm log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:47:53,956 INFO mom centosvm2 log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:47:54,476 INFO mom centosvm_nfs log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:47:54,477 INFO manager on centosvm as root: create resource foo_str3 {'flag': 'h', 'type': 'string'} 2017-05-17 13:47:54,477 INFOCLI centosvm: sudo -H -u root /opt/pbs/bin/qmgr -c create resource foo_str3 flag=h,type=string 2017-05-17 13:47:54,560 INFO mom centosvm log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:47:54,941 INFO mom centosvm2 log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:47:55,471 INFO mom centosvm_nfs log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:47:55,472 INFO manager on centosvm as root: create resource foo_str4 {'flag': 'h', 'type': 'string'} 2017-05-17 13:47:55,472 INFOCLI centosvm: sudo -H -u root /opt/pbs/bin/qmgr -c create resource foo_str4 flag=h,type=string 2017-05-17 13:47:55,563 INFO mom centosvm log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:47:55,947 INFO mom centosvm2 log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:47:56,400 INFO mom centosvm_nfs log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:47:56,401 INFO manager on centosvm as root: create resource stra {'flag': 'h', 'type': 'string_array'} 2017-05-17 13:47:56,401 INFOCLI centosvm: sudo -H -u root /opt/pbs/bin/qmgr -c create resource stra flag=h,type=string_array 2017-05-17 13:47:56,491 INFO mom centosvm log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:47:56,845 INFO mom centosvm2 log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:47:57,222 INFO mom centosvm_nfs log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:47:57,223 INFO status on centosvm: hook 2017-05-17 13:47:57,223 INFO manager on centosvm: list hook 2017-05-17 13:47:57,224 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-05-17 13:47:57,286 INFO status on centosvm: hook 2017-05-17 13:47:57,287 INFO manager on centosvm: list hook 2017-05-17 13:47:57,287 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-05-17 13:47:57,310 INFO manager on centosvm: create hook prolo 2017-05-17 13:47:57,310 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c create hook prolo 2017-05-17 13:47:57,337 INFO manager on centosvm: set hook prolo {'enabled': 'True', 'event': 'execjob_prologue'} 2017-05-17 13:47:57,337 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c set hook prolo enabled=True,event=execjob_prologue 2017-05-17 13:47:57,358 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook prolo 2017-05-17 13:47:57,385 INFO expect on server centosvm: enabled set True || event set execjob_prologue hook prolo ... OK 2017-05-17 13:47:57,386 INFO manager on centosvm: import hook prolo {'content-type': 'application/x-python', 'input-file': '/tmp/PtlPbs9wMjdl', 'content-encoding': 'default'} 2017-05-17 13:47:57,386 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c import hook prolo application/x-python default /tmp/PtlPbs9wMjdl 2017-05-17 13:47:57,408 INFO server centosvm: imported hook body --- import pbs e=pbs.event() pbs.logmsg(pbs.LOG_DEBUG, "executed prologue hook") if e.job.in_ms_mom(): e.job.resources_used["vmem"] = pbs.size("11gb") e.job.resources_used["foo_i"] = 11 e.job.resources_used["foo_f"] = 0.11 e.job.resources_used["foo_str"] = '{"seven":7}' e.job.resources_used["cput"] = 11 e.job.resources_used["stra"] = '"glad,elated","happy"' e.job.resources_used["foo_str4"] = "eight" else: e.job.resources_used["vmem"] = pbs.size("12gb") e.job.resources_used["foo_i"] = 12 e.job.resources_used["foo_f"] = 0.12 e.job.resources_used["foo_str"] = '{"eight":8,"nine":9}' e.job.resources_used["cput"] = 12 e.job.resources_used["stra"] = '"cucumbers,bananas"' --- 2017-05-17 13:47:57,409 INFO job: executable set to /bin/sleep with arguments: 100 2017-05-17 13:47:57,409 INFOCLI2 centosvm: sudo -H /usr/bin/chmod 0755 /tmp/PtlPbsJobScriptadqI6P 2017-05-17 13:47:57,421 INFOCLI centosvm: sudo -H -u pbsuser /opt/pbs/bin/qsub -l place=scatter -l walltime=20 -l select=3:ncpus=1 /tmp/PtlPbsJobScriptadqI6P 2017-05-17 13:47:57,434 INFO submit to centosvm as pbsuser: job 15.centosvm OrderedDict([('Resource_List.place', 'scatter'), ('Resource_List.walltime', 20), ('Resource_List.select', '3:ncpus=1')]) 2017-05-17 13:47:57,434 INFOCLI job script /tmp/PtlPbsJobScriptadqI6P --- pbsdsh -n 1 hostname pbsdsh -n 2 hostname sleep 10 --- 2017-05-17 13:47:57,434 INFOCLI centosvm: /opt/pbs/bin/qstat -f 15.centosvm 2017-05-17 13:47:57,452 INFO expect on server centosvm: job_state = R || substate = 42 job 15.centosvm got: job_state = Q 2017-05-17 13:47:57,953 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:47:58,002 INFOCLI centosvm: /opt/pbs/bin/qstat -f 15.centosvm 2017-05-17 13:47:58,043 INFO expect on server centosvm: job_state = R || substate = 42 job 15.centosvm attempt: 2 got: substate = 41 2017-05-17 13:47:58,544 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:47:58,588 INFOCLI centosvm: /opt/pbs/bin/qstat -f 15.centosvm 2017-05-17 13:47:58,613 INFO expect on server centosvm: job_state = R || substate = 42 job 15.centosvm attempt: 3 got: substate = 41 2017-05-17 13:47:59,114 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:47:59,146 INFOCLI centosvm: /opt/pbs/bin/qstat -f 15.centosvm 2017-05-17 13:47:59,175 INFO expect on server centosvm: job_state = R || substate = 42 job 15.centosvm attempt: 4 ... OK 2017-05-17 13:47:59,175 INFOCLI centosvm: /opt/pbs/bin/qstat -Bf centosvm 2017-05-17 13:47:59,194 INFO server centosvm: stopping Server on host centosvm 2017-05-17 13:47:59,194 INFOCLI2 centosvm: sudo -H kill -TERM 90652 2017-05-17 13:47:59,210 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/server_priv/server.lock 2017-05-17 13:48:04,534 INFO service: starting pbs_server 2017-05-17 13:48:04,534 INFOCLI centosvm: sudo -H /opt/pbs/sbin/pbs_server 2017-05-17 13:48:10,391 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/server_priv/server.lock 2017-05-17 13:48:10,483 INFOCLI centosvm: /opt/pbs/bin/qstat -Bf centosvm 2017-05-17 13:48:10,497 INFO server centosvm: expect offset set to 5 2017-05-17 13:48:15,503 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 15.centosvm 2017-05-17 13:48:15,550 INFO expect on server centosvm: resources_used.stra = "glad,elated","happy" || resources_used.foo_str4 = eight || resources_used.foo_i = 35 || job_state = F || resources_used.foo_f = 0.35 || resources_used.vmem = 35gb || resources_used.cput = 00:00:35 job 15.centosvm got: resources_used.foo_i = 11 2017-05-17 13:48:16,553 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:48:16,606 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 15.centosvm 2017-05-17 13:48:16,629 INFO expect on server centosvm: resources_used.stra = "glad,elated","happy" || resources_used.foo_str4 = eight || resources_used.foo_i = 35 || job_state = F || resources_used.foo_f = 0.35 || resources_used.vmem = 35gb || resources_used.cput = 00:00:35 job 15.centosvm attempt: 2 got: resources_used.foo_i = 11 2017-05-17 13:48:17,631 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:48:17,678 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 15.centosvm 2017-05-17 13:48:17,709 INFO expect on server centosvm: resources_used.stra = "glad,elated","happy" || resources_used.foo_str4 = eight || resources_used.foo_i = 35 || job_state = F || resources_used.foo_f = 0.35 || resources_used.vmem = 35gb || resources_used.cput = 00:00:35 job 15.centosvm attempt: 3 got: resources_used.foo_i = 11 2017-05-17 13:48:18,711 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:48:18,760 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 15.centosvm 2017-05-17 13:48:18,791 INFO expect on server centosvm: resources_used.stra = "glad,elated","happy" || resources_used.foo_str4 = eight || resources_used.foo_i = 35 || job_state = F || resources_used.foo_f = 0.35 || resources_used.vmem = 35gb || resources_used.cput = 00:00:35 job 15.centosvm attempt: 4 got: resources_used.foo_i = 11 2017-05-17 13:48:19,795 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:48:19,811 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 15.centosvm 2017-05-17 13:48:19,824 INFO expect on server centosvm: resources_used.stra = "glad,elated","happy" || resources_used.foo_str4 = eight || resources_used.foo_i = 35 || job_state = F || resources_used.foo_f = 0.35 || resources_used.vmem = 35gb || resources_used.cput = 00:00:35 job 15.centosvm attempt: 5 got: resources_used.foo_i = 11 2017-05-17 13:48:20,828 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:48:20,876 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 15.centosvm 2017-05-17 13:48:20,919 INFO expect on server centosvm: resources_used.stra = "glad,elated","happy" || resources_used.foo_str4 = eight || resources_used.foo_i = 35 || job_state = F || resources_used.foo_f = 0.35 || resources_used.vmem = 35gb || resources_used.cput = 00:00:35 job 15.centosvm attempt: 6 got: resources_used.foo_i = 11 2017-05-17 13:48:21,923 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:48:21,937 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 15.centosvm 2017-05-17 13:48:21,952 INFO expect on server centosvm: resources_used.stra = "glad,elated","happy" || resources_used.foo_str4 = eight || resources_used.foo_i = 35 || job_state = F || resources_used.foo_f = 0.35 || resources_used.vmem = 35gb || resources_used.cput = 00:00:35 job 15.centosvm attempt: 7 got: resources_used.foo_i = 11 2017-05-17 13:48:22,954 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:48:22,969 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 15.centosvm 2017-05-17 13:48:22,982 INFO expect on server centosvm: resources_used.stra = "glad,elated","happy" || resources_used.foo_str4 = eight || resources_used.foo_i = 35 || job_state = F || resources_used.foo_f = 0.35 || resources_used.vmem = 35gb || resources_used.cput = 00:00:35 job 15.centosvm attempt: 8 got: resources_used.foo_i = 11 2017-05-17 13:48:23,984 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:48:24,013 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 15.centosvm 2017-05-17 13:48:24,039 INFO expect on server centosvm: resources_used.stra = "glad,elated","happy" || resources_used.foo_str4 = eight || resources_used.foo_i = 35 || job_state = F || resources_used.foo_f = 0.35 || resources_used.vmem = 35gb || resources_used.cput = 00:00:35 job 15.centosvm attempt: 9 got: resources_used.foo_i = 11 2017-05-17 13:48:25,041 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:48:25,092 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 15.centosvm 2017-05-17 13:48:25,130 INFO expect on server centosvm: resources_used.stra = "glad,elated","happy" || resources_used.foo_str4 = eight || resources_used.foo_i = 35 || job_state = F || resources_used.foo_f = 0.35 || resources_used.vmem = 35gb || resources_used.cput = 00:00:35 job 15.centosvm attempt: 10 got: resources_used.foo_i = 11 2017-05-17 13:48:26,133 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:48:26,179 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 15.centosvm 2017-05-17 13:48:26,213 INFO expect on server centosvm: resources_used.stra = "glad,elated","happy" || resources_used.foo_str4 = eight || resources_used.foo_i = 35 || job_state = F || resources_used.foo_f = 0.35 || resources_used.vmem = 35gb || resources_used.cput = 00:00:35 job 15.centosvm attempt: 11 got: resources_used.foo_i = 11 2017-05-17 13:48:27,216 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:48:27,261 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 15.centosvm 2017-05-17 13:48:27,288 INFO expect on server centosvm: resources_used.stra = "glad,elated","happy" || resources_used.foo_str4 = eight || resources_used.foo_i = 35 || job_state = F || resources_used.foo_f = 0.35 || resources_used.vmem = 35gb || resources_used.cput = 00:00:35 job 15.centosvm attempt: 12 got: resources_used.foo_i = 11 2017-05-17 13:48:28,291 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:48:28,338 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 15.centosvm 2017-05-17 13:48:28,373 INFO expect on server centosvm: resources_used.stra = "glad,elated","happy" || resources_used.foo_str4 = eight || resources_used.foo_i = 35 || job_state = F || resources_used.foo_f = 0.35 || resources_used.vmem = 35gb || resources_used.cput = 00:00:35 job 15.centosvm attempt: 13 got: resources_used.foo_i = 11 2017-05-17 13:48:29,375 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:48:29,426 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 15.centosvm 2017-05-17 13:48:29,458 INFO expect on server centosvm: resources_used.stra = "glad,elated","happy" || resources_used.foo_str4 = eight || resources_used.foo_i = 35 || job_state = F || resources_used.foo_f = 0.35 || resources_used.vmem = 35gb || resources_used.cput = 00:00:35 job 15.centosvm attempt: 14 got: resources_used.foo_i = 11 2017-05-17 13:48:30,461 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:48:30,508 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 15.centosvm 2017-05-17 13:48:30,538 INFO expect on server centosvm: resources_used.stra = "glad,elated","happy" || resources_used.foo_str4 = eight || resources_used.foo_i = 35 || job_state = F || resources_used.foo_f = 0.35 || resources_used.vmem = 35gb || resources_used.cput = 00:00:35 job 15.centosvm attempt: 15 got: resources_used.foo_i = 11 2017-05-17 13:48:31,541 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:48:31,567 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 15.centosvm 2017-05-17 13:48:31,585 INFO expect on server centosvm: resources_used.stra = "glad,elated","happy" || resources_used.foo_str4 = eight || resources_used.foo_i = 35 || job_state = F || resources_used.foo_f = 0.35 || resources_used.vmem = 35gb || resources_used.cput = 00:00:35 job 15.centosvm attempt: 16 got: resources_used.foo_i = 11 2017-05-17 13:48:32,588 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:48:32,632 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 15.centosvm 2017-05-17 13:48:32,661 INFO expect on server centosvm: resources_used.stra = "glad,elated","happy" || resources_used.foo_str4 = eight || resources_used.foo_i = 35 || job_state = F || resources_used.foo_f = 0.35 || resources_used.vmem = 35gb || resources_used.cput = 00:00:35 job 15.centosvm attempt: 17 got: resources_used.foo_i = 11 2017-05-17 13:48:33,664 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:48:33,712 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 15.centosvm 2017-05-17 13:48:33,752 INFO expect on server centosvm: resources_used.stra = "glad,elated","happy" || resources_used.foo_str4 = eight || resources_used.foo_i = 35 || job_state = F || resources_used.foo_f = 0.35 || resources_used.vmem = 35gb || resources_used.cput = 00:00:35 job 15.centosvm attempt: 18 got: resources_used.foo_i = 11 2017-05-17 13:48:34,754 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:48:34,797 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 15.centosvm 2017-05-17 13:48:34,824 INFO expect on server centosvm: resources_used.stra = "glad,elated","happy" || resources_used.foo_str4 = eight || resources_used.foo_i = 35 || job_state = F || resources_used.foo_f = 0.35 || resources_used.vmem = 35gb || resources_used.cput = 00:00:35 job 15.centosvm attempt: 19 got: resources_used.foo_i = 11 2017-05-17 13:48:35,827 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:48:35,873 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 15.centosvm 2017-05-17 13:48:35,906 INFO expect on server centosvm: resources_used.stra = "glad,elated","happy" || resources_used.foo_str4 = eight || resources_used.foo_i = 35 || job_state = F || resources_used.foo_f = 0.35 || resources_used.vmem = 35gb || resources_used.cput = 00:00:35 job 15.centosvm attempt: 20 got: resources_used.foo_i = 11 2017-05-17 13:48:36,907 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:48:36,940 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 15.centosvm 2017-05-17 13:48:36,962 INFO expect on server centosvm: resources_used.stra = "glad,elated","happy" || resources_used.foo_str4 = eight || resources_used.foo_i = 35 || job_state = F || resources_used.foo_f = 0.35 || resources_used.vmem = 35gb || resources_used.cput = 00:00:35 job 15.centosvm attempt: 21 got: resources_used.foo_i = 11 2017-05-17 13:48:37,964 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:48:37,980 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 15.centosvm 2017-05-17 13:48:37,992 INFO expect on server centosvm: resources_used.stra = "glad,elated","happy" || resources_used.foo_str4 = eight || resources_used.foo_i = 35 || job_state = F || resources_used.foo_f = 0.35 || resources_used.vmem = 35gb || resources_used.cput = 00:00:35 job 15.centosvm attempt: 22 got: resources_used.foo_i = 11 2017-05-17 13:48:38,996 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:48:39,050 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 15.centosvm 2017-05-17 13:48:39,097 INFO expect on server centosvm: resources_used.stra = "glad,elated","happy" || resources_used.foo_str4 = eight || resources_used.foo_i = 35 || job_state = F || resources_used.foo_f = 0.35 || resources_used.vmem = 35gb || resources_used.cput = 00:00:35 job 15.centosvm attempt: 23 got: resources_used.foo_i = 11 2017-05-17 13:48:40,100 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:48:40,150 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 15.centosvm 2017-05-17 13:48:40,188 INFO expect on server centosvm: resources_used.stra = "glad,elated","happy" || resources_used.foo_str4 = eight || resources_used.foo_i = 35 || job_state = F || resources_used.foo_f = 0.35 || resources_used.vmem = 35gb || resources_used.cput = 00:00:35 job 15.centosvm attempt: 24 got: resources_used.foo_i = 11 2017-05-17 13:48:41,191 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:48:41,241 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 15.centosvm 2017-05-17 13:48:41,289 INFO expect on server centosvm: resources_used.stra = "glad,elated","happy" || resources_used.foo_str4 = eight || resources_used.foo_i = 35 || job_state = F || resources_used.foo_f = 0.35 || resources_used.vmem = 35gb || resources_used.cput = 00:00:35 job 15.centosvm attempt: 25 got: resources_used.foo_i = 11 2017-05-17 13:48:42,291 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:48:42,339 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 15.centosvm 2017-05-17 13:48:42,375 INFO expect on server centosvm: resources_used.stra = "glad,elated","happy" || resources_used.foo_str4 = eight || resources_used.foo_i = 35 || job_state = F || resources_used.foo_f = 0.35 || resources_used.vmem = 35gb || resources_used.cput = 00:00:35 job 15.centosvm attempt: 26 got: resources_used.foo_i = 11 2017-05-17 13:48:43,378 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:48:43,423 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 15.centosvm 2017-05-17 13:48:43,469 INFO expect on server centosvm: resources_used.stra = "glad,elated","happy" || resources_used.foo_str4 = eight || resources_used.foo_i = 35 || job_state = F || resources_used.foo_f = 0.35 || resources_used.vmem = 35gb || resources_used.cput = 00:00:35 job 15.centosvm attempt: 27 got: resources_used.foo_i = 11 2017-05-17 13:48:44,473 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:48:44,518 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 15.centosvm 2017-05-17 13:48:44,555 INFO expect on server centosvm: resources_used.stra = "glad,elated","happy" || resources_used.foo_str4 = eight || resources_used.foo_i = 35 || job_state = F || resources_used.foo_f = 0.35 || resources_used.vmem = 35gb || resources_used.cput = 00:00:35 job 15.centosvm attempt: 28 got: resources_used.foo_i = 11 2017-05-17 13:48:45,559 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:48:45,607 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 15.centosvm 2017-05-17 13:48:45,645 INFO expect on server centosvm: resources_used.stra = "glad,elated","happy" || resources_used.foo_str4 = eight || resources_used.foo_i = 35 || job_state = F || resources_used.foo_f = 0.35 || resources_used.vmem = 35gb || resources_used.cput = 00:00:35 job 15.centosvm attempt: 29 got: resources_used.foo_i = 11 2017-05-17 13:48:46,650 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:48:46,699 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 15.centosvm 2017-05-17 13:48:46,726 INFO expect on server centosvm: resources_used.stra = "glad,elated","happy" || resources_used.foo_str4 = eight || resources_used.foo_i = 35 || job_state = F || resources_used.foo_f = 0.35 || resources_used.vmem = 35gb || resources_used.cput = 00:00:35 job 15.centosvm attempt: 30 got: resources_used.foo_i = 11 2017-05-17 13:48:47,727 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:48:47,745 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 15.centosvm 2017-05-17 13:48:47,758 INFO expect on server centosvm: resources_used.stra = "glad,elated","happy" || resources_used.foo_str4 = eight || resources_used.foo_i = 35 || job_state = F || resources_used.foo_f = 0.35 || resources_used.vmem = 35gb || resources_used.cput = 00:00:35 job 15.centosvm attempt: 31 got: resources_used.foo_i = 11 2017-05-17 13:48:48,760 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:48:48,801 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 15.centosvm 2017-05-17 13:48:48,834 INFO expect on server centosvm: resources_used.stra = "glad,elated","happy" || resources_used.foo_str4 = eight || resources_used.foo_i = 35 || job_state = F || resources_used.foo_f = 0.35 || resources_used.vmem = 35gb || resources_used.cput = 00:00:35 job 15.centosvm attempt: 32 got: resources_used.foo_i = 11 2017-05-17 13:48:49,837 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:48:49,885 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 15.centosvm 2017-05-17 13:48:49,916 INFO expect on server centosvm: resources_used.stra = "glad,elated","happy" || resources_used.foo_str4 = eight || resources_used.foo_i = 35 || job_state = F || resources_used.foo_f = 0.35 || resources_used.vmem = 35gb || resources_used.cput = 00:00:35 job 15.centosvm attempt: 33 got: resources_used.foo_i = 11 2017-05-17 13:48:50,920 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:48:50,949 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 15.centosvm 2017-05-17 13:48:50,971 INFO expect on server centosvm: resources_used.stra = "glad,elated","happy" || resources_used.foo_str4 = eight || resources_used.foo_i = 35 || job_state = F || resources_used.foo_f = 0.35 || resources_used.vmem = 35gb || resources_used.cput = 00:00:35 job 15.centosvm attempt: 34 got: resources_used.foo_i = 11 2017-05-17 13:48:51,973 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:48:52,024 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 15.centosvm 2017-05-17 13:48:52,069 INFO expect on server centosvm: resources_used.stra = "glad,elated","happy" || resources_used.foo_str4 = eight || resources_used.foo_i = 35 || job_state = F || resources_used.foo_f = 0.35 || resources_used.vmem = 35gb || resources_used.cput = 00:00:35 job 15.centosvm attempt: 35 got: resources_used.foo_i = 11 2017-05-17 13:48:53,074 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:48:53,120 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 15.centosvm 2017-05-17 13:48:53,166 INFO expect on server centosvm: resources_used.stra = "glad,elated","happy" || resources_used.foo_str4 = eight || resources_used.foo_i = 35 || job_state = F || resources_used.foo_f = 0.35 || resources_used.vmem = 35gb || resources_used.cput = 00:00:35 job 15.centosvm attempt: 36 got: resources_used.foo_i = 11 2017-05-17 13:48:54,169 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:48:54,212 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 15.centosvm 2017-05-17 13:48:54,245 INFO expect on server centosvm: resources_used.stra = "glad,elated","happy" || resources_used.foo_str4 = eight || resources_used.foo_i = 35 || job_state = F || resources_used.foo_f = 0.35 || resources_used.vmem = 35gb || resources_used.cput = 00:00:35 job 15.centosvm attempt: 37 got: resources_used.foo_i = 11 2017-05-17 13:48:55,250 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:48:55,297 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 15.centosvm 2017-05-17 13:48:55,334 INFO expect on server centosvm: resources_used.stra = "glad,elated","happy" || resources_used.foo_str4 = eight || resources_used.foo_i = 35 || job_state = F || resources_used.foo_f = 0.35 || resources_used.vmem = 35gb || resources_used.cput = 00:00:35 job 15.centosvm attempt: 38 got: resources_used.foo_i = 11 2017-05-17 13:48:56,338 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:48:56,383 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 15.centosvm 2017-05-17 13:48:56,411 INFO expect on server centosvm: resources_used.stra = "glad,elated","happy" || resources_used.foo_str4 = eight || resources_used.foo_i = 35 || job_state = F || resources_used.foo_f = 0.35 || resources_used.vmem = 35gb || resources_used.cput = 00:00:35 job 15.centosvm attempt: 39 got: resources_used.foo_i = 11 2017-05-17 13:48:57,415 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:48:57,460 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 15.centosvm 2017-05-17 13:48:57,497 INFO expect on server centosvm: resources_used.stra = "glad,elated","happy" || resources_used.foo_str4 = eight || resources_used.foo_i = 35 || job_state = F || resources_used.foo_f = 0.35 || resources_used.vmem = 35gb || resources_used.cput = 00:00:35 job 15.centosvm attempt: 40 got: resources_used.foo_i = 11 2017-05-17 13:48:58,502 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-05-17 13:48:58,548 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 15.centosvm 2017-05-17 13:48:58,585 INFO expect on server centosvm: resources_used.stra = "glad,elated","happy" || resources_used.foo_str4 = eight || resources_used.foo_i = 35 || job_state = F || resources_used.foo_f = 0.35 || resources_used.vmem = 35gb || resources_used.cput = 00:00:35 job 15.centosvm attempt: 41 ... OK 2017-05-17 13:48:58,586 INFO status on centosvm: job 15.centosvm resources_used.foo_str 2017-05-17 13:48:58,586 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 15.centosvm 2017-05-17 13:48:58,620 INFO =========================================== 2017-05-17 13:48:58,621 INFO Entered TestPbsAccumulateRescUsed tearDown 2017-05-17 13:48:58,621 INFO =========================================== 2017-05-17 13:48:58,622 INFO ============================================ 2017-05-17 13:48:58,622 INFO Completed TestPbsAccumulateRescUsed tearDown 2017-05-17 13:48:58,622 INFO ============================================ 2017-05-17 13:48:58,623 INFO ok 2017-05-17 13:48:58,624 INFO test name: test_server_restart2 (tests.functional.pbs_accumulate_resc_used.TestPbsAccumulateRescUsed)... 2017-05-17 13:48:58,624 INFO test start time: Wed May 17 13:48:58 2017 2017-05-17 13:48:58,624 INFO test docstring: Test that server restart during hook execution has no impact 2017-05-17 13:48:58,624 INFO ======================================== 2017-05-17 13:48:58,625 INFO Entered TestPbsAccumulateRescUsed setUp 2017-05-17 13:48:58,625 INFO ======================================== 2017-05-17 13:48:58,625 INFOCLI centosvm: /opt/pbs/bin/qstat -Bf centosvm 2017-05-17 13:48:58,646 INFO status on centosvm: server 2017-05-17 13:48:58,646 INFOCLI centosvm: /opt/pbs/bin/qstat -Bf centosvm 2017-05-17 13:48:58,661 INFO manager on centosvm: set server {'managers': (3, 'dilipk@*')} 2017-05-17 13:48:58,661 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c set server managers-=dilipk@* 2017-05-17 13:48:58,693 INFO manager on centosvm: set server {'managers': (2, 'dilipk@*')} 2017-05-17 13:48:58,694 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c set server managers+=dilipk@* 2017-05-17 13:48:58,723 INFO server centosvm: reverting configuration to defaults 2017-05-17 13:48:58,724 INFOCLI centosvm: /opt/pbs/bin/qstat -Bf centosvm 2017-05-17 13:48:58,736 INFO select on centosvm: __ALL__ 2017-05-17 13:48:58,736 INFOCLI centosvm: /opt/pbs/bin/qselect -x 2017-05-17 13:48:58,751 INFO delete job on centosvm: 15.centosvm 2017-05-17 13:48:58,752 INFOCLI centosvm: /opt/pbs/bin/qdel -W force -x 15.centosvm 2017-05-17 13:48:58,768 INFOCLI centosvm: /opt/pbs/bin/qstat -f 15.centosvm 2017-05-17 13:48:58,785 INFOCLI centosvm: /opt/pbs/bin/qstat -f @centosvm 2017-05-17 13:48:58,798 INFO expect on server centosvm: job_state set 0 job ... OK 2017-05-17 13:48:58,799 INFOCLI centosvm: /opt/pbs/bin/pbs_rstat -f 2017-05-17 13:48:58,812 INFO manager on centosvm: unset server ['pbs_license_max', 'pbs_license_min', 'job_history_enable'] 2017-05-17 13:48:58,812 INFOCLI centosvm: /opt/pbs/bin/qmgr -c unset server pbs_license_max,pbs_license_min,job_history_enable 2017-05-17 13:48:58,837 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-05-17 13:48:58,860 INFO manager on centosvm: delete hook start,prolo 2017-05-17 13:48:58,861 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c delete hook start,prolo 2017-05-17 13:48:58,891 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook start 2017-05-17 13:48:58,910 INFO expect on server centosvm: unset hook start ... OK 2017-05-17 13:48:58,910 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook prolo 2017-05-17 13:48:58,935 INFO expect on server centosvm: unset hook prolo ... OK 2017-05-17 13:48:58,935 INFOCLI centosvm: /opt/pbs/bin/qstat -Qf @centosvm 2017-05-17 13:48:58,945 INFO manager on centosvm: delete queue workq 2017-05-17 13:48:58,946 INFOCLI centosvm: /opt/pbs/bin/qmgr -c delete queue workq 2017-05-17 13:48:58,959 INFOCLI centosvm: /opt/pbs/bin/qstat -Qf workq@centosvm 2017-05-17 13:48:58,970 INFO expect on server centosvm: unset queue workq ... OK 2017-05-17 13:48:58,970 INFO manager on centosvm: create queue workq {'started': 'True', 'queue_type': 'Execution', 'enabled': 'True'} 2017-05-17 13:48:58,970 INFOCLI centosvm: /opt/pbs/bin/qmgr -c create queue workq started=True,queue_type=Execution,enabled=True 2017-05-17 13:48:58,983 INFOCLI centosvm: /opt/pbs/bin/qstat -Qf workq@centosvm 2017-05-17 13:48:58,995 INFO expect on server centosvm: started set True || queue_type set Execution || enabled set True queue workq ... OK 2017-05-17 13:48:58,995 INFO manager on centosvm: set server {'default_queue': 'workq'} 2017-05-17 13:48:58,995 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server default_queue=workq 2017-05-17 13:48:59,009 INFO status on centosvm: resource 2017-05-17 13:48:59,009 INFO manager on centosvm: list resource 2017-05-17 13:48:59,009 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource 2017-05-17 13:48:59,021 INFO manager on centosvm: delete resource foo_f,foo_i,foo_str2,foo_str3,foo_str4,foo_str,stra 2017-05-17 13:48:59,021 INFOCLI centosvm: /opt/pbs/bin/qmgr -c delete resource foo_f,foo_i,foo_str2,foo_str3,foo_str4,foo_str,stra 2017-05-17 13:48:59,182 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_f 2017-05-17 13:48:59,193 INFO expect on server centosvm: unset resource foo_f ... OK 2017-05-17 13:48:59,193 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_i 2017-05-17 13:48:59,204 INFO expect on server centosvm: unset resource foo_i ... OK 2017-05-17 13:48:59,204 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_str2 2017-05-17 13:48:59,214 INFO expect on server centosvm: unset resource foo_str2 ... OK 2017-05-17 13:48:59,215 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_str3 2017-05-17 13:48:59,227 INFO expect on server centosvm: unset resource foo_str3 ... OK 2017-05-17 13:48:59,227 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_str4 2017-05-17 13:48:59,237 INFO expect on server centosvm: unset resource foo_str4 ... OK 2017-05-17 13:48:59,238 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource foo_str 2017-05-17 13:48:59,251 INFO expect on server centosvm: unset resource foo_str ... OK 2017-05-17 13:48:59,251 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list resource stra 2017-05-17 13:48:59,262 INFO expect on server centosvm: unset resource stra ... OK 2017-05-17 13:48:59,262 INFOCLI status on centosvm: server license_count 2017-05-17 13:48:59,262 INFOCLI centosvm: /opt/pbs/bin/qstat -Bf centosvm 2017-05-17 13:48:59,273 INFO server: centosvm licensed 2017-05-17 13:48:59,289 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/server_priv/comm.lock 2017-05-17 13:48:59,313 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-05-17 13:48:59,324 INFO scheduler centosvm: reverting configuration to defaults 2017-05-17 13:48:59,324 INFO manager on centosvm: list sched 2017-05-17 13:48:59,324 INFOCLI centosvm: /opt/pbs/bin/qmgr -c list sched 2017-05-17 13:48:59,336 INFO manager on centosvm: unset sched ['sched_cycle_length'] 2017-05-17 13:48:59,336 INFOCLI centosvm: /opt/pbs/bin/qmgr -c unset sched sched_cycle_length 2017-05-17 13:48:59,350 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/dedicated_time 2017-05-17 13:48:59,360 INFOCLI2 centosvm: cmp /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-05-17 13:48:59,363 ERROR err: ['cmp: /var/spool/pbs/sched_priv/resource_group: Permission denied'] 2017-05-17 13:48:59,363 INFOCLI2 centosvm: sudo -H /usr/bin/cp -p /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2017-05-17 13:48:59,375 INFOCLI2 centosvm: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/resource_group 2017-05-17 13:48:59,386 INFO scheduler centosvm: reverting holidays file to default 2017-05-17 13:48:59,386 INFOCLI2 centosvm: cmp /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-05-17 13:48:59,389 ERROR err: ['cmp: /var/spool/pbs/sched_priv/holidays: Permission denied'] 2017-05-17 13:48:59,390 INFOCLI2 centosvm: sudo -H /usr/bin/cp -p /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2017-05-17 13:48:59,400 INFOCLI2 centosvm: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/holidays 2017-05-17 13:48:59,411 INFO scheduler centosvm: Parsing holidays file 2017-05-17 13:48:59,412 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2017-05-17 13:48:59,423 INFOCLI2 centosvm: cmp /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-05-17 13:48:59,426 ERROR err: ['cmp: /var/spool/pbs/sched_priv/sched_config: Permission denied'] 2017-05-17 13:48:59,426 INFOCLI2 centosvm: sudo -H /usr/bin/cp -p /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2017-05-17 13:48:59,437 INFOCLI2 centosvm: sudo -H /usr/bin/chmod 0644 /var/spool/pbs/sched_priv/sched_config 2017-05-17 13:48:59,451 INFO scheduler centosvm: sent signal -HUP 2017-05-17 13:48:59,451 INFOCLI2 centosvm: sudo -H kill -HUP 84281 2017-05-17 13:48:59,462 INFOCLI2 centosvm: sudo -H /usr/bin/rm /var/spool/pbs/sched_priv/usage 2017-05-17 13:48:59,475 ERROR err: ['/usr/bin/rm: cannot remove \xe2\x80\x98/var/spool/pbs/sched_priv/usage\xe2\x80\x99: No such file or directory'] 2017-05-17 13:48:59,502 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2017-05-17 13:48:59,513 INFOCLI2 centosvm: sudo -H cmp /opt/pbs/sbin/pbs_mom /opt/pbs/sbin/pbs_mom.cpuset 2017-05-17 13:48:59,537 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:48:59,553 INFO mom centosvm: reverting configuration to defaults 2017-05-17 13:48:59,554 INFOCLI2 centosvm: sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2017-05-17 13:48:59,564 INFOCLI2 centosvm: sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2017-05-17 13:48:59,575 INFOCLI2 centosvm: sudo -H ls /var/spool/pbs/mom_priv/config.d 2017-05-17 13:48:59,588 INFO mom centosvm: sent signal -HUP 2017-05-17 13:48:59,588 INFOCLI2 centosvm: sudo -H kill -HUP 90805 2017-05-17 13:48:59,614 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:48:59,627 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v -a 2017-05-17 13:48:59,649 INFO status on centosvm: node centosvm 2017-05-17 13:48:59,649 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:48:59,661 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:48:59,674 INFO expect on server centosvm: state = free node centosvm ... OK 2017-05-17 13:48:59,675 INFOCLI2 centosvm2: ssh centosvm2 sudo -H cmp /opt/pbs/sbin/pbs_mom /opt/pbs/sbin/pbs_mom.cpuset 2017-05-17 13:49:00,619 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:49:01,015 INFO mom centosvm2: reverting configuration to defaults 2017-05-17 13:49:01,015 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2017-05-17 13:49:01,401 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2017-05-17 13:49:01,892 INFOCLI2 centosvm2: ssh centosvm2 sudo -H ls /var/spool/pbs/mom_priv/config.d 2017-05-17 13:49:02,350 INFO mom centosvm2: sent signal -HUP 2017-05-17 13:49:02,351 INFOCLI2 centosvm2: ssh centosvm2 sudo -H kill -HUP 15795 2017-05-17 13:49:03,510 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:49:03,974 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v -a 2017-05-17 13:49:04,007 INFO status on centosvm: node centosvm2 2017-05-17 13:49:04,008 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm2 2017-05-17 13:49:04,041 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm2 2017-05-17 13:49:04,063 INFO expect on server centosvm: state = free node centosvm2 ... OK 2017-05-17 13:49:04,064 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H cmp /opt/pbs/sbin/pbs_mom /opt/pbs/sbin/pbs_mom.cpuset 2017-05-17 13:49:04,928 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:49:05,303 INFO mom centosvm_nfs: reverting configuration to defaults 2017-05-17 13:49:05,303 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2017-05-17 13:49:05,891 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2017-05-17 13:49:06,364 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H ls /var/spool/pbs/mom_priv/config.d 2017-05-17 13:49:06,783 INFO mom centosvm_nfs: sent signal -HUP 2017-05-17 13:49:06,783 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H kill -HUP 76746 2017-05-17 13:49:07,893 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-05-17 13:49:08,284 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v -a 2017-05-17 13:49:08,296 INFO status on centosvm: node centosvm_nfs 2017-05-17 13:49:08,296 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm_nfs 2017-05-17 13:49:08,309 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm_nfs 2017-05-17 13:49:08,322 INFO expect on server centosvm: state = free node centosvm_nfs ... OK 2017-05-17 13:49:08,323 INFO ========================================= 2017-05-17 13:49:08,323 INFO Completed TestPbsAccumulateRescUsed setUp 2017-05-17 13:49:08,323 INFO ========================================= 2017-05-17 13:49:08,323 INFO len moms = 3 2017-05-17 13:49:08,323 INFOCLI2 centosvm: sudo -H /usr/bin/rm -rf /var/spool/pbs/mom_priv/config.d 2017-05-17 13:49:08,337 INFOCLI2 centosvm2: ssh centosvm2 sudo -H /usr/bin/rm -rf /var/spool/pbs/mom_priv/config.d 2017-05-17 13:49:08,756 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H /usr/bin/rm -rf /var/spool/pbs/mom_priv/config.d 2017-05-17 13:49:09,233 INFO manager on centosvm: delete node 2017-05-17 13:49:09,234 INFOCLI centosvm: /opt/pbs/bin/qmgr -c delete node @default 2017-05-17 13:49:09,258 INFO manager on centosvm: create node centosvm 2017-05-17 13:49:09,258 INFOCLI centosvm: /opt/pbs/bin/qmgr -c create node centosvm 2017-05-17 13:49:09,282 INFO manager on centosvm: create node centosvm2 2017-05-17 13:49:09,282 INFOCLI centosvm: /opt/pbs/bin/qmgr -c create node centosvm2 2017-05-17 13:49:09,305 INFO manager on centosvm: create node centosvm_nfs 2017-05-17 13:49:09,305 INFOCLI centosvm: /opt/pbs/bin/qmgr -c create node centosvm_nfs 2017-05-17 13:49:09,334 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:49:09,352 INFO expect on server centosvm: state = free node centosvm got: state = state-unknown,down 2017-05-17 13:49:09,854 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:49:09,896 INFO expect on server centosvm: state = free node centosvm attempt: 2 got: state = state-unknown,down 2017-05-17 13:49:10,398 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:49:10,444 INFO expect on server centosvm: state = free node centosvm attempt: 3 got: state = state-unknown,down 2017-05-17 13:49:10,946 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:49:10,986 INFO expect on server centosvm: state = free node centosvm attempt: 4 got: state = state-unknown,down 2017-05-17 13:49:11,488 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm 2017-05-17 13:49:11,533 INFO expect on server centosvm: state = free node centosvm attempt: 5 ... OK 2017-05-17 13:49:11,535 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm2 2017-05-17 13:49:11,573 INFO expect on server centosvm: state = free node centosvm2 ... OK 2017-05-17 13:49:11,574 INFOCLI centosvm: /opt/pbs/bin/pbsnodes -s centosvm -v centosvm_nfs 2017-05-17 13:49:11,615 INFO expect on server centosvm: state = free node centosvm_nfs ... OK 2017-05-17 13:49:11,616 INFO status on centosvm: hook 2017-05-17 13:49:11,617 INFO manager on centosvm: list hook 2017-05-17 13:49:11,617 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-05-17 13:49:11,674 INFO status on centosvm: hook 2017-05-17 13:49:11,674 INFO manager on centosvm: list hook 2017-05-17 13:49:11,675 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-05-17 13:49:11,716 INFO manager on centosvm: create hook start 2017-05-17 13:49:11,716 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c create hook start 2017-05-17 13:49:11,739 INFO manager on centosvm: set hook start {'enabled': 'True', 'event': 'exechost_startup'} 2017-05-17 13:49:11,739 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c set hook start enabled=True,event=exechost_startup 2017-05-17 13:49:11,773 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook start 2017-05-17 13:49:11,794 INFO expect on server centosvm: enabled set True || event set exechost_startup hook start ... OK 2017-05-17 13:49:11,794 INFO manager on centosvm: import hook start {'content-type': 'application/x-python', 'input-file': '/tmp/PtlPbsCg0UjP', 'content-encoding': 'default'} 2017-05-17 13:49:11,794 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c import hook start application/x-python default /tmp/PtlPbsCg0UjP 2017-05-17 13:49:11,815 INFO server centosvm: imported hook body --- import pbs e=pbs.event() localnode=pbs.get_local_nodename() e.vnode_list[localnode].resources_available['foo_i'] = 7 e.vnode_list[localnode].resources_available['foo_f'] = 5.0 e.vnode_list[localnode].resources_available['foo_str'] = "seventyseven" --- 2017-05-17 13:49:11,815 INFO mom centosvm: sent signal -HUP 2017-05-17 13:49:11,815 INFOCLI2 centosvm: sudo -H kill -HUP 90805 2017-05-17 13:49:11,831 INFO mom centosvm2: sent signal -HUP 2017-05-17 13:49:11,831 INFOCLI2 centosvm2: ssh centosvm2 sudo -H kill -HUP 15795 2017-05-17 13:49:12,227 INFO mom centosvm_nfs: sent signal -HUP 2017-05-17 13:49:12,229 INFOCLI2 centosvm_nfs: ssh centosvm_nfs sudo -H kill -HUP 76746 2017-05-17 13:49:12,693 INFO manager on centosvm: set server {'job_history_enable': 'True'} 2017-05-17 13:49:12,693 INFOCLI centosvm: /opt/pbs/bin/qmgr -c set server job_history_enable=True 2017-05-17 13:49:12,707 INFO manager on centosvm as root: create resource foo_str2 {'flag': 'h', 'type': 'string'} 2017-05-17 13:49:12,707 INFOCLI centosvm: sudo -H -u root /opt/pbs/bin/qmgr -c create resource foo_str2 flag=h,type=string 2017-05-17 13:49:12,746 INFO mom centosvm log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:49:13,124 INFO mom centosvm2 log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:49:13,620 INFO mom centosvm_nfs log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:49:13,620 INFO manager on centosvm as root: create resource foo_str3 {'flag': 'h', 'type': 'string'} 2017-05-17 13:49:13,621 INFOCLI centosvm: sudo -H -u root /opt/pbs/bin/qmgr -c create resource foo_str3 flag=h,type=string 2017-05-17 13:49:13,668 INFO mom centosvm log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:49:14,130 INFO mom centosvm2 log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:49:14,482 INFO mom centosvm_nfs log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:49:14,483 INFO manager on centosvm as root: create resource foo_str4 {'flag': 'h', 'type': 'string'} 2017-05-17 13:49:14,483 INFOCLI centosvm: sudo -H -u root /opt/pbs/bin/qmgr -c create resource foo_str4 flag=h,type=string 2017-05-17 13:49:14,518 INFO mom centosvm log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:49:14,942 INFO mom centosvm2 log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:49:15,339 INFO mom centosvm_nfs log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:49:15,339 INFO manager on centosvm as root: create resource stra {'flag': 'h', 'type': 'string_array'} 2017-05-17 13:49:15,340 INFOCLI centosvm: sudo -H -u root /opt/pbs/bin/qmgr -c create resource stra flag=h,type=string_array 2017-05-17 13:49:15,377 INFO mom centosvm log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:49:15,760 INFO mom centosvm2 log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:49:16,128 INFO mom centosvm_nfs log match: searching for "resourcedef;copy hook-related file"... OK 2017-05-17 13:49:16,129 INFO status on centosvm: hook 2017-05-17 13:49:16,129 INFO manager on centosvm: list hook 2017-05-17 13:49:16,129 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-05-17 13:49:16,187 INFO status on centosvm: hook 2017-05-17 13:49:16,187 INFO manager on centosvm: list hook 2017-05-17 13:49:16,188 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-05-17 13:49:16,223 INFO manager on centosvm: create hook epi 2017-05-17 13:49:16,224 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c create hook epi 2017-05-17 13:49:16,261 INFO manager on centosvm: set hook epi {'enabled': 'True', 'event': 'execjob_epilogue'} 2017-05-17 13:49:16,261 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c set hook epi enabled=True,event=execjob_epilogue 2017-05-17 13:49:16,302 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c list hook epi 2017-05-17 13:49:16,336 INFO expect on server centosvm: enabled set True || event set execjob_epilogue hook epi ... OK 2017-05-17 13:49:16,337 INFO manager on centosvm: import hook epi {'content-type': 'application/x-python', 'input-file': '/tmp/PtlPbshdmSji', 'content-encoding': 'default'} 2017-05-17 13:49:16,337 INFOCLI centosvm: sudo -H /opt/pbs/bin/qmgr -c import hook epi application/x-python default /tmp/PtlPbshdmSji 2017-05-17 13:49:16,361 INFO server centosvm: imported hook body --- import pbs import time e = pbs.event() pbs.logmsg(pbs.LOG_DEBUG, "executed epilogue hook") if e.job.in_ms_mom(): e.job.resources_used["vmem"] = pbs.size("9gb") e.job.resources_used["foo_i"] = 9 e.job.resources_used["foo_f"] = 0.09 e.job.resources_used["foo_str"] = '{"seven":7}' e.job.resources_used["cput"] = 10 else: e.job.resources_used["vmem"] = pbs.size("10gb") e.job.resources_used["foo_i"] = 10 e.job.resources_used["foo_f"] = 0.10 e.job.resources_used["foo_str"] = '{"eight":8,"nine":9}' e.job.resources_used["cput"] = 20 time.sleep(15) --- 2017-05-17 13:49:16,362 INFO job: executable set to /bin/sleep with arguments: 100 2017-05-17 13:49:16,362 INFO job: executable set to /bin/sleep with arguments: 5 2017-05-17 13:49:16,363 INFOCLI centosvm: sudo -H -u pbsuser /opt/pbs/bin/qsub -l place=scatter -l walltime=10 -l select=3:ncpus=1 -- /bin/sleep 5 2017-05-17 13:49:16,388 INFO submit to centosvm as pbsuser: job 16.centosvm OrderedDict([('Resource_List.place', 'scatter'), ('Resource_List.walltime', 10), ('Resource_List.select', '3:ncpus=1')]) 2017-05-17 13:49:16,388 INFO server centosvm: expect offset set to 5 2017-05-17 13:49:21,392 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 16.centosvm 2017-05-17 13:49:22,437 INFO expect on server centosvm: no data for resources_used.foo_i = 29 || resources_used.foo_f = 0.29 || resources_used.foo_str = '{"eight": 8, "seven": 7, "nine": 9}' job 16.centosvm 2017-05-17 13:49:22,438 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 16.centosvm 2017-05-17 13:49:23,479 INFO expect on server centosvm: no data for resources_used.foo_i = 29 || resources_used.foo_f = 0.29 || resources_used.foo_str = '{"eight": 8, "seven": 7, "nine": 9}' job 16.centosvm attempt: 2 2017-05-17 13:49:23,480 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 16.centosvm 2017-05-17 13:49:24,524 INFO expect on server centosvm: no data for resources_used.foo_i = 29 || resources_used.foo_f = 0.29 || resources_used.foo_str = '{"eight": 8, "seven": 7, "nine": 9}' job 16.centosvm attempt: 3 2017-05-17 13:49:24,525 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 16.centosvm 2017-05-17 13:49:25,566 INFO expect on server centosvm: no data for resources_used.foo_i = 29 || resources_used.foo_f = 0.29 || resources_used.foo_str = '{"eight": 8, "seven": 7, "nine": 9}' job 16.centosvm attempt: 4 2017-05-17 13:49:25,567 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 16.centosvm 2017-05-17 13:49:26,613 INFO expect on server centosvm: no data for resources_used.foo_i = 29 || resources_used.foo_f = 0.29 || resources_used.foo_str = '{"eight": 8, "seven": 7, "nine": 9}' job 16.centosvm attempt: 5 2017-05-17 13:49:26,614 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 16.centosvm 2017-05-17 13:49:27,659 INFO expect on server centosvm: no data for resources_used.foo_i = 29 || resources_used.foo_f = 0.29 || resources_used.foo_str = '{"eight": 8, "seven": 7, "nine": 9}' job 16.centosvm attempt: 6 2017-05-17 13:49:27,661 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 16.centosvm 2017-05-17 13:49:28,703 INFO expect on server centosvm: no data for resources_used.foo_i = 29 || resources_used.foo_f = 0.29 || resources_used.foo_str = '{"eight": 8, "seven": 7, "nine": 9}' job 16.centosvm attempt: 7 2017-05-17 13:49:28,704 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 16.centosvm 2017-05-17 13:49:29,749 INFO expect on server centosvm: no data for resources_used.foo_i = 29 || resources_used.foo_f = 0.29 || resources_used.foo_str = '{"eight": 8, "seven": 7, "nine": 9}' job 16.centosvm attempt: 8 2017-05-17 13:49:29,751 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 16.centosvm 2017-05-17 13:49:30,797 INFO expect on server centosvm: no data for resources_used.foo_i = 29 || resources_used.foo_f = 0.29 || resources_used.foo_str = '{"eight": 8, "seven": 7, "nine": 9}' job 16.centosvm attempt: 9 2017-05-17 13:49:30,798 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 16.centosvm 2017-05-17 13:49:31,827 INFO expect on server centosvm: no data for resources_used.foo_i = 29 || resources_used.foo_f = 0.29 || resources_used.foo_str = '{"eight": 8, "seven": 7, "nine": 9}' job 16.centosvm attempt: 10 2017-05-17 13:49:31,828 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 16.centosvm 2017-05-17 13:49:32,872 INFO expect on server centosvm: no data for resources_used.foo_i = 29 || resources_used.foo_f = 0.29 || resources_used.foo_str = '{"eight": 8, "seven": 7, "nine": 9}' job 16.centosvm attempt: 11 2017-05-17 13:49:32,873 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 16.centosvm 2017-05-17 13:49:33,919 INFO expect on server centosvm: no data for resources_used.foo_i = 29 || resources_used.foo_f = 0.29 || resources_used.foo_str = '{"eight": 8, "seven": 7, "nine": 9}' job 16.centosvm attempt: 12 2017-05-17 13:49:33,920 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 16.centosvm 2017-05-17 13:49:34,963 INFO expect on server centosvm: no data for resources_used.foo_i = 29 || resources_used.foo_f = 0.29 || resources_used.foo_str = '{"eight": 8, "seven": 7, "nine": 9}' job 16.centosvm attempt: 13 2017-05-17 13:49:34,964 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 16.centosvm 2017-05-17 13:49:36,006 INFO expect on server centosvm: no data for resources_used.foo_i = 29 || resources_used.foo_f = 0.29 || resources_used.foo_str = '{"eight": 8, "seven": 7, "nine": 9}' job 16.centosvm attempt: 14 2017-05-17 13:49:36,007 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 16.centosvm 2017-05-17 13:49:37,055 INFO expect on server centosvm: no data for resources_used.foo_i = 29 || resources_used.foo_f = 0.29 || resources_used.foo_str = '{"eight": 8, "seven": 7, "nine": 9}' job 16.centosvm attempt: 15 2017-05-17 13:49:37,056 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 16.centosvm 2017-05-17 13:49:38,102 INFO expect on server centosvm: no data for resources_used.foo_i = 29 || resources_used.foo_f = 0.29 || resources_used.foo_str = '{"eight": 8, "seven": 7, "nine": 9}' job 16.centosvm attempt: 16 2017-05-17 13:49:38,103 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 16.centosvm 2017-05-17 13:49:39,139 INFO expect on server centosvm: no data for resources_used.foo_i = 29 || resources_used.foo_f = 0.29 || resources_used.foo_str = '{"eight": 8, "seven": 7, "nine": 9}' job 16.centosvm attempt: 17 2017-05-17 13:49:39,141 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 16.centosvm 2017-05-17 13:49:40,184 INFO expect on server centosvm: no data for resources_used.foo_i = 29 || resources_used.foo_f = 0.29 || resources_used.foo_str = '{"eight": 8, "seven": 7, "nine": 9}' job 16.centosvm attempt: 18 2017-05-17 13:49:40,185 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 16.centosvm 2017-05-17 13:49:41,228 INFO expect on server centosvm: no data for resources_used.foo_i = 29 || resources_used.foo_f = 0.29 || resources_used.foo_str = '{"eight": 8, "seven": 7, "nine": 9}' job 16.centosvm attempt: 19 2017-05-17 13:49:41,229 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 16.centosvm 2017-05-17 13:49:42,271 INFO expect on server centosvm: no data for resources_used.foo_i = 29 || resources_used.foo_f = 0.29 || resources_used.foo_str = '{"eight": 8, "seven": 7, "nine": 9}' job 16.centosvm attempt: 20 2017-05-17 13:49:42,272 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 16.centosvm 2017-05-17 13:49:43,314 INFO expect on server centosvm: no data for resources_used.foo_i = 29 || resources_used.foo_f = 0.29 || resources_used.foo_str = '{"eight": 8, "seven": 7, "nine": 9}' job 16.centosvm attempt: 21 2017-05-17 13:49:43,315 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 16.centosvm 2017-05-17 13:49:44,360 INFO expect on server centosvm: no data for resources_used.foo_i = 29 || resources_used.foo_f = 0.29 || resources_used.foo_str = '{"eight": 8, "seven": 7, "nine": 9}' job 16.centosvm attempt: 22 2017-05-17 13:49:44,362 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 16.centosvm 2017-05-17 13:49:45,408 INFO expect on server centosvm: no data for resources_used.foo_i = 29 || resources_used.foo_f = 0.29 || resources_used.foo_str = '{"eight": 8, "seven": 7, "nine": 9}' job 16.centosvm attempt: 23 2017-05-17 13:49:45,409 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 16.centosvm 2017-05-17 13:49:46,454 INFO expect on server centosvm: no data for resources_used.foo_i = 29 || resources_used.foo_f = 0.29 || resources_used.foo_str = '{"eight": 8, "seven": 7, "nine": 9}' job 16.centosvm attempt: 24 2017-05-17 13:49:46,456 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 16.centosvm 2017-05-17 13:49:47,502 INFO expect on server centosvm: no data for resources_used.foo_i = 29 || resources_used.foo_f = 0.29 || resources_used.foo_str = '{"eight": 8, "seven": 7, "nine": 9}' job 16.centosvm attempt: 25 2017-05-17 13:49:47,503 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 16.centosvm 2017-05-17 13:49:48,549 INFO expect on server centosvm: no data for resources_used.foo_i = 29 || resources_used.foo_f = 0.29 || resources_used.foo_str = '{"eight": 8, "seven": 7, "nine": 9}' job 16.centosvm attempt: 26 2017-05-17 13:49:48,550 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 16.centosvm 2017-05-17 13:49:49,595 INFO expect on server centosvm: no data for resources_used.foo_i = 29 || resources_used.foo_f = 0.29 || resources_used.foo_str = '{"eight": 8, "seven": 7, "nine": 9}' job 16.centosvm attempt: 27 2017-05-17 13:49:49,596 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 16.centosvm 2017-05-17 13:49:50,642 INFO expect on server centosvm: no data for resources_used.foo_i = 29 || resources_used.foo_f = 0.29 || resources_used.foo_str = '{"eight": 8, "seven": 7, "nine": 9}' job 16.centosvm attempt: 28 2017-05-17 13:49:50,643 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 16.centosvm 2017-05-17 13:49:51,687 INFO expect on server centosvm: no data for resources_used.foo_i = 29 || resources_used.foo_f = 0.29 || resources_used.foo_str = '{"eight": 8, "seven": 7, "nine": 9}' job 16.centosvm attempt: 29 2017-05-17 13:49:51,688 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 16.centosvm 2017-05-17 13:49:52,734 INFO expect on server centosvm: no data for resources_used.foo_i = 29 || resources_used.foo_f = 0.29 || resources_used.foo_str = '{"eight": 8, "seven": 7, "nine": 9}' job 16.centosvm attempt: 30 2017-05-17 13:49:52,735 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 16.centosvm 2017-05-17 13:49:52,789 INFO expect on server centosvm: resources_used.foo_i = 29 || resources_used.foo_f = 0.29 || resources_used.foo_str = '{"eight": 8, "seven": 7, "nine": 9}' job 16.centosvm attempt: 31 ... OK 2017-05-17 13:49:52,790 INFOCLI centosvm: /opt/pbs/bin/qstat -Bf centosvm 2017-05-17 13:49:52,831 INFO server centosvm: stopping Server on host centosvm 2017-05-17 13:49:52,831 INFOCLI2 centosvm: sudo -H kill -TERM 92669 2017-05-17 13:49:52,870 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/server_priv/server.lock 2017-05-17 13:49:58,215 INFO service: starting pbs_server 2017-05-17 13:49:58,216 INFOCLI centosvm: sudo -H /opt/pbs/sbin/pbs_server 2017-05-17 13:50:04,331 INFOCLI2 centosvm: sudo -H /usr/bin/cat /var/spool/pbs/server_priv/server.lock 2017-05-17 13:50:04,405 INFOCLI centosvm: /opt/pbs/bin/qstat -Bf centosvm 2017-05-17 13:50:04,423 INFOCLI centosvm: /opt/pbs/bin/qstat -x -f 16.centosvm 2017-05-17 13:50:04,436 INFO expect on server centosvm: resources_used.foo_i = 29 || resources_used.foo_f = 0.29 || resources_used.foo_str = '{"eight": 8, "seven": 7, "nine": 9}' job 16.centosvm ... OK 2017-05-17 13:50:04,437 INFO =========================================== 2017-05-17 13:50:04,437 INFO Entered TestPbsAccumulateRescUsed tearDown 2017-05-17 13:50:04,437 INFO =========================================== 2017-05-17 13:50:04,437 INFO ============================================ 2017-05-17 13:50:04,437 INFO Completed TestPbsAccumulateRescUsed tearDown 2017-05-17 13:50:04,437 INFO ============================================ 2017-05-17 13:50:04,437 INFO ok 2017-05-17 13:50:04,438 INFO 2017-05-17 13:50:04,438 INFO ====================================================================== 2017-05-17 13:50:04,438 INFO ERROR: test_mom_down (tests.functional.pbs_accumulate_resc_used.TestPbsAccumulateRescUsed) 2017-05-17 13:50:04,438 INFO ___m_oo_m___ 2017-05-17 13:50:04,438 INFO Traceback (most recent call last): File "/home/dilipk/workspace/pbspro/test/tests/functional/pbs_accumulate_resc_used.py", line 1010, in test_mom_down self.server.expect(JOB, "resources_used.foo_i", op=UNSET, id=jid2) File "/usr/lib/python2.7/site-packages/PbsTestLab-1.0.0-py2.7.egg/ptl/lib/pbs_testlib.py", line 8418, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/PbsTestLab-1.0.0-py2.7.egg/ptl/lib/pbs_testlib.py", line 8418, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/PbsTestLab-1.0.0-py2.7.egg/ptl/lib/pbs_testlib.py", line 8418, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/PbsTestLab-1.0.0-py2.7.egg/ptl/lib/pbs_testlib.py", line 8418, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/PbsTestLab-1.0.0-py2.7.egg/ptl/lib/pbs_testlib.py", line 8418, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/PbsTestLab-1.0.0-py2.7.egg/ptl/lib/pbs_testlib.py", line 8418, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/PbsTestLab-1.0.0-py2.7.egg/ptl/lib/pbs_testlib.py", line 8418, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/PbsTestLab-1.0.0-py2.7.egg/ptl/lib/pbs_testlib.py", line 8418, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/PbsTestLab-1.0.0-py2.7.egg/ptl/lib/pbs_testlib.py", line 8418, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/PbsTestLab-1.0.0-py2.7.egg/ptl/lib/pbs_testlib.py", line 8418, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/PbsTestLab-1.0.0-py2.7.egg/ptl/lib/pbs_testlib.py", line 8418, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/PbsTestLab-1.0.0-py2.7.egg/ptl/lib/pbs_testlib.py", line 8418, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/PbsTestLab-1.0.0-py2.7.egg/ptl/lib/pbs_testlib.py", line 8418, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/PbsTestLab-1.0.0-py2.7.egg/ptl/lib/pbs_testlib.py", line 8418, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/PbsTestLab-1.0.0-py2.7.egg/ptl/lib/pbs_testlib.py", line 8418, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/PbsTestLab-1.0.0-py2.7.egg/ptl/lib/pbs_testlib.py", line 8418, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/PbsTestLab-1.0.0-py2.7.egg/ptl/lib/pbs_testlib.py", line 8418, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/PbsTestLab-1.0.0-py2.7.egg/ptl/lib/pbs_testlib.py", line 8418, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/PbsTestLab-1.0.0-py2.7.egg/ptl/lib/pbs_testlib.py", line 8418, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/PbsTestLab-1.0.0-py2.7.egg/ptl/lib/pbs_testlib.py", line 8418, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/PbsTestLab-1.0.0-py2.7.egg/ptl/lib/pbs_testlib.py", line 8418, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/PbsTestLab-1.0.0-py2.7.egg/ptl/lib/pbs_testlib.py", line 8418, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/PbsTestLab-1.0.0-py2.7.egg/ptl/lib/pbs_testlib.py", line 8418, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/PbsTestLab-1.0.0-py2.7.egg/ptl/lib/pbs_testlib.py", line 8418, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/PbsTestLab-1.0.0-py2.7.egg/ptl/lib/pbs_testlib.py", line 8418, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/PbsTestLab-1.0.0-py2.7.egg/ptl/lib/pbs_testlib.py", line 8418, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/PbsTestLab-1.0.0-py2.7.egg/ptl/lib/pbs_testlib.py", line 8418, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/PbsTestLab-1.0.0-py2.7.egg/ptl/lib/pbs_testlib.py", line 8418, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/PbsTestLab-1.0.0-py2.7.egg/ptl/lib/pbs_testlib.py", line 8418, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/PbsTestLab-1.0.0-py2.7.egg/ptl/lib/pbs_testlib.py", line 8418, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/PbsTestLab-1.0.0-py2.7.egg/ptl/lib/pbs_testlib.py", line 8418, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/PbsTestLab-1.0.0-py2.7.egg/ptl/lib/pbs_testlib.py", line 8418, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/PbsTestLab-1.0.0-py2.7.egg/ptl/lib/pbs_testlib.py", line 8418, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/PbsTestLab-1.0.0-py2.7.egg/ptl/lib/pbs_testlib.py", line 8418, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/PbsTestLab-1.0.0-py2.7.egg/ptl/lib/pbs_testlib.py", line 8418, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/PbsTestLab-1.0.0-py2.7.egg/ptl/lib/pbs_testlib.py", line 8418, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/PbsTestLab-1.0.0-py2.7.egg/ptl/lib/pbs_testlib.py", line 8418, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/PbsTestLab-1.0.0-py2.7.egg/ptl/lib/pbs_testlib.py", line 8418, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/PbsTestLab-1.0.0-py2.7.egg/ptl/lib/pbs_testlib.py", line 8418, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/PbsTestLab-1.0.0-py2.7.egg/ptl/lib/pbs_testlib.py", line 8418, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/PbsTestLab-1.0.0-py2.7.egg/ptl/lib/pbs_testlib.py", line 8418, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/PbsTestLab-1.0.0-py2.7.egg/ptl/lib/pbs_testlib.py", line 8418, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/PbsTestLab-1.0.0-py2.7.egg/ptl/lib/pbs_testlib.py", line 8418, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/PbsTestLab-1.0.0-py2.7.egg/ptl/lib/pbs_testlib.py", line 8418, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/PbsTestLab-1.0.0-py2.7.egg/ptl/lib/pbs_testlib.py", line 8418, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/PbsTestLab-1.0.0-py2.7.egg/ptl/lib/pbs_testlib.py", line 8418, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/PbsTestLab-1.0.0-py2.7.egg/ptl/lib/pbs_testlib.py", line 8418, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/PbsTestLab-1.0.0-py2.7.egg/ptl/lib/pbs_testlib.py", line 8418, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/PbsTestLab-1.0.0-py2.7.egg/ptl/lib/pbs_testlib.py", line 8418, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/PbsTestLab-1.0.0-py2.7.egg/ptl/lib/pbs_testlib.py", line 8418, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/PbsTestLab-1.0.0-py2.7.egg/ptl/lib/pbs_testlib.py", line 8418, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/PbsTestLab-1.0.0-py2.7.egg/ptl/lib/pbs_testlib.py", line 8418, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/PbsTestLab-1.0.0-py2.7.egg/ptl/lib/pbs_testlib.py", line 8418, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/PbsTestLab-1.0.0-py2.7.egg/ptl/lib/pbs_testlib.py", line 8418, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/PbsTestLab-1.0.0-py2.7.egg/ptl/lib/pbs_testlib.py", line 8418, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/PbsTestLab-1.0.0-py2.7.egg/ptl/lib/pbs_testlib.py", line 8418, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/PbsTestLab-1.0.0-py2.7.egg/ptl/lib/pbs_testlib.py", line 8418, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/PbsTestLab-1.0.0-py2.7.egg/ptl/lib/pbs_testlib.py", line 8418, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/PbsTestLab-1.0.0-py2.7.egg/ptl/lib/pbs_testlib.py", line 8418, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/PbsTestLab-1.0.0-py2.7.egg/ptl/lib/pbs_testlib.py", line 8418, in expect extend, level=level, msg=" ".join(msg)) File "/usr/lib/python2.7/site-packages/PbsTestLab-1.0.0-py2.7.egg/ptl/lib/pbs_testlib.py", line 8227, in expect raise PtlExpectError(rc=1, rv=False, msg=_msg) PtlExpectError: rc=1, rv=False, msg=expected on server centosvm: resources_used.foo_i unset job 6.centosvm attempt: 60 got: resources_used.foo_i = 3 2017-05-17 13:50:04,439 INFO ================================================================================ error: test_mom_down (tests.functional.pbs_accumulate_resc_used.TestPbsAccumulateRescUsed) Test cases with failures: TestPbsAccumulateRescUsed.test_mom_down Test suites with failures: TestPbsAccumulateRescUsed run: 13, succeeded: 12, failed: 0, errors: 1, skipped: 0, timedout: 0 Tests run in 0:12:23.162116 2017-05-17 13:50:04,439 INFO Cleaning up temporary files