2017-12-19 21:31:29,170 INFO input command: pbs_benchpress -f pbs_provisioning.py -p moms=c7-lrn -o app_prov.txt 2017-12-19 21:31:29,170 INFO param: moms=c7-lrn 2017-12-19 21:31:29,170 INFO ptl version: 1.0.0 2017-12-19 21:31:29,170 INFO platform: Linux c7-dev 3.10.0-229.el7.x86_64 #1 SMP Fri Mar 6 11:36:42 UTC 2015 x86_64 x86_64 2017-12-19 21:31:29,170 INFO python version: 2.7.5 2017-12-19 21:31:29,170 INFO user: root 2017-12-19 21:31:29,171 INFO -------------------------------------------------------------------------------- 2017-12-19 21:31:29,171 INFO Cleaning up temporary files 2017-12-19 21:31:29,180 INFO ====================================================================== 2017-12-19 21:31:29,180 INFO suite name: TestProvisioningJobWithHook 2017-12-19 21:31:29,180 INFO suite docstring: This testsuite tests whether OS provisioned jobs are getting all required hook files at MOM PRE: Have a cluster of PBS with a MOM installed on other than PBS server host. 1) Enable provisioning and set AOEs to the provisioning MOM 2) Create Hooks i) Provisioning hook which is facking OS provisioning. ii) execjob_begin hook prints a log message on MOM node. 3) Submit job with an aoe. 4) Node will go into the provisioning state while it is running provisioning hook. 5) Deletes execjob_begin hook file begin.PY from MOM 6) Restarts MOM as we are not doing actual OS provisioning 7) Then check for hook files are copied to the MOM node and job is printing log message from execjob_begin hook. 2017-12-19 21:31:29,180 INFO ====================================================================== 2017-12-19 21:31:29,180 INFO =============================================== 2017-12-19 21:31:29,180 INFO Entered TestProvisioningJobWithHook setUpClass 2017-12-19 21:31:29,180 INFO =============================================== 2017-12-19 21:31:29,180 INFOCLI2 c7-dev: id pbsuser 2017-12-19 21:31:29,185 INFOCLI2 c7-dev: id pbsuser1 2017-12-19 21:31:29,190 INFOCLI2 c7-dev: id pbsuser2 2017-12-19 21:31:29,196 INFOCLI2 c7-dev: id pbsuser3 2017-12-19 21:31:29,205 INFOCLI2 c7-dev: which cat 2017-12-19 21:31:29,220 INFO server c7-dev: server operating mode set to cli 2017-12-19 21:31:29,220 INFOCLI c7-dev: /opt/pbs/bin/qstat -Bf c7-dev 2017-12-19 21:31:29,247 INFO server c7-dev: version 17.1.0 2017-12-19 21:31:29,247 INFO expect action: created new action kicksched 2017-12-19 21:31:29,247 INFO expect action: added action kicksched to server c7-dev 2017-12-19 21:31:29,293 INFOCLI2 c7-dev: sudo -H /opt/pbs/sbin/pbsfs 2017-12-19 21:31:29,310 INFOCLI2 c7-dev: sudo -H /bin/cat /var/spool/pbs/sched_priv/resource_group 2017-12-19 21:31:29,366 INFOCLI c7-dev: /opt/pbs/bin/qmgr -c list sched 2017-12-19 21:31:29,387 INFOCLI2 c7-dev: sudo -H /bin/cat /var/spool/pbs/sched_priv/holidays 2017-12-19 21:31:29,408 INFOCLI2 c7-lrn: ssh c7-lrn python -c "import os;print [False, os.environ['PBS_CONF_FILE']]['PBS_CONF_FILE' in os.environ]" 2017-12-19 21:31:29,863 INFOCLI2 c7-lrn: ssh c7-lrn which cat 2017-12-19 21:31:31,127 INFO server c7-lrn: server operating mode set to cli 2017-12-19 21:31:31,128 INFOCLI c7-dev: /opt/pbs/bin/qstat -Bf c7-lrn 2017-12-19 21:31:31,206 ERROR err: ['Connection refused', 'qstat: cannot connect to server c7-lrn (errno=111)'] 2017-12-19 21:31:31,206 INFO server c7-lrn: version unknown 2017-12-19 21:31:32,127 INFO ================================================ 2017-12-19 21:31:32,127 INFO Completed TestProvisioningJobWithHook setUpClass 2017-12-19 21:31:32,127 INFO ================================================ 2017-12-19 21:31:32,127 INFO test name: test_app_provisioning (tests.functional.pbs_provisioning.TestProvisioningJobWithHook)... 2017-12-19 21:31:32,127 INFO test start time: Tue Dec 19 21:31:32 2017 2017-12-19 21:31:32,127 INFO test docstring: Test the application provisioning" 2017-12-19 21:31:32,128 INFOCLI2 c7-lrn: ssh c7-lrn which rm 2017-12-19 21:31:32,638 INFOCLI2 c7-lrn: ssh c7-lrn sudo -H /usr/bin/rm -rf /var/spool/pbs/mom_priv/config.d 2017-12-19 21:31:33,165 INFO c7-lrn 2017-12-19 21:31:33,165 INFO manager on c7-dev as root: delete node 2017-12-19 21:31:33,165 INFOCLI c7-dev: /opt/pbs/bin/qmgr -c delete node @default 2017-12-19 21:31:33,183 INFO manager on c7-dev: create node c7-lrn 2017-12-19 21:31:33,183 INFOCLI c7-dev: /opt/pbs/bin/qmgr -c create node c7-lrn 2017-12-19 21:31:33,202 INFO manager on c7-dev: set node c7-lrn {'provision_enable': 'true'} 2017-12-19 21:31:33,203 INFOCLI c7-dev: /opt/pbs/bin/qmgr -c set node c7-lrn provision_enable=true 2017-12-19 21:31:33,266 INFOCLI c7-dev: /opt/pbs/bin/pbsnodes -s c7-dev -v c7-lrn 2017-12-19 21:31:33,346 INFO server c7-lrn: server operating mode set to cli 2017-12-19 21:31:33,346 INFOCLI c7-dev: /opt/pbs/bin/qstat -Bf c7-lrn 2017-12-19 21:31:33,356 ERROR err: ['Connection refused', 'qstat: cannot connect to server c7-lrn (errno=111)'] 2017-12-19 21:31:33,356 INFO server c7-lrn: version unknown 2017-12-19 21:31:33,357 INFO expect on server c7-dev: provision_enable set true node c7-lrn ... OK 2017-12-19 21:31:33,357 INFO manager on c7-dev: set server {'log_events': 2047} 2017-12-19 21:31:33,357 INFOCLI c7-dev: /opt/pbs/bin/qmgr -c set server log_events=2047 2017-12-19 21:31:33,387 INFOCLI c7-dev: /opt/pbs/bin/qstat -Bf c7-dev 2017-12-19 21:31:33,402 INFO expect on server c7-dev: log_events set 2047 server c7-dev ... OK 2017-12-19 21:31:33,403 INFOCLI c7-dev: /opt/pbs/bin/pbsnodes -s c7-dev -v c7-lrn 2017-12-19 21:31:33,416 INFO expect on server c7-dev: state = free node c7-lrn got: state = state-unknown,down 2017-12-19 21:31:33,917 INFOCLI c7-dev: /opt/pbs/bin/pbsnodes -s c7-dev -v c7-lrn 2017-12-19 21:31:33,928 INFO expect on server c7-dev: state = free node c7-lrn attempt: 2 got: state = state-unknown,down 2017-12-19 21:31:34,429 INFOCLI c7-dev: /opt/pbs/bin/pbsnodes -s c7-dev -v c7-lrn 2017-12-19 21:31:34,453 INFO expect on server c7-dev: state = free node c7-lrn attempt: 3 got: state = state-unknown,down 2017-12-19 21:31:34,954 INFOCLI c7-dev: /opt/pbs/bin/pbsnodes -s c7-dev -v c7-lrn 2017-12-19 21:31:34,995 INFO expect on server c7-dev: state = free node c7-lrn attempt: 4 got: state = state-unknown,down 2017-12-19 21:31:35,497 INFOCLI c7-dev: /opt/pbs/bin/pbsnodes -s c7-dev -v c7-lrn 2017-12-19 21:31:35,536 INFO expect on server c7-dev: state = free node c7-lrn attempt: 5 got: state = state-unknown,down 2017-12-19 21:31:36,038 INFOCLI c7-dev: /opt/pbs/bin/pbsnodes -s c7-dev -v c7-lrn 2017-12-19 21:31:36,049 INFO expect on server c7-dev: state = free node c7-lrn attempt: 6 ... OK 2017-12-19 21:31:36,050 INFO status on c7-dev: hook 2017-12-19 21:31:36,050 INFO manager on c7-dev: list hook 2017-12-19 21:31:36,050 INFOCLI c7-dev: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-12-19 21:31:36,086 INFO status on c7-dev: hook 2017-12-19 21:31:36,087 INFO manager on c7-dev: list hook 2017-12-19 21:31:36,087 INFOCLI c7-dev: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-12-19 21:31:36,110 INFO manager on c7-dev: create hook begin 2017-12-19 21:31:36,110 INFOCLI c7-dev: sudo -H /opt/pbs/bin/qmgr -c create hook begin 2017-12-19 21:31:36,137 INFO manager on c7-dev: set hook begin {'enabled': 'True', 'event': 'execjob_begin'} 2017-12-19 21:31:36,137 INFOCLI c7-dev: sudo -H /opt/pbs/bin/qmgr -c set hook begin enabled=True,event=execjob_begin 2017-12-19 21:31:36,202 INFOCLI c7-dev: sudo -H /opt/pbs/bin/qmgr -c list hook begin 2017-12-19 21:31:36,275 INFO expect on server c7-dev: enabled set True || event set execjob_begin hook begin ... OK 2017-12-19 21:31:36,277 INFO manager on c7-dev: import hook begin {'content-type': 'application/x-python', 'input-file': '/tmp/PtlPbsjYna8I', 'content-encoding': 'default'} 2017-12-19 21:31:36,277 INFOCLI c7-dev: sudo -H /opt/pbs/bin/qmgr -c import hook begin application/x-python default /tmp/PtlPbsjYna8I 2017-12-19 21:31:36,307 INFO server c7-dev: imported hook body --- import pbs pbs.logmsg(pbs.LOG_DEBUG, "executed execjob_begin hook on job %s" % (pbs.event().job.id)) --- 2017-12-19 21:31:36,308 INFO status on c7-dev: hook 2017-12-19 21:31:36,308 INFO manager on c7-dev: list hook 2017-12-19 21:31:36,308 INFOCLI c7-dev: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-12-19 21:31:36,435 INFO status on c7-dev: hook 2017-12-19 21:31:36,435 INFO manager on c7-dev: list hook 2017-12-19 21:31:36,436 INFOCLI c7-dev: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-12-19 21:31:36,541 INFO manager on c7-dev: create hook my_provisioning 2017-12-19 21:31:36,541 INFOCLI c7-dev: sudo -H /opt/pbs/bin/qmgr -c create hook my_provisioning 2017-12-19 21:31:36,571 INFO manager on c7-dev: set hook my_provisioning {'alarm': '300', 'enabled': 'True', 'event': 'provision'} 2017-12-19 21:31:36,572 INFOCLI c7-dev: sudo -H /opt/pbs/bin/qmgr -c set hook my_provisioning alarm=300,enabled=True,event=provision 2017-12-19 21:31:36,609 INFOCLI c7-dev: sudo -H /opt/pbs/bin/qmgr -c list hook my_provisioning 2017-12-19 21:31:36,639 INFO expect on server c7-dev: alarm set 300 || enabled set True || event set provision hook my_provisioning ... OK 2017-12-19 21:31:36,640 INFO manager on c7-dev: import hook my_provisioning {'content-type': 'application/x-python', 'input-file': '/tmp/PtlPbsnFCUwG', 'content-encoding': 'default'} 2017-12-19 21:31:36,641 INFOCLI c7-dev: sudo -H /opt/pbs/bin/qmgr -c import hook my_provisioning application/x-python default /tmp/PtlPbsnFCUwG 2017-12-19 21:31:36,673 INFO server c7-dev: imported hook body --- import pbs import time e = pbs.event() vnode = e.vnode aoe = e.aoe if (aoe == 'App1'): pbs.logmsg(pbs.LOG_DEBUG, "fake Application provisioning script") e.accept(1) pbs.logmsg(pbs.LOG_DEBUG, "aoe=%s,vnode=%s" % (aoe,vnode)) pbs.logmsg(pbs.LOG_DEBUG, "fake OS provisioning script") e.accept(0) --- 2017-12-19 21:31:36,673 INFO manager on c7-dev: set node c7-lrn {'resources_available.aoe': 'App1'} 2017-12-19 21:31:36,673 INFOCLI c7-dev: /opt/pbs/bin/qmgr -c set node c7-lrn resources_available.aoe=App1 2017-12-19 21:31:36,695 INFOCLI c7-dev: /opt/pbs/bin/pbsnodes -s c7-dev -v c7-lrn 2017-12-19 21:31:36,717 INFO expect on server c7-dev: resources_available.aoe set App1 node c7-lrn ... OK 2017-12-19 21:31:36,718 INFO job: executable set to /bin/sleep with arguments: 100 2017-12-19 21:31:36,719 INFO job: executable set to /bin/sleep with arguments: 1 2017-12-19 21:31:36,719 INFOCLI c7-dev: sudo -H -u pbsuser1 /opt/pbs/bin/qsub -l aoe=App1 -- /bin/sleep 1 2017-12-19 21:31:36,804 INFO submit to c7-dev as pbsuser1: job 58.c7-dev OrderedDict([('Resource_List', 'aoe=App1')]) 2017-12-19 21:31:36,804 INFOCLI c7-dev: /opt/pbs/bin/qstat -f 58.c7-dev 2017-12-19 21:31:36,822 INFO expect on server c7-dev: job_state = R job 58.c7-dev got: job_state = Q 2017-12-19 21:31:37,324 INFOCLI c7-dev: /opt/pbs/bin/qmgr -c set server scheduling=True 2017-12-19 21:31:37,380 INFOCLI c7-dev: /opt/pbs/bin/qstat -f 58.c7-dev 2017-12-19 21:31:37,397 INFO expect on server c7-dev: job_state = R job 58.c7-dev attempt: 2 ... OK 2017-12-19 21:31:37,397 INFO server c7-dev log match: searching for "fake Application provisioning script"... OK 2017-12-19 21:31:37,397 INFO delete on c7-dev: 58.c7-dev with force 2017-12-19 21:31:37,397 INFO delete job on c7-dev: 58.c7-dev 2017-12-19 21:31:37,397 INFOCLI c7-dev: /opt/pbs/bin/qdel -W force 58.c7-dev 2017-12-19 21:31:37,460 INFO manager on c7-dev: delete hook begin 2017-12-19 21:31:37,460 INFOCLI c7-dev: sudo -H /opt/pbs/bin/qmgr -c delete hook begin 2017-12-19 21:31:37,599 INFO manager on c7-dev: delete hook my_provisioning 2017-12-19 21:31:37,599 INFOCLI c7-dev: sudo -H /opt/pbs/bin/qmgr -c delete hook my_provisioning 2017-12-19 21:31:37,620 INFO ok 2017-12-19 21:31:37,620 INFO test name: test_execjob_begin_hook_on_os_provisioned_job (tests.functional.pbs_provisioning.TestProvisioningJobWithHook)... 2017-12-19 21:31:37,620 INFO test start time: Tue Dec 19 21:31:37 2017 2017-12-19 21:31:37,620 INFO test docstring: Test the execjob_begin hook is seen by OS provisioned job. 2017-12-19 21:31:37,620 INFOCLI2 c7-lrn: ssh c7-lrn sudo -H /usr/bin/rm -rf /var/spool/pbs/mom_priv/config.d 2017-12-19 21:31:37,976 INFO c7-lrn 2017-12-19 21:31:37,976 INFO manager on c7-dev as root: delete node 2017-12-19 21:31:37,976 INFOCLI c7-dev: /opt/pbs/bin/qmgr -c delete node @default 2017-12-19 21:31:38,064 INFO manager on c7-dev: create node c7-lrn 2017-12-19 21:31:38,064 INFOCLI c7-dev: /opt/pbs/bin/qmgr -c create node c7-lrn 2017-12-19 21:31:38,109 INFO manager on c7-dev: set node c7-lrn {'provision_enable': 'true'} 2017-12-19 21:31:38,110 INFOCLI c7-dev: /opt/pbs/bin/qmgr -c set node c7-lrn provision_enable=true 2017-12-19 21:31:38,130 INFOCLI c7-dev: /opt/pbs/bin/pbsnodes -s c7-dev -v c7-lrn 2017-12-19 21:31:38,148 INFO expect on server c7-dev: provision_enable set true node c7-lrn ... OK 2017-12-19 21:31:38,149 INFO manager on c7-dev: set server {'log_events': 2047} 2017-12-19 21:31:38,149 INFOCLI c7-dev: /opt/pbs/bin/qmgr -c set server log_events=2047 2017-12-19 21:31:38,181 INFOCLI c7-dev: /opt/pbs/bin/qstat -Bf c7-dev 2017-12-19 21:31:38,198 INFO expect on server c7-dev: log_events set 2047 server c7-dev ... OK 2017-12-19 21:31:38,198 INFOCLI c7-dev: /opt/pbs/bin/pbsnodes -s c7-dev -v c7-lrn 2017-12-19 21:31:38,216 INFO expect on server c7-dev: state = free node c7-lrn got: state = state-unknown,down 2017-12-19 21:31:38,718 INFOCLI c7-dev: /opt/pbs/bin/pbsnodes -s c7-dev -v c7-lrn 2017-12-19 21:31:38,783 INFO expect on server c7-dev: state = free node c7-lrn attempt: 2 got: state = state-unknown,down 2017-12-19 21:31:39,284 INFOCLI c7-dev: /opt/pbs/bin/pbsnodes -s c7-dev -v c7-lrn 2017-12-19 21:31:39,330 INFO expect on server c7-dev: state = free node c7-lrn attempt: 3 got: state = state-unknown,down 2017-12-19 21:31:39,831 INFOCLI c7-dev: /opt/pbs/bin/pbsnodes -s c7-dev -v c7-lrn 2017-12-19 21:31:39,863 INFO expect on server c7-dev: state = free node c7-lrn attempt: 4 got: state = state-unknown,down 2017-12-19 21:31:40,364 INFOCLI c7-dev: /opt/pbs/bin/pbsnodes -s c7-dev -v c7-lrn 2017-12-19 21:31:40,404 INFO expect on server c7-dev: state = free node c7-lrn attempt: 5 got: state = state-unknown,down 2017-12-19 21:31:40,906 INFOCLI c7-dev: /opt/pbs/bin/pbsnodes -s c7-dev -v c7-lrn 2017-12-19 21:31:40,942 INFO expect on server c7-dev: state = free node c7-lrn attempt: 6 ... OK 2017-12-19 21:31:40,943 INFO status on c7-dev: hook 2017-12-19 21:31:40,943 INFO manager on c7-dev: list hook 2017-12-19 21:31:40,944 INFOCLI c7-dev: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-12-19 21:31:40,974 INFO status on c7-dev: hook 2017-12-19 21:31:40,975 INFO manager on c7-dev: list hook 2017-12-19 21:31:40,975 INFOCLI c7-dev: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-12-19 21:31:41,029 INFO manager on c7-dev: create hook begin 2017-12-19 21:31:41,029 INFOCLI c7-dev: sudo -H /opt/pbs/bin/qmgr -c create hook begin 2017-12-19 21:31:41,107 INFO manager on c7-dev: set hook begin {'enabled': 'True', 'event': 'execjob_begin'} 2017-12-19 21:31:41,107 INFOCLI c7-dev: sudo -H /opt/pbs/bin/qmgr -c set hook begin enabled=True,event=execjob_begin 2017-12-19 21:31:41,148 INFOCLI c7-dev: sudo -H /opt/pbs/bin/qmgr -c list hook begin 2017-12-19 21:31:41,241 INFO expect on server c7-dev: enabled set True || event set execjob_begin hook begin ... OK 2017-12-19 21:31:41,242 INFO manager on c7-dev: import hook begin {'content-type': 'application/x-python', 'input-file': '/tmp/PtlPbsbO2CTl', 'content-encoding': 'default'} 2017-12-19 21:31:41,242 INFOCLI c7-dev: sudo -H /opt/pbs/bin/qmgr -c import hook begin application/x-python default /tmp/PtlPbsbO2CTl 2017-12-19 21:31:41,280 INFO server c7-dev: imported hook body --- import pbs pbs.logmsg(pbs.LOG_DEBUG, "executed execjob_begin hook on job %s" % (pbs.event().job.id)) --- 2017-12-19 21:31:41,281 INFO status on c7-dev: hook 2017-12-19 21:31:41,281 INFO manager on c7-dev: list hook 2017-12-19 21:31:41,281 INFOCLI c7-dev: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-12-19 21:31:41,327 INFO status on c7-dev: hook 2017-12-19 21:31:41,328 INFO manager on c7-dev: list hook 2017-12-19 21:31:41,328 INFOCLI c7-dev: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-12-19 21:31:41,457 INFO manager on c7-dev: create hook my_provisioning 2017-12-19 21:31:41,458 INFOCLI c7-dev: sudo -H /opt/pbs/bin/qmgr -c create hook my_provisioning 2017-12-19 21:31:41,554 INFO manager on c7-dev: set hook my_provisioning {'alarm': '300', 'enabled': 'True', 'event': 'provision'} 2017-12-19 21:31:41,554 INFOCLI c7-dev: sudo -H /opt/pbs/bin/qmgr -c set hook my_provisioning alarm=300,enabled=True,event=provision 2017-12-19 21:31:41,591 INFOCLI c7-dev: sudo -H /opt/pbs/bin/qmgr -c list hook my_provisioning 2017-12-19 21:31:41,633 INFO expect on server c7-dev: alarm set 300 || enabled set True || event set provision hook my_provisioning ... OK 2017-12-19 21:31:41,634 INFO manager on c7-dev: import hook my_provisioning {'content-type': 'application/x-python', 'input-file': '/tmp/PtlPbsFBg1MM', 'content-encoding': 'default'} 2017-12-19 21:31:41,634 INFOCLI c7-dev: sudo -H /opt/pbs/bin/qmgr -c import hook my_provisioning application/x-python default /tmp/PtlPbsFBg1MM 2017-12-19 21:31:41,670 INFO server c7-dev: imported hook body --- import pbs import time e = pbs.event() vnode = e.vnode aoe = e.aoe if (aoe == 'App1'): pbs.logmsg(pbs.LOG_DEBUG, "fake Application provisioning script") e.accept(1) pbs.logmsg(pbs.LOG_DEBUG, "aoe=%s,vnode=%s" % (aoe,vnode)) pbs.logmsg(pbs.LOG_DEBUG, "fake OS provisioning script") e.accept(0) --- 2017-12-19 21:31:41,671 INFO manager on c7-dev: set node c7-lrn {'resources_available.aoe': 'osimage1'} 2017-12-19 21:31:41,671 INFOCLI c7-dev: /opt/pbs/bin/qmgr -c set node c7-lrn resources_available.aoe=osimage1 2017-12-19 21:31:41,692 INFOCLI c7-dev: /opt/pbs/bin/pbsnodes -s c7-dev -v c7-lrn 2017-12-19 21:31:41,706 INFO expect on server c7-dev: resources_available.aoe set osimage1 node c7-lrn ... OK 2017-12-19 21:31:41,707 INFO job: executable set to /bin/sleep with arguments: 100 2017-12-19 21:31:41,707 INFO job: executable set to /bin/sleep with arguments: 1 2017-12-19 21:31:41,707 INFOCLI c7-dev: sudo -H -u pbsuser1 /opt/pbs/bin/qsub -l aoe=osimage1 -- /bin/sleep 1 2017-12-19 21:31:41,735 INFO submit to c7-dev as pbsuser1: job 59.c7-dev OrderedDict([('Resource_List', 'aoe=osimage1')]) 2017-12-19 21:31:41,736 INFOCLI c7-dev: /opt/pbs/bin/pbsnodes -s c7-dev -v c7-lrn 2017-12-19 21:31:41,762 INFO expect on server c7-dev: state = provisioning node c7-lrn got: state = free 2017-12-19 21:31:42,264 INFOCLI c7-dev: /opt/pbs/bin/pbsnodes -s c7-dev -v c7-lrn 2017-12-19 21:31:42,306 INFO expect on server c7-dev: state = provisioning node c7-lrn attempt: 2 ... OK 2017-12-19 21:31:42,308 INFOCLI2 c7-lrn: ssh c7-lrn sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/hooks/begin.PY 2017-12-19 21:31:42,667 INFO mom c7-lrn: stopping MoM on host c7-lrn 2017-12-19 21:31:42,667 INFOCLI2 c7-lrn: ssh c7-lrn sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-12-19 21:31:43,089 INFOCLI2 c7-lrn: ssh c7-lrn sudo -H kill -TERM 37745 2017-12-19 21:31:43,554 INFOCLI2 c7-lrn: ssh c7-lrn sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-12-19 21:31:43,934 INFOCLI2 c7-lrn: ssh c7-lrn which python 2017-12-19 21:31:44,415 INFOCLI2 c7-lrn: ssh c7-lrn /usr/bin/python -c "import sys; print sys.platform" 2017-12-19 21:31:45,231 INFO service: starting pbs_mom -d /var/spool/pbs 2017-12-19 21:31:45,231 INFOCLI c7-lrn: ssh c7-lrn sudo -H /opt/pbs/sbin/pbs_mom -d /var/spool/pbs 2017-12-19 21:31:45,668 INFOCLI2 c7-lrn: ssh c7-lrn sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-12-19 21:31:51,556 INFO server c7-dev log match: searching for "successfully sent hook file /var/spool/pbs/server_priv/hooks/begin.PY"... OK 2017-12-19 21:31:51,942 INFO mom c7-lrn log match: searching for "begin.PY;copy hook-related file request received" - using regular expression ... OK 2017-12-19 21:31:52,433 INFO mom c7-lrn log match: searching for "executed execjob_begin hook on job 59.c7-dev" - using regular expression ... OK 2017-12-19 21:31:52,433 INFO delete on c7-dev: 59.c7-dev with force 2017-12-19 21:31:52,433 INFO delete job on c7-dev: 59.c7-dev 2017-12-19 21:31:52,433 INFOCLI c7-dev: /opt/pbs/bin/qdel -W force 59.c7-dev 2017-12-19 21:31:52,524 INFO manager on c7-dev: delete hook begin 2017-12-19 21:31:52,524 INFOCLI c7-dev: sudo -H /opt/pbs/bin/qmgr -c delete hook begin 2017-12-19 21:31:52,603 INFO manager on c7-dev: delete hook my_provisioning 2017-12-19 21:31:52,603 INFOCLI c7-dev: sudo -H /opt/pbs/bin/qmgr -c delete hook my_provisioning 2017-12-19 21:31:52,665 INFO ok 2017-12-19 21:31:52,666 INFO ================================================================================ run: 2, succeeded: 2, failed: 0, errors: 0, skipped: 0, timedout: 0 Tests run in 0:00:23.486610 2017-12-19 21:31:52,667 INFO Cleaning up temporary files