2017-07-13 16:34:05,787 INFO input command: pbs_benchpress -f pbs_provisioning.py -p moms=c7-lrn -o result.txt 2017-07-13 16:34:05,788 INFO param: moms=c7-lrn 2017-07-13 16:34:05,788 INFO ptl version: 1.0.0 2017-07-13 16:34:05,788 INFO platform: Linux c7-dev 3.10.0-229.el7.x86_64 #1 SMP Fri Mar 6 11:36:42 UTC 2015 x86_64 x86_64 2017-07-13 16:34:05,788 INFO python version: 2.7.5 2017-07-13 16:34:05,788 INFO user: root 2017-07-13 16:34:05,788 INFO -------------------------------------------------------------------------------- 2017-07-13 16:34:05,788 INFO Cleaning up temporary files 2017-07-13 16:34:05,797 INFO ====================================================================== 2017-07-13 16:34:05,797 INFO suite name: TestProvisioningJobWithHook 2017-07-13 16:34:05,797 INFO suite docstring: This testsuite tests whether OS provisioned jobs are getting all required hook files at MOM PRE: Have a cluster of PBS with a MOM installed on other than PBS server host. 1) Enable provisioning and set AOEs to the provisioning MOM 2) Create Hooks i) Provisioning hook which is facking OS provisioning. ii) execjob_begin hook prints a log message on MOM node. 3) Submit job with an aoe. 4) Node will go into the provisioning state while it is running provisioning hook. 5) Deletes execjob_begin hook file begin.PY from MOM 6) Restarts MOM as we are not doing actual OS provisioning 7) Then check for hook files are copied to the MOM node and job is printing log message from execjob_begin hook. 2017-07-13 16:34:05,797 INFO ====================================================================== 2017-07-13 16:34:05,797 INFO =============================================== 2017-07-13 16:34:05,797 INFO Entered TestProvisioningJobWithHook setUpClass 2017-07-13 16:34:05,797 INFO =============================================== 2017-07-13 16:34:05,798 INFOCLI2 c7-dev: id pbsuser 2017-07-13 16:34:05,802 INFOCLI2 c7-dev: id pbsuser1 2017-07-13 16:34:05,805 INFOCLI2 c7-dev: id pbsuser2 2017-07-13 16:34:05,809 INFOCLI2 c7-dev: id pbsuser3 2017-07-13 16:34:05,812 INFOCLI2 c7-dev: which cat 2017-07-13 16:34:05,843 INFO server c7-dev: server operating mode set to cli 2017-07-13 16:34:05,843 INFOCLI c7-dev: /opt/pbs/bin/qstat -Bf c7-dev 2017-07-13 16:34:05,864 INFO server c7-dev: version 14.0.1 2017-07-13 16:34:05,864 INFO expect action: created new action kicksched 2017-07-13 16:34:05,864 INFO expect action: added action kicksched to server c7-dev 2017-07-13 16:34:05,904 INFOCLI2 c7-dev: sudo -H /opt/pbs/sbin/pbsfs 2017-07-13 16:34:05,924 INFOCLI2 c7-dev: sudo -H /bin/cat /var/spool/pbs/sched_priv/resource_group 2017-07-13 16:34:05,940 INFOCLI c7-dev: /opt/pbs/bin/qmgr -c list sched 2017-07-13 16:34:05,974 INFOCLI2 c7-dev: sudo -H /bin/cat /var/spool/pbs/sched_priv/holidays 2017-07-13 16:34:05,999 INFOCLI2 c7-lrn: ssh c7-lrn python -c "import os;print [False, os.environ['PBS_CONF_FILE']]['PBS_CONF_FILE' in os.environ]" 2017-07-13 16:34:06,414 INFOCLI2 c7-lrn: ssh c7-lrn which cat 2017-07-13 16:34:07,325 INFO server c7-lrn: server operating mode set to cli 2017-07-13 16:34:07,325 INFOCLI c7-dev: /opt/pbs/bin/qstat -Bf c7-lrn 2017-07-13 16:34:07,333 ERROR err: ['Connection refused', 'qstat: cannot connect to server c7-lrn (errno=111)'] 2017-07-13 16:34:07,334 INFO server c7-lrn: version unknown 2017-07-13 16:34:07,960 INFO ================================================ 2017-07-13 16:34:07,960 INFO Completed TestProvisioningJobWithHook setUpClass 2017-07-13 16:34:07,961 INFO ================================================ 2017-07-13 16:34:07,961 INFO test name: test_execjob_begin_hook_on_provisioned_job (tests.functional.pbs_provisioning.TestProvisioningJobWithHook)... 2017-07-13 16:34:07,961 INFO test start time: Thu Jul 13 16:34:07 2017 2017-07-13 16:34:07,961 INFO test docstring: Test the execjob_begin hook is seen by OS provisioned job. 2017-07-13 16:34:07,961 INFOCLI2 c7-lrn: ssh c7-lrn which rm 2017-07-13 16:34:08,302 INFOCLI2 c7-lrn: ssh c7-lrn sudo -H /usr/bin/rm -rf /var/spool/pbs/mom_priv/config.d 2017-07-13 16:34:08,696 INFO c7-lrn 2017-07-13 16:34:08,696 INFO manager on c7-dev: delete node 2017-07-13 16:34:08,696 INFOCLI c7-dev: /opt/pbs/bin/qmgr -c delete node @default 2017-07-13 16:34:08,769 INFO manager on c7-dev: create node c7-lrn 2017-07-13 16:34:08,769 INFOCLI c7-dev: /opt/pbs/bin/qmgr -c create node c7-lrn 2017-07-13 16:34:08,842 INFO manager on c7-dev: set node c7-lrn {'provision_enable': 'true'} 2017-07-13 16:34:08,842 INFOCLI c7-dev: /opt/pbs/bin/qmgr -c set node c7-lrn provision_enable=true 2017-07-13 16:34:08,897 INFOCLI c7-dev: /opt/pbs/bin/pbsnodes -s c7-dev -v c7-lrn 2017-07-13 16:34:08,938 INFO server c7-lrn: server operating mode set to cli 2017-07-13 16:34:08,938 INFOCLI c7-dev: /opt/pbs/bin/qstat -Bf c7-lrn 2017-07-13 16:34:08,951 ERROR err: ['Connection refused', 'qstat: cannot connect to server c7-lrn (errno=111)'] 2017-07-13 16:34:08,951 INFO server c7-lrn: version unknown 2017-07-13 16:34:08,952 INFO expect on server c7-dev: provision_enable set true node c7-lrn ... OK 2017-07-13 16:34:08,952 INFO manager on c7-dev: set node c7-lrn {'resources_available.aoe': 'osimage1'} 2017-07-13 16:34:08,952 INFOCLI c7-dev: /opt/pbs/bin/qmgr -c set node c7-lrn resources_available.aoe=osimage1 2017-07-13 16:34:08,979 INFOCLI c7-dev: /opt/pbs/bin/pbsnodes -s c7-dev -v c7-lrn 2017-07-13 16:34:09,012 INFO expect on server c7-dev: resources_available.aoe set osimage1 node c7-lrn ... OK 2017-07-13 16:34:09,012 INFO manager on c7-dev: set server {'log_events': 2047} 2017-07-13 16:34:09,013 INFOCLI c7-dev: /opt/pbs/bin/qmgr -c set server log_events=2047 2017-07-13 16:34:09,052 INFOCLI c7-dev: /opt/pbs/bin/qstat -Bf c7-dev 2017-07-13 16:34:09,079 INFO expect on server c7-dev: log_events set 2047 server c7-dev ... OK 2017-07-13 16:34:09,079 INFOCLI c7-dev: /opt/pbs/bin/pbsnodes -s c7-dev -v c7-lrn 2017-07-13 16:34:09,119 INFO expect on server c7-dev: state = free node c7-lrn got: state = state-unknown,down 2017-07-13 16:34:09,621 INFOCLI c7-dev: /opt/pbs/bin/pbsnodes -s c7-dev -v c7-lrn 2017-07-13 16:34:09,673 INFO expect on server c7-dev: state = free node c7-lrn attempt: 2 got: state = state-unknown,down 2017-07-13 16:34:10,175 INFOCLI c7-dev: /opt/pbs/bin/pbsnodes -s c7-dev -v c7-lrn 2017-07-13 16:34:10,231 INFO expect on server c7-dev: state = free node c7-lrn attempt: 3 got: state = state-unknown,down 2017-07-13 16:34:10,733 INFOCLI c7-dev: /opt/pbs/bin/pbsnodes -s c7-dev -v c7-lrn 2017-07-13 16:34:10,807 INFO expect on server c7-dev: state = free node c7-lrn attempt: 4 ... OK 2017-07-13 16:34:10,807 INFO status on c7-dev: hook 2017-07-13 16:34:10,807 INFO manager on c7-dev: list hook 2017-07-13 16:34:10,808 INFOCLI c7-dev: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-07-13 16:34:10,870 INFO manager on c7-dev: set hook begin {'enabled': 'True', 'event': 'execjob_begin'} 2017-07-13 16:34:10,871 INFOCLI c7-dev: sudo -H /opt/pbs/bin/qmgr -c set hook begin enabled=True,event=execjob_begin 2017-07-13 16:34:11,014 INFO manager on c7-dev: import hook begin {'content-type': 'application/x-python', 'input-file': '/tmp/PtlPbsbTfMXP', 'content-encoding': 'default'} 2017-07-13 16:34:11,015 INFOCLI c7-dev: sudo -H /opt/pbs/bin/qmgr -c import hook begin application/x-python default /tmp/PtlPbsbTfMXP 2017-07-13 16:34:11,157 ERROR err: ["hook 'begin' contents overwritten"] 2017-07-13 16:34:11,157 INFO server c7-dev: imported hook body --- import pbs pbs.logmsg(pbs.LOG_DEBUG, "executed execjob_begin hook on job %s" % (pbs.event().job.id)) --- 2017-07-13 16:34:11,157 INFO status on c7-dev: hook 2017-07-13 16:34:11,157 INFO manager on c7-dev: list hook 2017-07-13 16:34:11,158 INFOCLI c7-dev: sudo -H /opt/pbs/bin/qmgr -c list hook 2017-07-13 16:34:11,250 INFO manager on c7-dev: set hook my_provisioning {'alarm': '300', 'enabled': 'True', 'event': 'provision'} 2017-07-13 16:34:11,250 INFOCLI c7-dev: sudo -H /opt/pbs/bin/qmgr -c set hook my_provisioning alarm=300,enabled=True,event=provision 2017-07-13 16:34:11,298 INFO manager on c7-dev: import hook my_provisioning {'content-type': 'application/x-python', 'input-file': '/tmp/PtlPbszXR7ZB', 'content-encoding': 'default'} 2017-07-13 16:34:11,298 INFOCLI c7-dev: sudo -H /opt/pbs/bin/qmgr -c import hook my_provisioning application/x-python default /tmp/PtlPbszXR7ZB 2017-07-13 16:34:11,352 ERROR err: ["hook 'my_provisioning' contents overwritten"] 2017-07-13 16:34:11,352 INFO server c7-dev: imported hook body --- import pbs import time e = pbs.event() vnode = e.vnode aoe = e.aoe pbs.logmsg(pbs.LOG_DEBUG, "aoe=%s,vnode=%s" % (aoe,vnode)) pbs.logmsg(pbs.LOG_DEBUG, "fake calling os provisiong script") e.accept(0) --- 2017-07-13 16:34:11,353 INFO job: executable set to /bin/sleep with arguments: 100 2017-07-13 16:34:11,353 INFO job: executable set to /bin/sleep with arguments: 1 2017-07-13 16:34:11,353 INFOCLI c7-dev: sudo -H -u pbsuser1 /opt/pbs/bin/qsub -l aoe=osimage1 -- /bin/sleep 1 2017-07-13 16:34:11,404 INFO submit to c7-dev as pbsuser1: job 9.c7-dev OrderedDict([('Resource_List', 'aoe=osimage1')]) 2017-07-13 16:34:11,404 INFOCLI c7-dev: /opt/pbs/bin/pbsnodes -s c7-dev -v c7-lrn 2017-07-13 16:34:11,475 INFO expect on server c7-dev: state = provisioning node c7-lrn got: state = free 2017-07-13 16:34:11,976 INFOCLI c7-dev: /opt/pbs/bin/pbsnodes -s c7-dev -v c7-lrn 2017-07-13 16:34:12,001 INFO expect on server c7-dev: state = provisioning node c7-lrn attempt: 2 ... OK 2017-07-13 16:34:12,002 INFOCLI2 c7-lrn: ssh c7-lrn sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/hooks/begin.PY 2017-07-13 16:34:12,324 INFO mom c7-lrn: stopping MoM on host c7-lrn 2017-07-13 16:34:12,325 INFOCLI2 c7-lrn: ssh c7-lrn sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-07-13 16:34:12,641 INFOCLI2 c7-lrn: ssh c7-lrn sudo -H kill -TERM 45435 2017-07-13 16:34:12,982 INFOCLI2 c7-lrn: ssh c7-lrn sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-07-13 16:34:13,464 INFOCLI2 c7-lrn: ssh c7-lrn which python 2017-07-13 16:34:13,755 INFOCLI2 c7-lrn: ssh c7-lrn /usr/bin/python -c "import sys; print sys.platform" 2017-07-13 16:34:14,393 INFO service: starting pbs_mom -d /var/spool/pbs 2017-07-13 16:34:14,393 INFOCLI c7-lrn: ssh c7-lrn sudo -H /opt/pbs/sbin/pbs_mom -d /var/spool/pbs 2017-07-13 16:34:14,844 INFOCLI2 c7-lrn: ssh c7-lrn sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2017-07-13 16:34:20,817 INFO server c7-dev log match: searching for "successfully sent hook file /var/spool/pbs/server_priv/hooks/begin.PY"... OK 2017-07-13 16:34:21,175 INFO mom c7-lrn log match: searching for "begin.PY;copy hook-related file request received" - using regular expression ... OK 2017-07-13 16:34:21,480 INFO mom c7-lrn log match: searching for "executed execjob_begin hook on job 9.c7-dev" - using regular expression ... OK 2017-07-13 16:34:21,480 INFO ============================================= 2017-07-13 16:34:21,480 INFO Entered TestProvisioningJobWithHook tearDown 2017-07-13 16:34:21,480 INFO ============================================= 2017-07-13 16:34:21,480 INFO ============================================== 2017-07-13 16:34:21,481 INFO Completed TestProvisioningJobWithHook tearDown 2017-07-13 16:34:21,481 INFO ============================================== 2017-07-13 16:34:21,481 INFO ok 2017-07-13 16:34:21,481 INFO ================================================================================ run: 1, succeeded: 1, failed: 0, errors: 0, skipped: 0, timedout: 0 Tests run in 0:00:15.684683 2017-07-13 16:34:21,481 INFO Cleaning up temporary files