2018-12-07 09:11:22,309 INFO     test name: test_fairshare (tests.pbs_smoketest.SmokeTest)...
2018-12-07 09:11:22,309 INFO     test start time: Fri Dec  7 09:11:22 2018
2018-12-07 09:11:22,309 INFO     test docstring: 

        Test for fairshare
        
2018-12-07 09:11:22,309 INFO     ========================
2018-12-07 09:11:22,310 INFO      Entered SmokeTest setUp
2018-12-07 09:11:22,310 INFO     ========================
2018-12-07 09:11:22,310 INFOCLI  testdev: /opt/pbs/bin/qstat -Bf testdev.pbspro.org
2018-12-07 09:11:22,316 INFO     status on testdev: server
2018-12-07 09:11:22,316 INFOCLI  testdev: /opt/pbs/bin/qstat -Bf testdev.pbspro.org
2018-12-07 09:11:22,322 INFO     manager on testdev: unset server managers
2018-12-07 09:11:22,322 INFOCLI  testdev: sudo -H /opt/pbs/bin/qmgr -c unset server managers
2018-12-07 09:11:22,334 INFOCLI  testdev: /opt/pbs/bin/qstat -Bf testdev.pbspro.org
2018-12-07 09:11:22,340 INFO     expect on server testdev: managers unset  server testdev.pbspro.org ...  OK
2018-12-07 09:11:22,340 INFO     manager on testdev: set server {'managers': (2, 'root@*')}
2018-12-07 09:11:22,341 INFOCLI  testdev: sudo -H /opt/pbs/bin/qmgr -c set server managers+=root@*
2018-12-07 09:11:22,351 INFO     server testdev: reverting configuration to defaults
2018-12-07 09:11:22,351 INFOCLI  testdev: /opt/pbs/bin/qstat -Bf testdev.pbspro.org
2018-12-07 09:11:22,357 INFO     select on testdev: __ALL__
2018-12-07 09:11:22,357 INFOCLI  testdev: /opt/pbs/bin/qselect
2018-12-07 09:11:22,363 INFOCLI  testdev: /opt/pbs/bin/qstat -f @testdev.pbspro.org
2018-12-07 09:11:22,368 INFO     expect on server testdev: job_state set 0 job ...  OK
2018-12-07 09:11:22,368 INFOCLI  testdev: /opt/pbs/bin/pbs_rstat -f
2018-12-07 09:11:22,375 INFO     delete resv on testdev: R17.testdev
2018-12-07 09:11:22,376 INFOCLI  testdev: /opt/pbs/bin/pbs_rdel R17.testdev
2018-12-07 09:11:22,386 INFOCLI  testdev: /opt/pbs/bin/pbs_rstat -f
2018-12-07 09:11:22,393 INFO     manager on testdev: unset server ['reserve_retry_init', 'reserve_retry_cutoff']
2018-12-07 09:11:22,394 INFOCLI  testdev: /opt/pbs/bin/qmgr -c unset server reserve_retry_init,reserve_retry_cutoff
2018-12-07 09:11:22,403 INFOCLI  testdev: sudo -H /opt/pbs/bin/qmgr -c list hook
2018-12-07 09:11:22,412 INFOCLI  testdev: /opt/pbs/bin/qstat -Qf @testdev.pbspro.org
2018-12-07 09:11:22,418 INFO     status on testdev: node
2018-12-07 09:11:22,419 INFOCLI  testdev: /opt/pbs/bin/pbsnodes -s testdev.pbspro.org -v -a
2018-12-07 09:11:22,425 INFO     manager on testdev: delete queue workq
2018-12-07 09:11:22,425 INFOCLI  testdev: /opt/pbs/bin/qmgr -c delete queue workq
2018-12-07 09:11:22,432 INFO     server testdev: expect offset set to 0.5
2018-12-07 09:11:22,933 INFOCLI  testdev: /opt/pbs/bin/qstat -Qf workq@testdev.pbspro.org
2018-12-07 09:11:22,940 INFO     expect on server testdev:  unset  queue workq ...  OK
2018-12-07 09:11:22,941 INFO     manager on testdev: create queue workq {'started': 'True', 'queue_type': 'Execution', 'enabled': 'True'}
2018-12-07 09:11:22,941 INFOCLI  testdev: /opt/pbs/bin/qmgr -c create queue workq started=True,queue_type=Execution,enabled=True
2018-12-07 09:11:22,949 INFO     status on testdev: queue workq
2018-12-07 09:11:22,949 INFOCLI  testdev: /opt/pbs/bin/qstat -Qf workq@testdev.pbspro.org
2018-12-07 09:11:22,955 INFO     server testdev: expect offset set to 0.5
2018-12-07 09:11:23,456 INFOCLI  testdev: /opt/pbs/bin/qstat -Qf workq@testdev.pbspro.org
2018-12-07 09:11:23,464 INFO     expect on server testdev: started set True || queue_type set Execution || enabled set True queue workq ...  OK
2018-12-07 09:11:23,464 INFO     manager on testdev: list sched
2018-12-07 09:11:23,464 INFOCLI  testdev: /opt/pbs/bin/qmgr -c list sched
2018-12-07 09:11:23,476 INFOCLI2 testdev: sudo -H /opt/pbs/sbin/pbsfs
2018-12-07 09:11:23,485 INFOCLI2 testdev: sudo -H /bin/cat /var/spool/pbs/sched_priv/resource_group
2018-12-07 09:11:23,491 INFOCLI2 testdev: sudo -H /bin/cat /var/spool/pbs/sched_priv/holidays
2018-12-07 09:11:23,497 INFO     manager on testdev: set server {'scheduler_iteration': '600', 'default_queue': 'workq'}
2018-12-07 09:11:23,497 INFOCLI  testdev: /opt/pbs/bin/qmgr -c set server scheduler_iteration=600,default_queue=workq
2018-12-07 09:11:23,506 INFO     status on testdev: resource
2018-12-07 09:11:23,507 INFO     manager on testdev: list resource
2018-12-07 09:11:23,507 INFOCLI  testdev: /opt/pbs/bin/qmgr -c list resource
2018-12-07 09:11:23,513 INFOCLI  status on testdev: server license_count
2018-12-07 09:11:23,513 INFOCLI  testdev: /opt/pbs/bin/qstat -Bf testdev.pbspro.org
2018-12-07 09:11:23,519 INFO     server: testdev.pbspro.org licensed
2018-12-07 09:11:23,523 INFOCLI2 testdev: sudo -H /bin/cat /var/spool/pbs/server_priv/comm.lock
2018-12-07 09:11:23,532 INFOCLI2 testdev: sudo -H /bin/cat /var/spool/pbs/sched_priv/sched.lock
2018-12-07 09:11:23,538 INFO     scheduler testdev: reverting configuration to defaults
2018-12-07 09:11:23,538 INFO     manager on testdev: unset sched ['sched_priv', 'sched_cycle_length', 'scheduler_iteration', 'scheduling', 'sched_log']
2018-12-07 09:11:23,539 INFOCLI  testdev: /opt/pbs/bin/qmgr -c unset sched sched_priv,sched_cycle_length,scheduler_iteration,scheduling,sched_log
2018-12-07 09:11:23,551 INFOCLI2 testdev: sudo -H /bin/cat /var/spool/pbs/sched_priv/dedicated_time
2018-12-07 09:11:23,557 INFOCLI2 testdev: sudo -H cmp /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group
2018-12-07 09:11:23,563 INFO     scheduler testdev: reverting holidays file to default
2018-12-07 09:11:23,564 INFOCLI2 testdev: sudo -H cmp /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays
2018-12-07 09:11:23,569 INFOCLI2 testdev: sudo -H cmp /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config
2018-12-07 09:11:23,575 INFO     scheduler testdev: sent signal -HUP
2018-12-07 09:11:23,575 INFOCLI2 testdev: sudo -H kill -HUP 10888
2018-12-07 09:11:23,581 INFOCLI2 testdev: sudo -H /opt/pbs/sbin/pbsfs -e -I default
2018-12-07 09:11:23,601 INFOCLI2 testdev: sudo -H /bin/cat /var/spool/pbs/sched_priv/sched.lock
2018-12-07 09:11:23,611 INFOCLI2 testdev: sudo -H /bin/cat /var/spool/pbs/mom_priv/mom.lock
2018-12-07 09:11:23,616 INFO     mom testdev: reverting configuration to defaults
2018-12-07 09:11:23,617 INFOCLI2 testdev: sudo -H /bin/rm -f /var/spool/pbs/mom_priv/epilogue
2018-12-07 09:11:23,623 INFOCLI2 testdev: sudo -H /bin/rm -f /var/spool/pbs/mom_priv/prologue
2018-12-07 09:11:23,628 INFOCLI  testdev: sudo -H /opt/pbs/sbin/pbs_mom -s list
2018-12-07 09:11:23,669 INFOCLI  testdev: sudo -H /opt/pbs/sbin/pbs_mom -s list
2018-12-07 09:11:23,713 INFOCLI  testdev: sudo -H /opt/pbs/sbin/pbs_mom -s remove pbs_vnode_1544173873.def
2018-12-07 09:11:23,757 INFO     status on testdev: host testdev.pbspro.org {'resources_available.host': 'testdev.pbspro.org', 'resources_available.vnode': None}
2018-12-07 09:11:23,758 INFOCLI  testdev: /opt/pbs/bin/pbsnodes -s testdev.pbspro.org -H testdev.pbspro.org
2018-12-07 09:11:23,764 ERROR    err: ['Node: testdev.pbspro.org,  Error: Unknown node ']
2018-12-07 09:11:23,765 INFO     status on testdev: host testdev {'resources_available.host': 'testdev.pbspro.org', 'resources_available.vnode': None}
2018-12-07 09:11:23,765 INFOCLI  testdev: /opt/pbs/bin/pbsnodes -s testdev.pbspro.org -H testdev
2018-12-07 09:11:23,771 INFO     manager on testdev: delete node vn[0],vn[1]
2018-12-07 09:11:23,771 INFOCLI  testdev: /opt/pbs/bin/qmgr -c delete node vn[0],vn[1]
2018-12-07 09:11:23,785 INFOCLI2 testdev: sudo -H /bin/cat /var/spool/pbs/mom_priv/mom.lock
2018-12-07 09:11:23,794 INFOCLI2 testdev: sudo -H ls -l /opt/pbs/libexec/pbs_init.d
2018-12-07 09:11:23,799 INFO     running init script to stop pbs mom on testdev.pbspro.org using /etc/pbs.conf init_cmd=['sudo', 'PBS_START_MOM=1', 'PBS_START_SERVER=0', 'PBS_START_SCHED=0', 'PBS_START_COMM=0', '/opt/pbs/libexec/pbs_init.d', 'stop']
2018-12-07 09:11:23,800 INFOCLI2 testdev: /tmp/PtlPbsX212Wl
Contents of /tmp/PtlPbsX212Wl:
----------------------------------------
#!/bin/bash
sudo PBS_START_MOM=1 PBS_START_SERVER=0 PBS_START_SCHED=0 PBS_START_COMM=0 /opt/pbs/libexec/pbs_init.d stop
----------------------------------------
2018-12-07 09:11:24,879 INFOCLI2 testdev: /bin/rm /tmp/PtlPbsX212Wl
2018-12-07 09:11:24,887 INFOCLI2 testdev: sudo -H ls -l /opt/pbs/libexec/pbs_init.d
2018-12-07 09:11:24,896 INFO     running init script to start pbs mom on testdev.pbspro.org using /etc/pbs.conf init_cmd=['sudo', 'PBS_START_MOM=1', 'PBS_START_SERVER=0', 'PBS_START_SCHED=0', 'PBS_START_COMM=0', '/opt/pbs/libexec/pbs_init.d', 'start']
2018-12-07 09:11:24,897 INFOCLI2 testdev: /tmp/PtlPbsRx9vaD
Contents of /tmp/PtlPbsRx9vaD:
----------------------------------------
#!/bin/bash
sudo PBS_START_MOM=1 PBS_START_SERVER=0 PBS_START_SCHED=0 PBS_START_COMM=0 /opt/pbs/libexec/pbs_init.d start
----------------------------------------
2018-12-07 09:11:25,002 INFOCLI2 testdev: /bin/rm /tmp/PtlPbsRx9vaD
2018-12-07 09:11:25,015 INFOCLI2 testdev: sudo -H /bin/cat /var/spool/pbs/mom_priv/mom.lock
2018-12-07 09:11:25,024 INFOCLI2 testdev: sudo -H /bin/cat /var/spool/pbs/mom_priv/mom.lock
2018-12-07 09:11:25,029 INFOCLI  testdev: /opt/pbs/bin/pbsnodes -s testdev.pbspro.org -v -a
2018-12-07 09:11:25,035 INFO     manager on testdev: delete node 
2018-12-07 09:11:25,036 INFOCLI  testdev: /opt/pbs/bin/qmgr -c delete node @default
2018-12-07 09:11:25,045 INFOCLI  testdev: sudo -H /opt/pbs/sbin/pbs_mom -s list
2018-12-07 09:11:25,089 INFO     status on testdev: host testdev.pbspro.org {'resources_available.host': 'testdev.pbspro.org', 'resources_available.vnode': None}
2018-12-07 09:11:25,089 INFOCLI  testdev: /opt/pbs/bin/pbsnodes -s testdev.pbspro.org -H testdev.pbspro.org
2018-12-07 09:11:25,097 ERROR    err: ['/opt/pbs/bin/pbsnodes: Server has no node list']
2018-12-07 09:11:25,098 INFO     status on testdev: host testdev {'resources_available.host': 'testdev.pbspro.org', 'resources_available.vnode': None}
2018-12-07 09:11:25,098 INFOCLI  testdev: /opt/pbs/bin/pbsnodes -s testdev.pbspro.org -H testdev
2018-12-07 09:11:25,107 ERROR    err: ['/opt/pbs/bin/pbsnodes: Server has no node list']
2018-12-07 09:11:25,116 INFOCLI2 testdev: sudo -H /bin/cat /var/spool/pbs/mom_priv/mom.lock
2018-12-07 09:11:25,135 INFOCLI2 testdev: sudo -H ls -l /opt/pbs/libexec/pbs_init.d
2018-12-07 09:11:25,143 INFO     running init script to stop pbs mom on testdev.pbspro.org using /etc/pbs.conf init_cmd=['sudo', 'PBS_START_MOM=1', 'PBS_START_SERVER=0', 'PBS_START_SCHED=0', 'PBS_START_COMM=0', '/opt/pbs/libexec/pbs_init.d', 'stop']
2018-12-07 09:11:25,144 INFOCLI2 testdev: /tmp/PtlPbsrzwdqG
Contents of /tmp/PtlPbsrzwdqG:
----------------------------------------
#!/bin/bash
sudo PBS_START_MOM=1 PBS_START_SERVER=0 PBS_START_SCHED=0 PBS_START_COMM=0 /opt/pbs/libexec/pbs_init.d stop
----------------------------------------
2018-12-07 09:11:26,189 INFOCLI2 testdev: /bin/rm /tmp/PtlPbsrzwdqG
2018-12-07 09:11:26,196 INFOCLI2 testdev: sudo -H ls -l /opt/pbs/libexec/pbs_init.d
2018-12-07 09:11:26,204 INFO     running init script to start pbs mom on testdev.pbspro.org using /etc/pbs.conf init_cmd=['sudo', 'PBS_START_MOM=1', 'PBS_START_SERVER=0', 'PBS_START_SCHED=0', 'PBS_START_COMM=0', '/opt/pbs/libexec/pbs_init.d', 'start']
2018-12-07 09:11:26,205 INFOCLI2 testdev: /tmp/PtlPbs2Jxphz
Contents of /tmp/PtlPbs2Jxphz:
----------------------------------------
#!/bin/bash
sudo PBS_START_MOM=1 PBS_START_SERVER=0 PBS_START_SCHED=0 PBS_START_COMM=0 /opt/pbs/libexec/pbs_init.d start
----------------------------------------
2018-12-07 09:11:26,324 INFOCLI2 testdev: /bin/rm /tmp/PtlPbs2Jxphz
2018-12-07 09:11:26,339 INFOCLI2 testdev: sudo -H /bin/cat /var/spool/pbs/mom_priv/mom.lock
2018-12-07 09:11:26,347 INFO     server: no nodes defined, creating one
2018-12-07 09:11:26,347 INFO     manager on testdev: create node testdev
2018-12-07 09:11:26,347 INFOCLI  testdev: /opt/pbs/bin/qmgr -c create node testdev
2018-12-07 09:11:26,359 INFO     status on testdev: node testdev
2018-12-07 09:11:26,359 INFOCLI  testdev: /opt/pbs/bin/pbsnodes -s testdev.pbspro.org -v testdev
2018-12-07 09:11:26,370 INFO     status on testdev: node testdev
2018-12-07 09:11:26,371 INFOCLI  testdev: /opt/pbs/bin/pbsnodes -s testdev.pbspro.org -v testdev
2018-12-07 09:11:26,377 INFOCLI  testdev: /opt/pbs/bin/pbsnodes -s testdev.pbspro.org -v testdev
2018-12-07 09:11:26,383 INFO     expect on server testdev: state = free node testdev  got: state = state-unknown,down
2018-12-07 09:11:27,384 INFOCLI  testdev: /opt/pbs/bin/pbsnodes -s testdev.pbspro.org -v testdev
2018-12-07 09:11:27,394 INFO     expect on server testdev: state = free node testdev attempt: 2 ...  OK
2018-12-07 09:11:27,395 INFO     =========================
2018-12-07 09:11:27,396 INFO     Completed SmokeTest setUp
2018-12-07 09:11:27,396 INFO     =========================
2018-12-07 09:11:27,406 INFO     scheduler testdev: config {'unknown_shares': 10, 'fair_share': 'true ALL', 'fairshare_usage_res': 'ncpus*walltime'}
2018-12-07 09:11:27,407 INFOCLI2 testdev: sudo -H /bin/cp /tmp/PtlPbsMBJBPc /var/spool/pbs/sched_priv/sched_config
2018-12-07 09:11:27,415 INFOCLI2 testdev: sudo -H /bin/cat /var/spool/pbs/sched_priv/sched.lock
2018-12-07 09:11:27,423 INFO     scheduler testdev: sent signal -HUP
2018-12-07 09:11:27,423 INFOCLI2 testdev: sudo -H kill -HUP 10888
2018-12-07 09:11:27,432 INFO     scheduler testdev log match: searching for "Sched;reconfigure;Scheduler is reconfiguring"... OK
2018-12-07 09:11:27,433 INFO     scheduler testdev log match: searching for "Error reading line" - No match
2018-12-07 09:11:29,773 INFO     scheduler testdev log match: searching for "Error reading line" - attempt 2
2018-12-07 09:11:30,274 INFO     manager on testdev: delete node 
2018-12-07 09:11:30,275 INFOCLI  testdev: /opt/pbs/bin/qmgr -c delete node @default
2018-12-07 09:11:30,285 INFO     mom testdev: created 4 vnodes vnode with attr {'resources_available.ncpus': 4} on host testdev.pbspro.org
2018-12-07 09:11:30,286 INFOCLI  testdev: sudo -H /opt/pbs/sbin/pbs_mom -s list
2018-12-07 09:11:30,335 INFOCLI  testdev: sudo -H /opt/pbs/sbin/pbs_mom -s insert pbs_vnode_1544173890.def /tmp/PtlPbs3FezKL
2018-12-07 09:11:30,387 INFOCLI2 testdev: /bin/rm -f /tmp/PtlPbs3FezKL
2018-12-07 09:11:30,393 INFO     mom testdev: inserted vnode definition file pbs_vnode_1544173890.def on host: testdev.pbspro.org
2018-12-07 09:11:30,398 INFOCLI2 testdev: sudo -H /bin/cat /var/spool/pbs/mom_priv/mom.lock
2018-12-07 09:11:30,408 INFOCLI2 testdev: sudo -H ls -l /opt/pbs/libexec/pbs_init.d
2018-12-07 09:11:30,416 INFO     running init script to stop pbs mom on testdev.pbspro.org using /etc/pbs.conf init_cmd=['sudo', 'PBS_START_MOM=1', 'PBS_START_SERVER=0', 'PBS_START_SCHED=0', 'PBS_START_COMM=0', '/opt/pbs/libexec/pbs_init.d', 'stop']
2018-12-07 09:11:30,416 INFOCLI2 testdev: /tmp/PtlPbs0mHAep
Contents of /tmp/PtlPbs0mHAep:
----------------------------------------
#!/bin/bash
sudo PBS_START_MOM=1 PBS_START_SERVER=0 PBS_START_SCHED=0 PBS_START_COMM=0 /opt/pbs/libexec/pbs_init.d stop
----------------------------------------
2018-12-07 09:11:31,471 INFOCLI2 testdev: /bin/rm /tmp/PtlPbs0mHAep
2018-12-07 09:11:31,479 INFOCLI2 testdev: sudo -H ls -l /opt/pbs/libexec/pbs_init.d
2018-12-07 09:11:31,487 INFO     running init script to start pbs mom on testdev.pbspro.org using /etc/pbs.conf init_cmd=['sudo', 'PBS_START_MOM=1', 'PBS_START_SERVER=0', 'PBS_START_SCHED=0', 'PBS_START_COMM=0', '/opt/pbs/libexec/pbs_init.d', 'start']
2018-12-07 09:11:31,487 INFOCLI2 testdev: /tmp/PtlPbs4Fdszn
Contents of /tmp/PtlPbs4Fdszn:
----------------------------------------
#!/bin/bash
sudo PBS_START_MOM=1 PBS_START_SERVER=0 PBS_START_SCHED=0 PBS_START_COMM=0 /opt/pbs/libexec/pbs_init.d start
----------------------------------------
2018-12-07 09:11:31,608 INFOCLI2 testdev: /bin/rm /tmp/PtlPbs4Fdszn
2018-12-07 09:11:31,636 INFOCLI2 testdev: sudo -H /bin/cat /var/spool/pbs/mom_priv/mom.lock
2018-12-07 09:11:31,649 INFO     status on testdev: node testdev
2018-12-07 09:11:31,649 INFOCLI  testdev: /opt/pbs/bin/pbsnodes -s testdev.pbspro.org -v testdev
2018-12-07 09:11:31,655 ERROR    err: ['/opt/pbs/bin/pbsnodes: Server has no node list']
2018-12-07 09:11:31,655 INFO     manager on testdev: create node testdev
2018-12-07 09:11:31,656 INFOCLI  testdev: /opt/pbs/bin/qmgr -c create node testdev
2018-12-07 09:11:31,663 INFO     status on testdev: node testdev
2018-12-07 09:11:31,664 INFOCLI  testdev: /opt/pbs/bin/pbsnodes -s testdev.pbspro.org -v testdev
2018-12-07 09:11:31,670 INFOCLI  testdev: /opt/pbs/bin/pbsnodes -s testdev.pbspro.org -v vnode[0]
2018-12-07 09:11:32,177 INFO     expect on server testdev:  no data for state = free || resources_available.ncpus = 4 node vnode[0]
2018-12-07 09:11:32,178 INFOCLI  testdev: /opt/pbs/bin/pbsnodes -s testdev.pbspro.org -v vnode[0]
2018-12-07 09:11:32,687 INFO     expect on server testdev:  no data for state = free || resources_available.ncpus = 4 node vnode[0] attempt: 2
2018-12-07 09:11:32,688 INFOCLI  testdev: /opt/pbs/bin/pbsnodes -s testdev.pbspro.org -v vnode[0]
2018-12-07 09:11:33,196 INFO     expect on server testdev:  no data for state = free || resources_available.ncpus = 4 node vnode[0] attempt: 3
2018-12-07 09:11:33,197 INFOCLI  testdev: /opt/pbs/bin/pbsnodes -s testdev.pbspro.org -v vnode[0]
2018-12-07 09:11:33,205 INFO     expect on server testdev: state = free || resources_available.ncpus = 4 node vnode[0] attempt: 4 ...  OK
2018-12-07 09:11:33,205 INFOCLI  testdev: /opt/pbs/bin/pbsnodes -s testdev.pbspro.org -v vnode[1]
2018-12-07 09:11:33,211 INFO     expect on server testdev: state = free || resources_available.ncpus = 4 node vnode[1] ...  OK
2018-12-07 09:11:33,212 INFOCLI  testdev: /opt/pbs/bin/pbsnodes -s testdev.pbspro.org -v vnode[2]
2018-12-07 09:11:33,217 INFO     expect on server testdev: state = free || resources_available.ncpus = 4 node vnode[2] ...  OK
2018-12-07 09:11:33,218 INFOCLI  testdev: /opt/pbs/bin/pbsnodes -s testdev.pbspro.org -v vnode[3]
2018-12-07 09:11:33,223 INFO     expect on server testdev: state = free || resources_available.ncpus = 4 node vnode[3] ...  OK
2018-12-07 09:11:33,224 INFO     job: executable set to /bin/sleep with arguments: 100
2018-12-07 09:11:33,224 INFOCLI  testdev: sudo -H -u pbsuser1 /opt/pbs/bin/qsub -l select=1:ncpus=4 -- /bin/sleep 100
2018-12-07 09:11:33,235 INFO     submit to testdev as pbsuser1: job 18.testdev OrderedDict([('Resource_List.select', '1:ncpus=4')])
2018-12-07 09:11:33,236 INFO     job: executable set to /bin/sleep with arguments: 100
2018-12-07 09:11:33,236 INFOCLI  testdev: sudo -H -u pbsuser1 /opt/pbs/bin/qsub -l select=1:ncpus=4 -- /bin/sleep 100
2018-12-07 09:11:33,245 INFO     submit to testdev as pbsuser1: job 19.testdev OrderedDict([('Resource_List.select', '1:ncpus=4')])
2018-12-07 09:11:33,245 INFO     job: executable set to /bin/sleep with arguments: 100
2018-12-07 09:11:33,246 INFOCLI  testdev: sudo -H -u pbsuser1 /opt/pbs/bin/qsub -l select=1:ncpus=4 -- /bin/sleep 100
2018-12-07 09:11:33,254 INFO     submit to testdev as pbsuser1: job 20.testdev OrderedDict([('Resource_List.select', '1:ncpus=4')])
2018-12-07 09:11:33,254 INFO     job: executable set to /bin/sleep with arguments: 100
2018-12-07 09:11:33,255 INFOCLI  testdev: sudo -H -u pbsuser1 /opt/pbs/bin/qsub -l select=1:ncpus=4 -- /bin/sleep 100
2018-12-07 09:11:33,263 INFO     submit to testdev as pbsuser1: job 21.testdev OrderedDict([('Resource_List.select', '1:ncpus=4')])
2018-12-07 09:11:33,263 INFO     job: executable set to /bin/sleep with arguments: 100
2018-12-07 09:11:33,264 INFOCLI  testdev: sudo -H -u pbsuser1 /opt/pbs/bin/qsub -l select=1:ncpus=4 -- /bin/sleep 100
2018-12-07 09:11:33,271 INFO     submit to testdev as pbsuser1: job 22.testdev OrderedDict([('Resource_List.select', '1:ncpus=4')])
2018-12-07 09:11:33,272 INFO     job: executable set to /bin/sleep with arguments: 100
2018-12-07 09:11:33,272 INFOCLI  testdev: sudo -H -u pbsuser1 /opt/pbs/bin/qsub -l select=1:ncpus=4 -- /bin/sleep 100
2018-12-07 09:11:33,280 INFO     submit to testdev as pbsuser1: job 23.testdev OrderedDict([('Resource_List.select', '1:ncpus=4')])
2018-12-07 09:11:33,281 INFO     job: executable set to /bin/sleep with arguments: 100
2018-12-07 09:11:33,281 INFOCLI  testdev: sudo -H -u pbsuser1 /opt/pbs/bin/qsub -l select=1:ncpus=4 -- /bin/sleep 100
2018-12-07 09:11:33,289 INFO     submit to testdev as pbsuser1: job 24.testdev OrderedDict([('Resource_List.select', '1:ncpus=4')])
2018-12-07 09:11:33,289 INFO     job: executable set to /bin/sleep with arguments: 100
2018-12-07 09:11:33,290 INFOCLI  testdev: sudo -H -u pbsuser1 /opt/pbs/bin/qsub -l select=1:ncpus=4 -- /bin/sleep 100
2018-12-07 09:11:33,301 INFO     submit to testdev as pbsuser1: job 25.testdev OrderedDict([('Resource_List.select', '1:ncpus=4')])
2018-12-07 09:11:33,301 INFO     job: executable set to /bin/sleep with arguments: 100
2018-12-07 09:11:33,302 INFOCLI  testdev: sudo -H -u pbsuser1 /opt/pbs/bin/qsub -l select=1:ncpus=4 -- /bin/sleep 100
2018-12-07 09:11:33,310 INFO     submit to testdev as pbsuser1: job 26.testdev OrderedDict([('Resource_List.select', '1:ncpus=4')])
2018-12-07 09:11:33,311 INFO     job: executable set to /bin/sleep with arguments: 100
2018-12-07 09:11:33,311 INFOCLI  testdev: sudo -H -u pbsuser1 /opt/pbs/bin/qsub -l select=1:ncpus=4 -- /bin/sleep 100
2018-12-07 09:11:33,320 INFO     submit to testdev as pbsuser1: job 27.testdev OrderedDict([('Resource_List.select', '1:ncpus=4')])
2018-12-07 09:11:33,320 INFOCLI  testdev: /opt/pbs/bin/qstat -f @testdev.pbspro.org
2018-12-07 09:11:33,330 INFO     expect on server testdev: job_state=R = 4 job  got: job_state=R = 2
2018-12-07 09:11:33,831 INFOCLI  testdev: /opt/pbs/bin/qmgr -c set server scheduling=True
2018-12-07 09:11:33,842 INFOCLI  testdev: /opt/pbs/bin/qstat -f @testdev.pbspro.org
2018-12-07 09:11:33,854 INFO     expect on server testdev: job_state=R = 4 job attempt: 2 ...  OK
2018-12-07 09:11:33,854 INFO     testinfo: waiting for walltime accumulation
2018-12-07 09:11:33,855 INFO     filter: job {'job_state': 'R'}
2018-12-07 09:11:33,855 INFOCLI  testdev: /opt/pbs/bin/qstat -f @testdev.pbspro.org
2018-12-07 09:11:33,867 INFOCLI  testdev: /opt/pbs/bin/qstat -f 18.testdev
2018-12-07 09:11:33,878 INFO     expect on server testdev: resources_used.walltime != 00:00:00 job 18.testdev  got: resources_used.walltime = 0
2018-12-07 09:11:34,880 INFOCLI  testdev: /opt/pbs/bin/qmgr -c set server scheduling=True
2018-12-07 09:11:34,891 INFOCLI  testdev: /opt/pbs/bin/qstat -f 18.testdev
2018-12-07 09:11:34,899 INFO     expect on server testdev: resources_used.walltime != 00:00:00 job 18.testdev attempt: 2  got: resources_used.walltime = 0
2018-12-07 09:11:35,900 INFOCLI  testdev: /opt/pbs/bin/qmgr -c set server scheduling=True
2018-12-07 09:11:35,911 INFOCLI  testdev: /opt/pbs/bin/qstat -f 18.testdev
2018-12-07 09:11:35,920 INFO     expect on server testdev: resources_used.walltime != 00:00:00 job 18.testdev attempt: 3  got: resources_used.walltime = 0
2018-12-07 09:11:36,922 INFOCLI  testdev: /opt/pbs/bin/qmgr -c set server scheduling=True
2018-12-07 09:11:36,935 INFOCLI  testdev: /opt/pbs/bin/qstat -f 18.testdev
2018-12-07 09:11:36,950 INFO     expect on server testdev: resources_used.walltime != 00:00:00 job 18.testdev attempt: 4  got: resources_used.walltime = 0
2018-12-07 09:11:37,952 INFOCLI  testdev: /opt/pbs/bin/qmgr -c set server scheduling=True
2018-12-07 09:11:37,966 INFOCLI  testdev: /opt/pbs/bin/qstat -f 18.testdev
2018-12-07 09:11:37,978 INFO     expect on server testdev: resources_used.walltime != 00:00:00 job 18.testdev attempt: 5  got: resources_used.walltime = 0
2018-12-07 09:11:38,979 INFOCLI  testdev: /opt/pbs/bin/qmgr -c set server scheduling=True
2018-12-07 09:11:38,993 INFOCLI  testdev: /opt/pbs/bin/qstat -f 18.testdev
2018-12-07 09:11:39,006 INFO     expect on server testdev: resources_used.walltime != 00:00:00 job 18.testdev attempt: 6  got: resources_used.walltime = 0
2018-12-07 09:11:40,007 INFOCLI  testdev: /opt/pbs/bin/qmgr -c set server scheduling=True
2018-12-07 09:11:40,021 INFOCLI  testdev: /opt/pbs/bin/qstat -f 18.testdev
2018-12-07 09:11:40,032 INFO     expect on server testdev: resources_used.walltime != 00:00:00 job 18.testdev attempt: 7  got: resources_used.walltime = 0
2018-12-07 09:11:41,034 INFOCLI  testdev: /opt/pbs/bin/qmgr -c set server scheduling=True
2018-12-07 09:11:41,047 INFOCLI  testdev: /opt/pbs/bin/qstat -f 18.testdev
2018-12-07 09:11:41,059 INFO     expect on server testdev: resources_used.walltime != 00:00:00 job 18.testdev attempt: 8  got: resources_used.walltime = 0
2018-12-07 09:11:42,061 INFOCLI  testdev: /opt/pbs/bin/qmgr -c set server scheduling=True
2018-12-07 09:11:42,077 INFOCLI  testdev: /opt/pbs/bin/qstat -f 18.testdev
2018-12-07 09:11:42,089 INFO     expect on server testdev: resources_used.walltime != 00:00:00 job 18.testdev attempt: 9  got: resources_used.walltime = 0
2018-12-07 09:11:43,090 INFOCLI  testdev: /opt/pbs/bin/qmgr -c set server scheduling=True
2018-12-07 09:11:43,104 INFOCLI  testdev: /opt/pbs/bin/qstat -f 18.testdev
2018-12-07 09:11:43,116 INFO     expect on server testdev: resources_used.walltime != 00:00:00 job 18.testdev attempt: 10  got: resources_used.walltime = 0
2018-12-07 09:11:44,118 INFOCLI  testdev: /opt/pbs/bin/qmgr -c set server scheduling=True
2018-12-07 09:11:44,130 INFOCLI  testdev: /opt/pbs/bin/qstat -f 18.testdev
2018-12-07 09:11:44,142 INFO     expect on server testdev: resources_used.walltime != 00:00:00 job 18.testdev attempt: 11 ...  OK
2018-12-07 09:11:44,143 INFOCLI  testdev: /opt/pbs/bin/qstat -f 19.testdev
2018-12-07 09:11:44,158 INFO     expect on server testdev: resources_used.walltime != 00:00:00 job 19.testdev ...  OK
2018-12-07 09:11:44,158 INFOCLI  testdev: /opt/pbs/bin/qstat -f 20.testdev
2018-12-07 09:11:44,170 INFO     expect on server testdev: resources_used.walltime != 00:00:00 job 20.testdev ...  OK
2018-12-07 09:11:44,171 INFOCLI  testdev: /opt/pbs/bin/qstat -f 21.testdev
2018-12-07 09:11:44,182 INFO     expect on server testdev: resources_used.walltime != 00:00:00 job 21.testdev ...  OK
2018-12-07 09:11:44,182 INFO     job: executable set to /bin/sleep with arguments: 100
2018-12-07 09:11:44,183 INFOCLI  testdev: sudo -H -u pbsuser2 /opt/pbs/bin/qsub -- /bin/sleep 100
2018-12-07 09:11:44,198 INFO     submit to testdev as pbsuser2: job 28.testdev
2018-12-07 09:11:44,198 INFO     server testdev: expect offset set to 5
2018-12-07 09:11:49,204 INFOCLI  testdev: /opt/pbs/bin/qstat -f 28.testdev
2018-12-07 09:11:49,218 INFO     expect on server testdev: job_state = Q job 28.testdev ...  OK
2018-12-07 09:11:49,218 INFO     manager on testdev: set server {'scheduling': 'True'}
2018-12-07 09:11:49,219 INFOCLI  testdev: /opt/pbs/bin/qmgr -c set server scheduling=True
2018-12-07 09:11:49,231 INFOCLI  testdev: /opt/pbs/bin/qstat -Bf testdev.pbspro.org
2018-12-07 09:11:49,241 INFO     expect on server testdev: server_state != Scheduling server testdev.pbspro.org  got: server_state = Scheduling
2018-12-07 09:11:49,742 INFOCLI  testdev: /opt/pbs/bin/qstat -Bf testdev.pbspro.org
2018-12-07 09:11:49,753 INFO     expect on server testdev: server_state != Scheduling server testdev.pbspro.org attempt: 2  got: server_state = Scheduling
2018-12-07 09:11:50,254 INFOCLI  testdev: /opt/pbs/bin/qstat -Bf testdev.pbspro.org
2018-12-07 09:11:50,266 INFO     expect on server testdev: server_state != Scheduling server testdev.pbspro.org attempt: 3  got: server_state = Scheduling
2018-12-07 09:11:50,767 INFOCLI  testdev: /opt/pbs/bin/qstat -Bf testdev.pbspro.org
2018-12-07 09:11:50,781 INFO     expect on server testdev: server_state != Scheduling server testdev.pbspro.org attempt: 4  got: server_state = Scheduling
2018-12-07 09:11:51,282 INFOCLI  testdev: /opt/pbs/bin/qstat -Bf testdev.pbspro.org
2018-12-07 09:11:51,293 INFO     expect on server testdev: server_state != Scheduling server testdev.pbspro.org attempt: 5  got: server_state = Scheduling
2018-12-07 09:11:51,794 INFOCLI  testdev: /opt/pbs/bin/qstat -Bf testdev.pbspro.org
2018-12-07 09:11:51,806 INFO     expect on server testdev: server_state != Scheduling server testdev.pbspro.org attempt: 6  got: server_state = Scheduling
2018-12-07 09:11:52,307 INFOCLI  testdev: /opt/pbs/bin/qstat -Bf testdev.pbspro.org
2018-12-07 09:11:52,322 INFO     expect on server testdev: server_state != Scheduling server testdev.pbspro.org attempt: 7  got: server_state = Scheduling
2018-12-07 09:11:52,823 INFOCLI  testdev: /opt/pbs/bin/qstat -Bf testdev.pbspro.org
2018-12-07 09:11:52,833 INFO     expect on server testdev: server_state != Scheduling server testdev.pbspro.org attempt: 8  got: server_state = Scheduling
2018-12-07 09:11:53,334 INFOCLI  testdev: /opt/pbs/bin/qstat -Bf testdev.pbspro.org
2018-12-07 09:11:53,345 INFO     expect on server testdev: server_state != Scheduling server testdev.pbspro.org attempt: 9  got: server_state = Scheduling
2018-12-07 09:11:53,846 INFOCLI  testdev: /opt/pbs/bin/qstat -Bf testdev.pbspro.org
2018-12-07 09:11:53,855 INFO     expect on server testdev: server_state != Scheduling server testdev.pbspro.org attempt: 10  got: server_state = Scheduling
2018-12-07 09:11:54,356 INFOCLI  testdev: /opt/pbs/bin/qstat -Bf testdev.pbspro.org
2018-12-07 09:11:54,367 INFO     expect on server testdev: server_state != Scheduling server testdev.pbspro.org attempt: 11  got: server_state = Scheduling
2018-12-07 09:11:54,868 INFOCLI  testdev: /opt/pbs/bin/qstat -Bf testdev.pbspro.org
2018-12-07 09:11:54,878 INFO     expect on server testdev: server_state != Scheduling server testdev.pbspro.org attempt: 12  got: server_state = Scheduling
2018-12-07 09:11:55,379 INFOCLI  testdev: /opt/pbs/bin/qstat -Bf testdev.pbspro.org
2018-12-07 09:11:55,390 INFO     expect on server testdev: server_state != Scheduling server testdev.pbspro.org attempt: 13  got: server_state = Scheduling
2018-12-07 09:11:55,891 INFOCLI  testdev: /opt/pbs/bin/qstat -Bf testdev.pbspro.org
2018-12-07 09:11:55,902 INFO     expect on server testdev: server_state != Scheduling server testdev.pbspro.org attempt: 14  got: server_state = Scheduling
2018-12-07 09:11:56,403 INFOCLI  testdev: /opt/pbs/bin/qstat -Bf testdev.pbspro.org
2018-12-07 09:11:56,413 INFO     expect on server testdev: server_state != Scheduling server testdev.pbspro.org attempt: 15  got: server_state = Scheduling
2018-12-07 09:11:56,914 INFOCLI  testdev: /opt/pbs/bin/qstat -Bf testdev.pbspro.org
2018-12-07 09:11:56,924 INFO     expect on server testdev: server_state != Scheduling server testdev.pbspro.org attempt: 16  got: server_state = Scheduling
2018-12-07 09:11:57,425 INFOCLI  testdev: /opt/pbs/bin/qstat -Bf testdev.pbspro.org
2018-12-07 09:11:57,435 INFO     expect on server testdev: server_state != Scheduling server testdev.pbspro.org attempt: 17  got: server_state = Scheduling
2018-12-07 09:11:57,936 INFOCLI  testdev: /opt/pbs/bin/qstat -Bf testdev.pbspro.org
2018-12-07 09:11:57,947 INFO     expect on server testdev: server_state != Scheduling server testdev.pbspro.org attempt: 18  got: server_state = Scheduling
2018-12-07 09:11:58,448 INFOCLI  testdev: /opt/pbs/bin/qstat -Bf testdev.pbspro.org
2018-12-07 09:11:58,459 INFO     expect on server testdev: server_state != Scheduling server testdev.pbspro.org attempt: 19  got: server_state = Scheduling
2018-12-07 09:11:58,960 INFOCLI  testdev: /opt/pbs/bin/qstat -Bf testdev.pbspro.org
2018-12-07 09:11:58,970 INFO     expect on server testdev: server_state != Scheduling server testdev.pbspro.org attempt: 20  got: server_state = Scheduling
2018-12-07 09:11:59,471 INFOCLI  testdev: /opt/pbs/bin/qstat -Bf testdev.pbspro.org
2018-12-07 09:11:59,482 INFO     expect on server testdev: server_state != Scheduling server testdev.pbspro.org attempt: 21  got: server_state = Scheduling
2018-12-07 09:11:59,983 INFOCLI  testdev: /opt/pbs/bin/qstat -Bf testdev.pbspro.org
2018-12-07 09:11:59,993 INFO     expect on server testdev: server_state != Scheduling server testdev.pbspro.org attempt: 22  got: server_state = Scheduling
2018-12-07 09:12:00,494 INFOCLI  testdev: /opt/pbs/bin/qstat -Bf testdev.pbspro.org
2018-12-07 09:12:00,508 INFO     expect on server testdev: server_state != Scheduling server testdev.pbspro.org attempt: 23  got: server_state = Scheduling
2018-12-07 09:12:01,009 INFOCLI  testdev: /opt/pbs/bin/qstat -Bf testdev.pbspro.org
2018-12-07 09:12:01,020 INFO     expect on server testdev: server_state != Scheduling server testdev.pbspro.org attempt: 24  got: server_state = Scheduling
2018-12-07 09:12:01,521 INFOCLI  testdev: /opt/pbs/bin/qstat -Bf testdev.pbspro.org
2018-12-07 09:12:01,532 INFO     expect on server testdev: server_state != Scheduling server testdev.pbspro.org attempt: 25  got: server_state = Scheduling
2018-12-07 09:12:02,033 INFOCLI  testdev: /opt/pbs/bin/qstat -Bf testdev.pbspro.org
2018-12-07 09:12:02,043 INFO     expect on server testdev: server_state != Scheduling server testdev.pbspro.org attempt: 26  got: server_state = Scheduling
2018-12-07 09:12:02,544 INFOCLI  testdev: /opt/pbs/bin/qstat -Bf testdev.pbspro.org
2018-12-07 09:12:02,554 INFO     expect on server testdev: server_state != Scheduling server testdev.pbspro.org attempt: 27  got: server_state = Scheduling
2018-12-07 09:12:03,055 INFOCLI  testdev: /opt/pbs/bin/qstat -Bf testdev.pbspro.org
2018-12-07 09:12:03,065 INFO     expect on server testdev: server_state != Scheduling server testdev.pbspro.org attempt: 28  got: server_state = Scheduling
2018-12-07 09:12:03,565 INFOCLI  testdev: /opt/pbs/bin/qstat -Bf testdev.pbspro.org
2018-12-07 09:12:03,575 INFO     expect on server testdev: server_state != Scheduling server testdev.pbspro.org attempt: 29  got: server_state = Scheduling
2018-12-07 09:12:04,076 INFOCLI  testdev: /opt/pbs/bin/qstat -Bf testdev.pbspro.org
2018-12-07 09:12:04,085 INFO     expect on server testdev: server_state != Scheduling server testdev.pbspro.org attempt: 30  got: server_state = Scheduling
2018-12-07 09:12:04,586 INFOCLI  testdev: /opt/pbs/bin/qstat -Bf testdev.pbspro.org
2018-12-07 09:12:04,596 INFO     expect on server testdev: server_state != Scheduling server testdev.pbspro.org attempt: 31  got: server_state = Scheduling
2018-12-07 09:12:05,097 INFOCLI  testdev: /opt/pbs/bin/qstat -Bf testdev.pbspro.org
2018-12-07 09:12:05,108 INFO     expect on server testdev: server_state != Scheduling server testdev.pbspro.org attempt: 32  got: server_state = Scheduling
2018-12-07 09:12:05,609 INFOCLI  testdev: /opt/pbs/bin/qstat -Bf testdev.pbspro.org
2018-12-07 09:12:05,620 INFO     expect on server testdev: server_state != Scheduling server testdev.pbspro.org attempt: 33  got: server_state = Scheduling
2018-12-07 09:12:06,122 INFOCLI  testdev: /opt/pbs/bin/qstat -Bf testdev.pbspro.org
2018-12-07 09:12:06,132 INFO     expect on server testdev: server_state != Scheduling server testdev.pbspro.org attempt: 34  got: server_state = Scheduling
2018-12-07 09:12:06,633 INFOCLI  testdev: /opt/pbs/bin/qstat -Bf testdev.pbspro.org
2018-12-07 09:12:06,642 INFO     expect on server testdev: server_state != Scheduling server testdev.pbspro.org attempt: 35  got: server_state = Scheduling
2018-12-07 09:12:07,143 INFOCLI  testdev: /opt/pbs/bin/qstat -Bf testdev.pbspro.org
2018-12-07 09:12:07,154 INFO     expect on server testdev: server_state != Scheduling server testdev.pbspro.org attempt: 36  got: server_state = Scheduling
2018-12-07 09:12:07,655 INFOCLI  testdev: /opt/pbs/bin/qstat -Bf testdev.pbspro.org
2018-12-07 09:12:07,665 INFO     expect on server testdev: server_state != Scheduling server testdev.pbspro.org attempt: 37  got: server_state = Scheduling
2018-12-07 09:12:08,166 INFOCLI  testdev: /opt/pbs/bin/qstat -Bf testdev.pbspro.org
2018-12-07 09:12:08,175 INFO     expect on server testdev: server_state != Scheduling server testdev.pbspro.org attempt: 38  got: server_state = Scheduling
2018-12-07 09:12:08,676 INFOCLI  testdev: /opt/pbs/bin/qstat -Bf testdev.pbspro.org
2018-12-07 09:12:08,686 INFO     expect on server testdev: server_state != Scheduling server testdev.pbspro.org attempt: 39  got: server_state = Scheduling
2018-12-07 09:12:09,187 INFOCLI  testdev: /opt/pbs/bin/qstat -Bf testdev.pbspro.org
2018-12-07 09:12:09,196 INFO     expect on server testdev: server_state != Scheduling server testdev.pbspro.org attempt: 40  got: server_state = Scheduling
2018-12-07 09:12:09,697 INFOCLI  testdev: /opt/pbs/bin/qstat -Bf testdev.pbspro.org
2018-12-07 09:12:09,707 INFO     expect on server testdev: server_state != Scheduling server testdev.pbspro.org attempt: 41  got: server_state = Scheduling
2018-12-07 09:12:10,208 INFOCLI  testdev: /opt/pbs/bin/qstat -Bf testdev.pbspro.org
2018-12-07 09:12:10,217 INFO     expect on server testdev: server_state != Scheduling server testdev.pbspro.org attempt: 42  got: server_state = Scheduling
2018-12-07 09:12:10,718 INFOCLI  testdev: /opt/pbs/bin/qstat -Bf testdev.pbspro.org
2018-12-07 09:12:10,728 INFO     expect on server testdev: server_state != Scheduling server testdev.pbspro.org attempt: 43  got: server_state = Scheduling
2018-12-07 09:12:11,229 INFOCLI  testdev: /opt/pbs/bin/qstat -Bf testdev.pbspro.org
2018-12-07 09:12:11,239 INFO     expect on server testdev: server_state != Scheduling server testdev.pbspro.org attempt: 44  got: server_state = Scheduling
2018-12-07 09:12:11,740 INFOCLI  testdev: /opt/pbs/bin/qstat -Bf testdev.pbspro.org
2018-12-07 09:12:11,750 INFO     expect on server testdev: server_state != Scheduling server testdev.pbspro.org attempt: 45  got: server_state = Scheduling
2018-12-07 09:12:12,251 INFOCLI  testdev: /opt/pbs/bin/qstat -Bf testdev.pbspro.org
2018-12-07 09:12:12,260 INFO     expect on server testdev: server_state != Scheduling server testdev.pbspro.org attempt: 46  got: server_state = Scheduling
2018-12-07 09:12:12,761 INFOCLI  testdev: /opt/pbs/bin/qstat -Bf testdev.pbspro.org
2018-12-07 09:12:12,773 INFO     expect on server testdev: server_state != Scheduling server testdev.pbspro.org attempt: 47  got: server_state = Scheduling
2018-12-07 09:12:13,274 INFOCLI  testdev: /opt/pbs/bin/qstat -Bf testdev.pbspro.org
2018-12-07 09:12:13,283 INFO     expect on server testdev: server_state != Scheduling server testdev.pbspro.org attempt: 48  got: server_state = Scheduling
2018-12-07 09:12:13,784 INFOCLI  testdev: /opt/pbs/bin/qstat -Bf testdev.pbspro.org
2018-12-07 09:12:13,794 INFO     expect on server testdev: server_state != Scheduling server testdev.pbspro.org attempt: 49  got: server_state = Scheduling
2018-12-07 09:12:14,295 INFOCLI  testdev: /opt/pbs/bin/qstat -Bf testdev.pbspro.org
2018-12-07 09:12:14,305 INFO     expect on server testdev: server_state != Scheduling server testdev.pbspro.org attempt: 50  got: server_state = Scheduling
2018-12-07 09:12:14,806 INFOCLI  testdev: /opt/pbs/bin/qstat -Bf testdev.pbspro.org
2018-12-07 09:12:14,819 INFO     expect on server testdev: server_state != Scheduling server testdev.pbspro.org attempt: 51  got: server_state = Scheduling
2018-12-07 09:12:15,320 INFOCLI  testdev: /opt/pbs/bin/qstat -Bf testdev.pbspro.org
2018-12-07 09:12:15,330 INFO     expect on server testdev: server_state != Scheduling server testdev.pbspro.org attempt: 52  got: server_state = Scheduling
2018-12-07 09:12:15,831 INFOCLI  testdev: /opt/pbs/bin/qstat -Bf testdev.pbspro.org
2018-12-07 09:12:15,842 INFO     expect on server testdev: server_state != Scheduling server testdev.pbspro.org attempt: 53  got: server_state = Scheduling
2018-12-07 09:12:16,343 INFOCLI  testdev: /opt/pbs/bin/qstat -Bf testdev.pbspro.org
2018-12-07 09:12:16,352 INFO     expect on server testdev: server_state != Scheduling server testdev.pbspro.org attempt: 54  got: server_state = Scheduling
2018-12-07 09:12:16,854 INFOCLI  testdev: /opt/pbs/bin/qstat -Bf testdev.pbspro.org
2018-12-07 09:12:16,863 INFO     expect on server testdev: server_state != Scheduling server testdev.pbspro.org attempt: 55  got: server_state = Scheduling
2018-12-07 09:12:17,364 INFOCLI  testdev: /opt/pbs/bin/qstat -Bf testdev.pbspro.org
2018-12-07 09:12:17,374 INFO     expect on server testdev: server_state != Scheduling server testdev.pbspro.org attempt: 56  got: server_state = Scheduling
2018-12-07 09:12:17,875 INFOCLI  testdev: /opt/pbs/bin/qstat -Bf testdev.pbspro.org
2018-12-07 09:12:17,886 INFO     expect on server testdev: server_state != Scheduling server testdev.pbspro.org attempt: 57  got: server_state = Scheduling
2018-12-07 09:12:18,387 INFOCLI  testdev: /opt/pbs/bin/qstat -Bf testdev.pbspro.org
2018-12-07 09:12:18,397 INFO     expect on server testdev: server_state != Scheduling server testdev.pbspro.org attempt: 58  got: server_state = Scheduling
2018-12-07 09:12:18,898 INFOCLI  testdev: /opt/pbs/bin/qstat -Bf testdev.pbspro.org
2018-12-07 09:12:18,909 INFO     expect on server testdev: server_state != Scheduling server testdev.pbspro.org attempt: 59  got: server_state = Scheduling
2018-12-07 09:12:19,410 INFOCLI  testdev: /opt/pbs/bin/qstat -Bf testdev.pbspro.org
2018-12-07 09:12:19,420 INFO     expect on server testdev: server_state != Scheduling server testdev.pbspro.org attempt: 60  got: server_state = Scheduling
2018-12-07 09:12:19,926 INFO     FAILED
...
2018-12-07 09:18:41,941 INFO     
2018-12-07 09:18:41,942 INFO     ======================================================================
2018-12-07 09:18:41,942 INFO     FAILED: test_fairshare (tests.pbs_smoketest.SmokeTest)

2018-12-07 09:18:41,942 INFO     ___m_oo_m___
2018-12-07 09:18:41,942 INFO     Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/ptl/utils/pbs_testsuite.py", line 208, in wrapper
    function(self, *args, **kwargs)
  File "/home/travis/build/PBSPro/pbspro/test/tests/pbs_smoketest.py", line 439, in test_fairshare
    self.server.expect(SERVER, a, op=NE)
  File "/usr/local/lib/python2.7/dist-packages/ptl/lib/pbs_testlib.py", line 8296, in expect
    extend, level=level, msg=" ".join(msg))
  File "/usr/local/lib/python2.7/dist-packages/ptl/lib/pbs_testlib.py", line 8296, in expect
    extend, level=level, msg=" ".join(msg))
  File "/usr/local/lib/python2.7/dist-packages/ptl/lib/pbs_testlib.py", line 8296, in expect
    extend, level=level, msg=" ".join(msg))
  File "/usr/local/lib/python2.7/dist-packages/ptl/lib/pbs_testlib.py", line 8296, in expect
    extend, level=level, msg=" ".join(msg))
  File "/usr/local/lib/python2.7/dist-packages/ptl/lib/pbs_testlib.py", line 8296, in expect
    extend, level=level, msg=" ".join(msg))
  File "/usr/local/lib/python2.7/dist-packages/ptl/lib/pbs_testlib.py", line 8296, in expect
    extend, level=level, msg=" ".join(msg))
  File "/usr/local/lib/python2.7/dist-packages/ptl/lib/pbs_testlib.py", line 8296, in expect
    extend, level=level, msg=" ".join(msg))
  File "/usr/local/lib/python2.7/dist-packages/ptl/lib/pbs_testlib.py", line 8296, in expect
    extend, level=level, msg=" ".join(msg))
  File "/usr/local/lib/python2.7/dist-packages/ptl/lib/pbs_testlib.py", line 8296, in expect
    extend, level=level, msg=" ".join(msg))
  File "/usr/local/lib/python2.7/dist-packages/ptl/lib/pbs_testlib.py", line 8296, in expect
    extend, level=level, msg=" ".join(msg))
  File "/usr/local/lib/python2.7/dist-packages/ptl/lib/pbs_testlib.py", line 8296, in expect
    extend, level=level, msg=" ".join(msg))
  File "/usr/local/lib/python2.7/dist-packages/ptl/lib/pbs_testlib.py", line 8296, in expect
    extend, level=level, msg=" ".join(msg))
  File "/usr/local/lib/python2.7/dist-packages/ptl/lib/pbs_testlib.py", line 8296, in expect
    extend, level=level, msg=" ".join(msg))
  File "/usr/local/lib/python2.7/dist-packages/ptl/lib/pbs_testlib.py", line 8296, in expect
    extend, level=level, msg=" ".join(msg))
  File "/usr/local/lib/python2.7/dist-packages/ptl/lib/pbs_testlib.py", line 8296, in expect
    extend, level=level, msg=" ".join(msg))
  File "/usr/local/lib/python2.7/dist-packages/ptl/lib/pbs_testlib.py", line 8296, in expect
    extend, level=level, msg=" ".join(msg))
  File "/usr/local/lib/python2.7/dist-packages/ptl/lib/pbs_testlib.py", line 8296, in expect
    extend, level=level, msg=" ".join(msg))
  File "/usr/local/lib/python2.7/dist-packages/ptl/lib/pbs_testlib.py", line 8296, in expect
    extend, level=level, msg=" ".join(msg))
  File "/usr/local/lib/python2.7/dist-packages/ptl/lib/pbs_testlib.py", line 8296, in expect
    extend, level=level, msg=" ".join(msg))
  File "/usr/local/lib/python2.7/dist-packages/ptl/lib/pbs_testlib.py", line 8296, in expect
    extend, level=level, msg=" ".join(msg))
  File "/usr/local/lib/python2.7/dist-packages/ptl/lib/pbs_testlib.py", line 8296, in expect
    extend, level=level, msg=" ".join(msg))
  File "/usr/local/lib/python2.7/dist-packages/ptl/lib/pbs_testlib.py", line 8296, in expect
    extend, level=level, msg=" ".join(msg))
  File "/usr/local/lib/python2.7/dist-packages/ptl/lib/pbs_testlib.py", line 8296, in expect
    extend, level=level, msg=" ".join(msg))
  File "/usr/local/lib/python2.7/dist-packages/ptl/lib/pbs_testlib.py", line 8296, in expect
    extend, level=level, msg=" ".join(msg))
  File "/usr/local/lib/python2.7/dist-packages/ptl/lib/pbs_testlib.py", line 8296, in expect
    extend, level=level, msg=" ".join(msg))
  File "/usr/local/lib/python2.7/dist-packages/ptl/lib/pbs_testlib.py", line 8296, in expect
    extend, level=level, msg=" ".join(msg))
  File "/usr/local/lib/python2.7/dist-packages/ptl/lib/pbs_testlib.py", line 8296, in expect
    extend, level=level, msg=" ".join(msg))
  File "/usr/local/lib/python2.7/dist-packages/ptl/lib/pbs_testlib.py", line 8296, in expect
    extend, level=level, msg=" ".join(msg))
  File "/usr/local/lib/python2.7/dist-packages/ptl/lib/pbs_testlib.py", line 8296, in expect
    extend, level=level, msg=" ".join(msg))
  File "/usr/local/lib/python2.7/dist-packages/ptl/lib/pbs_testlib.py", line 8296, in expect
    extend, level=level, msg=" ".join(msg))
  File "/usr/local/lib/python2.7/dist-packages/ptl/lib/pbs_testlib.py", line 8296, in expect
    extend, level=level, msg=" ".join(msg))
  File "/usr/local/lib/python2.7/dist-packages/ptl/lib/pbs_testlib.py", line 8296, in expect
    extend, level=level, msg=" ".join(msg))
  File "/usr/local/lib/python2.7/dist-packages/ptl/lib/pbs_testlib.py", line 8296, in expect
    extend, level=level, msg=" ".join(msg))
  File "/usr/local/lib/python2.7/dist-packages/ptl/lib/pbs_testlib.py", line 8296, in expect
    extend, level=level, msg=" ".join(msg))
  File "/usr/local/lib/python2.7/dist-packages/ptl/lib/pbs_testlib.py", line 8296, in expect
    extend, level=level, msg=" ".join(msg))
  File "/usr/local/lib/python2.7/dist-packages/ptl/lib/pbs_testlib.py", line 8296, in expect
    extend, level=level, msg=" ".join(msg))
  File "/usr/local/lib/python2.7/dist-packages/ptl/lib/pbs_testlib.py", line 8296, in expect
    extend, level=level, msg=" ".join(msg))
  File "/usr/local/lib/python2.7/dist-packages/ptl/lib/pbs_testlib.py", line 8296, in expect
    extend, level=level, msg=" ".join(msg))
  File "/usr/local/lib/python2.7/dist-packages/ptl/lib/pbs_testlib.py", line 8296, in expect
    extend, level=level, msg=" ".join(msg))
  File "/usr/local/lib/python2.7/dist-packages/ptl/lib/pbs_testlib.py", line 8296, in expect
    extend, level=level, msg=" ".join(msg))
  File "/usr/local/lib/python2.7/dist-packages/ptl/lib/pbs_testlib.py", line 8296, in expect
    extend, level=level, msg=" ".join(msg))
  File "/usr/local/lib/python2.7/dist-packages/ptl/lib/pbs_testlib.py", line 8296, in expect
    extend, level=level, msg=" ".join(msg))
  File "/usr/local/lib/python2.7/dist-packages/ptl/lib/pbs_testlib.py", line 8296, in expect
    extend, level=level, msg=" ".join(msg))
  File "/usr/local/lib/python2.7/dist-packages/ptl/lib/pbs_testlib.py", line 8296, in expect
    extend, level=level, msg=" ".join(msg))
  File "/usr/local/lib/python2.7/dist-packages/ptl/lib/pbs_testlib.py", line 8296, in expect
    extend, level=level, msg=" ".join(msg))
  File "/usr/local/lib/python2.7/dist-packages/ptl/lib/pbs_testlib.py", line 8296, in expect
    extend, level=level, msg=" ".join(msg))
  File "/usr/local/lib/python2.7/dist-packages/ptl/lib/pbs_testlib.py", line 8296, in expect
    extend, level=level, msg=" ".join(msg))
  File "/usr/local/lib/python2.7/dist-packages/ptl/lib/pbs_testlib.py", line 8296, in expect
    extend, level=level, msg=" ".join(msg))
  File "/usr/local/lib/python2.7/dist-packages/ptl/lib/pbs_testlib.py", line 8296, in expect
    extend, level=level, msg=" ".join(msg))
  File "/usr/local/lib/python2.7/dist-packages/ptl/lib/pbs_testlib.py", line 8296, in expect
    extend, level=level, msg=" ".join(msg))
  File "/usr/local/lib/python2.7/dist-packages/ptl/lib/pbs_testlib.py", line 8296, in expect
    extend, level=level, msg=" ".join(msg))
  File "/usr/local/lib/python2.7/dist-packages/ptl/lib/pbs_testlib.py", line 8296, in expect
    extend, level=level, msg=" ".join(msg))
  File "/usr/local/lib/python2.7/dist-packages/ptl/lib/pbs_testlib.py", line 8296, in expect
    extend, level=level, msg=" ".join(msg))
  File "/usr/local/lib/python2.7/dist-packages/ptl/lib/pbs_testlib.py", line 8296, in expect
    extend, level=level, msg=" ".join(msg))
  File "/usr/local/lib/python2.7/dist-packages/ptl/lib/pbs_testlib.py", line 8296, in expect
    extend, level=level, msg=" ".join(msg))
  File "/usr/local/lib/python2.7/dist-packages/ptl/lib/pbs_testlib.py", line 8296, in expect
    extend, level=level, msg=" ".join(msg))
  File "/usr/local/lib/python2.7/dist-packages/ptl/lib/pbs_testlib.py", line 8296, in expect
    extend, level=level, msg=" ".join(msg))
  File "/usr/local/lib/python2.7/dist-packages/ptl/lib/pbs_testlib.py", line 8296, in expect
    extend, level=level, msg=" ".join(msg))
  File "/usr/local/lib/python2.7/dist-packages/ptl/lib/pbs_testlib.py", line 8296, in expect
    extend, level=level, msg=" ".join(msg))
  File "/usr/local/lib/python2.7/dist-packages/ptl/lib/pbs_testlib.py", line 8296, in expect
    extend, level=level, msg=" ".join(msg))
  File "/usr/local/lib/python2.7/dist-packages/ptl/lib/pbs_testlib.py", line 8105, in expect
    raise PtlExpectError(rc=1, rv=False, msg=_msg)
PtlExpectError: rc=1, rv=False, msg=expected on server testdev: server_state != Scheduling server testdev.pbspro.org attempt: 60  got: server_state = Scheduling


2018-12-07 09:18:41,943 INFO     ================================================================================
failed: test_fairshare (tests.pbs_smoketest.SmokeTest)
skipped: test_cray_compute_job (tests.functional.pbs_cray_smoketest.TestCraySmokeTest) reason: Test suite only meant to run on a Cray
skipped: test_cray_login_job (tests.functional.pbs_cray_smoketest.TestCraySmokeTest) reason: Test suite only meant to run on a Cray
skipped: test_default_restrict_res_to_release_on_suspend_setting (tests.functional.pbs_cray_suspend_resume.TestSuspendResumeOnCray) reason: Test suite only meant to run on a Cray
skipped: test_settings (tests.functional.pbs_cray_vnode_per_numa.TestVnodePerNumaNode) reason: Test suite only meant to run on a Cray
Test cases with failures: SmokeTest.test_fairshare
Test suites with failures: SmokeTest
run: 57, succeeded: 52, failed: 1, errors: 0, skipped: 4, timedout: 0
Tests run in 0:08:26.529429
2018-12-07 09:18:41,943 INFO     Cleaning up temporary files
2018-12-07 09:18:41,944 INFO     Cleaning up /var/tmp dir
2018-12-07 09:18:41,944 INFO     Cleaning up /tmp dir
travis_time:end:277442eb:start=1544173814226617642,finish=1544174322052107216,duration=507825489574
The command "docker exec -it -w $(pwd)/test/tests testdev pbs_benchpress --tags=smoke" failed and exited with 1 during .

Your build has been stopped.