Versions Compared

Old Version 28

changes.mady.by.user nithin johnson

Saved on

compared with

New Version Current

changes.mady.by.user nithin johnson

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

  • Interface 1: As an admin or user, I would like qstat to be able to output full job attribute information in json format so that I can more easily write scripts consuming this information.

    • Visibility: Public
    • Change Control: Stable
    • Details: qstat -f will use -F json option to print the output in JSON format(http://www.json.org/).
    • Similar to pbsnodes -av -F json, timestamp, pbs_version and pbs_server will be used as the prolog of the output.

    • > Timestamp - system time in seconds since the Epoch. Server name - default server name.

      > The server name is based on the values configured in pbs.conf. This will follow the order of precedence when specifying server name in pbs.conf

    • Along with full job attributes this new interface is applicable to full server attributes (qstat -Bf -F json) and full queue attributes (qstat -Qf -F json)
      • Usage:
        qstat -f -F json
        qstat -f -F json <job_id>


      • Example: 

        {
             "timestamp":1479277336,
             "pbs_version":"14.1",
             "pbs_server":"vbox",
             "Jobs":{
                         "1.vbox":{
                                 "Job_Name":"STDIN",
                                 "Job_Owner":"root@vbox",
                                 "job_state":"Q",
                                 "queue":"workq",
                                 "server":"vbox",
                                 "Checkpoint":"u",
                                 "ctime":"Fri Nov 11 17:57:05 2016",
                                 "Error_Path":"vbox:/home/nithinj/Tasks/overlay/STDIN.e1",
                                 "Hold_Types":"n",
                                 "Join_Path":"n",
                                 "Keep_Files":"n",
                                 "Mail_Points":"a",
                                 "mtime":"Fri Nov 11 17:57:05 2016",
                                 "Output_Path":"vbox:/home/nithinj/Tasks/overlay/STDIN.o1",
                                 "Priority":0,
                                 "qtime":"Fri Nov 11 17:57:05 2016",
                                 "Rerunable":"True"
                                 "Resource_List":{

                                       "ncpus":1,
                                       "nodect":1,
                                       "place":"pack",
                                       "select":"1:ncpus=1"
                                  },             

                                 "schedselect":"1:ncpus=1",
                                 "substate":10,
                                 "Variable_List":{
                                      "PBS_O_HOME":"/root",
                                      "PBS_O_LANG":"en_US.utf8",
                                      "PBS_O_LOGNAME":"nithinj",
                                      "PBS_O_PATH":"/root/perl5/bin:/usr/lib64/qt- 3.3/bin:/home/nithinj/perl5/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:/usr/pbs/bin:/home/nithinj/.local/bin:/home/nithinj/bin:/opt/pbs/default/bin:/usr/local/git/bin:/opt/pbs/default/bin:/opt/pbs/default/sbin:/usr/local/git/bin",
                                      "PBS_O_MAIL":"/var/spool/mail/nithinj",
                                      "PBS_O_SHELL":"/bin/bash",
                                      "PBS_O_WORKDIR":"/home/nithinj/Tasks/overlay",
                                      "PBS_O_SYSTEM":"Linux",
                                      "PBS_O_QUEUE":"workq",
                                      "PBS_O_HOST" :"vbox"
                                  },
                                 "euser":"root",
                                 "egroup":"root",
                                 "queue_rank":1,
                                 "queue_type":"E",
                                 "comment":"Not Running: Not enough free nodes available",
                                 "etime":"Fri Nov 11 17:57:05 2016",
                                 "Submit_arguments":"-- /bin/sleep 1000",
                                 "executable":"<jsdl-hpcpa:Executable>/bin/sleep</jsdl-hpcpa:Executable>",
                                 "argument_list":"<jsdl-hpcpa:Argument>1000</jsdl-hpcpa:Argument>",
                                 "project":"_pbs_project_default"
                            }
                  }
        }

                  Special case (when user variables contains special characters):   Json output with special characters



    • Interface 2: As an admin or user, I would like qstat to be able to output full job attribute information in delimiter separated value format so that I can more easily write scripts consuming this information.



      • Visibility: Public
      • Change Control: Stable
      • Details: qstat -f will use -F dsv option to print the output in delimiter separated value format.
      • Default delimiter will be a pipe.("|")
      • The user can provide custom delimiter using -D option. (character or string)
      • Backslash ("\") is used as the escape character on Unix/Linux if delimiter itself is appearing in the value.
      • Caret ("^") is used as the escape character on Windows.
      • newline is used as the separator between multiple jobs. Using the newline as the delimiter will lead to undefined behavior.

      • Since feed characters could break the dsv format, they are converted into the following if appears in the value.

      • Along with full job attributes this new interface is applicable to full server attributes (qstat -Bf -F dsv) and full queue attributes (qstat -Qf -F dsv)

          newline → '\'+'n' and formfeed → '\'+'f'   (plus indicated that it is a concatenation)



    • Interface 3: New argument valid in conjunction with qstat -f


      • Visibility: Public
      • Change Control: Stable
      • Details: User/Admin is able to request a the output of qstat to return the full job output with each attribute on its own line (i.e. not splitting up the value of an attribute across multiple lines if it is greater than 78 chars) This is done by using "-w" in conjunction with "-f"
        • Usage:
          qstat -fw
          qstat -fw <job_id>


        • Using -w with -f is now accepted. All other usages of -w shall behave as previously defined for qstat
        • Since feed characters could break the one line output format, they are converted into the following.
            newline → '\'+'n' and formfeed → '\'+'f'   (plus indicated that it is a concatenation)
        • Along with full job attributes this new interface is applicable to full server attributes (qstat -Bf -w) and full queue attributes (qstat -Qf -w)

        • Example output:
        • [centos@pbs-server ~]$ /tmp/qstat -fw 88
          Job Id: 88.pbs-server
              Job_Name = swarm-worker
              Job_Owner = centos@pbs-server
              resources_used.cpupercent = 0
              resources_used.cput = 00:00:00
              resources_used.mem = 367016kb
              resources_used.ncpus = 1
              resources_used.swarm_id = pbs-server
              resources_used.swarm_node_status = {Resources: {Used: {cpu_time: 622, memory: 366908k}, Total: {ncpu: 2, memory: 256mb}}, Containers: {Paused: 0, Running: 20, Total: 22, Stopped: 2}}
              resources_used.vmem = 4316kb
              resources_used.walltime = 23:53:03
              job_state = R
              queue = workq
              server = pbs-server
              Checkpoint = u
              ctime = Mon Oct 17 21:07:50 2016
              Error_Path = pbs-server:/home/centos/swarm-worker.e88
              exec_host = pbs-node01/0
              exec_vnode = (pbs-node01:ncpus=1)
              Hold_Types = n
              Join_Path = n
              Keep_Files = n
              Mail_Points = a
              mtime = Tue Oct 18 21:00:51 2016
              Output_Path = pbs-server:/home/centos/swarm-worker.o88
              Priority = 0
              qtime = Mon Oct 17 21:07:50 2016
              Rerunable = True
              Resource_List.ncpus = 1
              Resource_List.nodect = 1
              Resource_List.place = excl
              Resource_List.select = 1:ncpus=1
              stime = Mon Oct 17 21:07:50 2016
              session_id = 3690
              jobdir = /home/centos
              substate = 42
              Variable_List = PBS_O_SYSTEM=Linux,PBS_O_SHELL=/bin/bash,PBS_O_HOME=/home/centos,PBS_O_HOST=pbs-server,swarm_id=pbs-server,PBS_O_LOGNAME=centos,PBS_O_WORKDIR=/home/centos,PBS_O_LANG=en_US.UTF-8,PBS_O_PATH=/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/pbs/bin:/opt/pbs/bin:/opt/pbs/sbin:/opt/pbs/unsupported:/home/centos/.local/bin:/home/centos/bin,PBS_O_QUEUE=workq,PBS_O_MAIL=/var/spool/mail/centos
              comment = Job run at Mon Oct 17 at 21:07 on (pbs-node01:ncpus=1)
              etime = Mon Oct 17 21:07:50 2016
              run_count = 1
              Submit_arguments = -N swarm-worker -lplace=excl -lselect=1:ncpus=1 -v swarm_id=pbs-server -- /bin/sleep 660000
              executable = <jsdl-hpcpa:Executable>/bin/sleep</jsdl-hpcpa:Executable>
              argument_list = <jsdl-hpcpa:Argument>660000</jsdl-hpcpa:Argument>
              project = _pbs_project_default


     NOTE:

            All the three new interfaces will be supported only with full set of attributes (" -f " option). This would include every vaild combination of "-f" (Ex: "qstat -Qf" or "qstat -Bf")

            Using it with other optargs will throw the already existing "conflicting options" error and prints the usage on the screen.


Follow the community discussion for json and dsv options

Community discussion for qstat -fw option