diff --git a/2026/HPCIC-AWS/README.md b/2026/HPCIC-AWS/README.md new file mode 100644 index 00000000..f2eea8ce --- /dev/null +++ b/2026/HPCIC-AWS/README.md @@ -0,0 +1,382 @@ +# Flux + Usernetes + Jupyter via KubeSpawner + +This set of tutorials provides: + + - [Building Base Images](#build-images) + - [Deploy A Cluster to AWS](#deploy-to-kubernetes) + +Pre-requisites: + + - AWS account and aws client installed locally + - Excitement to learn about Flux! + +For AWS Tutorial Day users: + +> To run the AWS tutorial, visit https://tutorial.flux-framework.org. You can use any login you want, but choose something relatvely uncommon (like your email address) or you may end up sharing a JupyterLab instance with another user. The tutorial password will be provided to you. + +Since we start usernetes and build the container, the startup takes a few minutes, and we should start a little early (before the user expects to need it). + +## Build Images + +Unlike previous tutorials, since this one uses Usernetes and Flux, it is done via an EC2 instance, and we have built a custom EC2 spawner for it. This was built on a hpc7g.16xlarge (Graviton 3) instance, and then saved to an AMI. Incremental changes were saved from the image directly. Note that we needed to build the docker image for usernetes to be cached on the node before a save: + +```bash +cd /home/ubuntu/usernetes +docker build -t usernetes_node . +``` + +The base logic is in [build](build). + +## Deploy to AWS + +**Do not forget to use the RADIUSS account** + +### 1. Setup + +#### IAM Policy + +```bash +# Create the IAM policies +aws iam create-policy --policy-name JupyterHub-EC2-Manager-Policy --policy-document file://ec2/jupyterhub-ec2-policy.json +``` +```console +{ + "Policy": { + "PolicyName": "JupyterHub-EC2-Manager-Policy", + "PolicyId": "ANPAZHDKVUIEA2NWPQFS4", + "Arn": "arn:aws:iam::633731392008:policy/JupyterHub-EC2-Manager-Policy", + "Path": "/", + "DefaultVersionId": "v1", + "AttachmentCount": 0, + "PermissionsBoundaryUsageCount": 0, + "IsAttachable": true, + "CreateDate": "2025-07-31T22:54:06+00:00", + "UpdateDate": "2025-07-31T22:54:06+00:00" + } +} +``` +And: + +```bash +aws iam create-policy --policy-name JupyterHub-PassRole-Policy --policy-document file://ec2/jupyterhub-passrole-policy.json +``` +```console +{ + "Policy": { + "PolicyName": "JupyterHub-PassRole-Policy", + "PolicyId": "ANPAZHDKVUIEMA2OTJQQS", + "Arn": "arn:aws:iam::633731392008:policy/JupyterHub-PassRole-Policy", + "Path": "/", + "DefaultVersionId": "v1", + "AttachmentCount": 0, + "PermissionsBoundaryUsageCount": 0, + "IsAttachable": true, + "CreateDate": "2025-08-01T00:47:28+00:00", + "UpdateDate": "2025-08-01T00:47:28+00:00" + } +} +``` + +```bash +# Create the IAM role that EC2 can assume +aws iam create-role --role-name JupyterHub-EC2-Manager-Role --assume-role-policy-document '{ + "Version": "2012-10-17", + "Statement": { + "Effect": "Allow", + "Principal": {"Service": "ec2.amazonaws.com"}, + "Action": "sts:AssumeRole" + } +}' +``` +```console +{ + "Role": { + "Path": "/", + "RoleName": "JupyterHub-EC2-Manager-Role", + "RoleId": "AROAZHDKVUIEJADZRZUZQ", + "Arn": "arn:aws:iam::633731392008:role/JupyterHub-EC2-Manager-Role", + "CreateDate": "2025-07-31T22:54:22+00:00", + "AssumeRolePolicyDocument": { + "Version": "2012-10-17", + "Statement": { + "Effect": "Allow", + "Principal": { + "Service": "ec2.amazonaws.com" + }, + "Action": "sts:AssumeRole" + } + } + } +} +``` + +```bash +aws iam attach-role-policy \ + --role-name JupyterHub-EC2-Manager-Role \ + --policy-arn arn:aws:iam::633731392008:policy/JupyterHub-PassRole-Policy +``` + +```bash +# Create an instance profile, which is the container for the role that EC2 uses +aws iam create-instance-profile --instance-profile-name JupyterHub-EC2-Manager-Profile +``` +```console +{ + "InstanceProfile": { + "Path": "/", + "InstanceProfileName": "JupyterHub-EC2-Manager-Profile", + "InstanceProfileId": "AIPAZHDKVUIEA6K5RNA33", + "Arn": "arn:aws:iam::633731392008:instance-profile/JupyterHub-EC2-Manager-Profile", + "CreateDate": "2025-07-31T22:55:40+00:00", + "Roles": [] + } +} +``` +```bash +aws iam add-role-to-instance-profile --instance-profile-name JupyterHub-EC2-Manager-Profile --role-name JupyterHub-EC2-Manager-Role +``` + +#### Braket Policy (for the Chapter 5 quantum demo) + +The Chapter 5 notebook runs a hybrid quantum-classical QAOA workload against the AWS Braket SV1 simulator. The spawned user instances use the `JupyterHub-EC2-Manager-Role` (see `iam_instance_profile_arn` in `ec2/jupyterhub_config.py`), so we attach a Braket policy to that same role. The Braket SDK then picks up the instance-role credentials from instance metadata — no keys are stored on the VM. + +The policy in `ec2/jupyterhub-braket-policy.json` is least-privilege and, per our AWS admin's recommendation, scopes the device-level actions (`braket:GetDevice`, `braket:CreateQuantumTask`) to just the SV1 device ARN `arn:aws:braket:::device/quantum-simulator/amazon/sv1`. It also allows the task polling/search actions (which do not accept a device ARN as a resource) and S3 access to the `amazon-braket-*` results bucket that SV1 writes to. + +First, capture your AWS account ID into an environment variable so the rest of the commands are copy-paste ready: + +```bash +export ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text) +echo "Using account: ${ACCOUNT_ID}" +``` + +Create the Braket policy and capture its ARN directly from the API response: + +```bash +export BRAKET_POLICY_ARN=$(aws iam create-policy \ + --policy-name JupyterHub-Braket-SV1-Policy \ + --policy-document file://ec2/jupyterhub-braket-policy.json \ + --query 'Policy.Arn' --output text) +echo "Created policy: ${BRAKET_POLICY_ARN}" +``` + +If the policy already exists (so `create-policy` errored), derive its ARN from the account ID instead: + +```bash +export BRAKET_POLICY_ARN="arn:aws:iam::${ACCOUNT_ID}:policy/JupyterHub-Braket-SV1-Policy" +``` + +Attach it to the role used by the spawned user instances: + +```bash +aws iam attach-role-policy \ + --role-name JupyterHub-EC2-Manager-Role \ + --policy-arn "${BRAKET_POLICY_ARN}" +``` + +Finally, enable Braket in the account once by creating its service-linked role (this is an account-level admin step, intentionally NOT granted to the instance role). It is safe to re-run — if the role already exists you get a harmless error. + +```bash +aws iam create-service-linked-role --aws-service-name braket.amazonaws.com +``` + +> Note: if `braket:CreateQuantumTask` is ever denied despite the device-scoped statement, widen the resource on that one action to also include the account's quantum-task ARN. Using the `ACCOUNT_ID` env var from above, that is `arn:aws:braket:us-east-1:${ACCOUNT_ID}:quantum-task/*`. The device-ARN scoping matches AWS's documented device-restriction model and is what the admin recommended, so start there. + +#### Security Grouph + +Create the security group (we will do this once): + +```bash +# Get the default VPC (if subnets are in right region) +VPC_ID=$(aws ec2 describe-vpcs --filters "Name=isDefault,Values=true" --query "Vpcs[0].VpcId" --output text --region us-east-1) + +# Create a new VPC + +# create the security group +SG_ID=$(aws ec2 create-security-group --group-name "JupyterHub-Hub-SG" --description "Security group for the main JupyterHub instance" --vpc-id $VPC_ID --query 'GroupId' --output text --region us-east-1) + +# add rules to the security group for ssh from my address (so nobody is angry with me) +MY_IP=$(curl -s http://checkip.amazonaws.com) +aws ec2 authorize-security-group-ingress --group-id $SG_ID --protocol tcp --port 22 --cidr $MY_IP/32 --region us-east-1 + +# Allow HTTP (port 80) and HTTPS (port 443) from anywhere for users +aws ec2 authorize-security-group-ingress --group-id $SG_ID --protocol tcp --port 80 --cidr 0.0.0.0/0 --region us-east-1 +aws ec2 authorize-security-group-ingress --group-id $SG_ID --protocol tcp --port 443 --cidr 0.0.0.0/0 --region us-east-1 +aws ec2 authorize-security-group-ingress --group-id $SG_ID --protocol tcp --port 8888 --cidr 0.0.0.0/0 --region us-east-1 +aws ec2 authorize-security-group-ingress --group-id $SG_ID --protocol tcp --port 8000 --cidr 0.0.0.0/0 --region us-east-1 +aws ec2 authorize-security-group-ingress --group-id $SG_ID --protocol tcp --port 8001 --cidr 0.0.0.0/0 --region us-east-1 + +# Flux +# Created Security Group with ID: sg-0ddfbe3cd0f9253d3 +``` + +#### Subnet + +```bash +SUBNET_ID=$(aws ec2 describe-subnets --filters "Name=vpc-id,Values=$VPC_ID" "Name=map-public-ip-on-launch,Values=true" --query "Subnets[0].SubnetId" --output text --region us-east-1) +echo "Using Subnet ID: $SUBNET_ID" +# Using Subnet ID: subnet-0e3f3782bee64eede +``` + +#### Launch Instance + +```bash +# 1) Which AZs actually offer hpc7g.16xlarge in us-east-1? +aws ec2 describe-instance-type-offerings \ + --region us-east-1 \ + --location-type availability-zone \ + --filters Name=instance-type,Values=hpc7g.16xlarge \ + --query 'InstanceTypeOfferings[].Location' --output text + +# 2) Grab the first offered AZ into an env var +export TARGET_AZ=$(aws ec2 describe-instance-type-offerings \ + --region us-east-1 \ + --location-type availability-zone \ + --filters Name=instance-type,Values=hpc7g.16xlarge \ + --query 'InstanceTypeOfferings[0].Location' --output text) +echo "Target AZ: $TARGET_AZ" + +# 3) Find the VPC your current (wrong-AZ) subnet belongs to +export VPC_ID=$(aws ec2 describe-subnets --region us-east-1 \ + --subnet-ids "$SUBNET_ID" --query 'Subnets[0].VpcId' --output text) +echo "VPC: $VPC_ID" + +# 4) Reassign SUBNET_ID to a subnet in that VPC that lives in the target AZ +export SUBNET_ID=$(aws ec2 describe-subnets --region us-east-1 \ + --filters Name=vpc-id,Values=$VPC_ID Name=availability-zone,Values=$TARGET_AZ \ + --query 'Subnets[0].SubnetId' --output text) +echo "New subnet: $SUBNET_ID" +``` +```console +us-east-1a +Target AZ: us-east-1a +VPC: vpc-0231726e66ed7bb8b +New subnet: subnet-0b0e2408f9960d3f6 +``` + +```bash +# This AMI has Flux, LAMMPS, Usernetes +AMI_ID="ami-0f8da7e8cf5544ed0" +KEY_NAME="ed-the-dinosaur" +SECURITY_GROUP_ID="sg-0ddfbe3cd0f9253d3" + +aws ec2 run-instances \ + --image-id "$AMI_ID" \ + --region us-east-1 \ + --instance-type "hpc7g.16xlarge" \ + --iam-instance-profile Name="JupyterHub-EC2-Manager-Profile" \ + --key-name "$KEY_NAME" \ + --security-group-ids "$SECURITY_GROUP_ID" \ + --subnet-id "$SUBNET_ID" \ + --associate-public-ip-address \ + --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=jupyter-hub-instance},{Key=Project,Value=HPCIC-2025-Tutorial}]' + +# If we make a startup script. +# --user-data file://hub-startup-script.sh +``` + +You can describe the instance to get the public ip. + +```bash +aws ec2 describe-instances --region us-east-1 +``` + +And ssh. + +```bash +ssh -i ~/.ssh/.pem -o IdentitiesOnly=yes ubuntu@13.220.230.82 +sudo mkdir -p /srv/jupyterhub +# I copied jupyterhub_config.py and ec2_spawner.py there. +# Our custom login page needs to be in ./templates there too +``` + +I added access for my ip address: + +```bash +aws ec2 authorize-security-group-ingress --group-id $SG_ID --protocol tcp --port 8000 --cidr $MY_IP/32 --region us-east-2 +aws ec2 authorize-security-group-ingress --group-id $SG_ID --protocol tcp --port 8081 --cidr $MY_IP/32 --region us-east-2 +``` + +- submit slides to IM + +#### Start Jupyter + +```bash +sudo chown -R ubuntu /srv/jupyterhub/ +TOKEN=$(curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600") +export HUB_CONNECT_IP=$(curl -H "X-aws-ec2-metadata-token: $TOKEN" -s http://169.254.169.254/latest/meta-data/local-ipv4) + +# Kolomogorov + +# Development (no culling) +# ~/.local/bin/jupyterhub -f /srv/jupyterhub/jupyterhub_config_no_culler.py + +# With culling +~/.local/bin/jupyterhub -f /srv/jupyterhub/jupyterhub_config.py + +# To keep running +screen +nohup ~/.local/bin/jupyterhub -f /srv/jupyterhub/jupyterhub_config.py & +``` + +#### SSL / Certificates + +Note that we will want to generate a certificate. First, install and configure certbot. + +```bash +# Certbot! +# tutorial.flux-framework.org +sudo systemctl stop nginx +sudo certbot certonly --standalone +sudo chown -R ubuntu /etc/letsencrypt/ +``` + +Then add this content to `/etc/nginx/sites-available/default` + +And restart: + +```bash +sudo systemctl reload nginx +``` + +#### SSL and Snapshots + +I originally created the AMI in the wrong account. To share between accounts I needed to take off automatic encryption of snapshots. This is `/etc/nginx/sites-enabled/default` + +``` +server { + listen 80; + server_name tutorial.flux-framework.org; + return 301 https://$host$request_uri; +} + +server { + listen 443 ssl; + server_name tutorial.flux-framework.org; + + ssl_certificate /etc/letsencrypt/live/tutorial.flux-framework.org/fullchain.pem; + ssl_certificate_key /etc/letsencrypt/live/tutorial.flux-framework.org/privkey.pem; + + location / { + proxy_pass http://127.0.0.1:8000; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + + # WebSocket support + proxy_http_version 1.1; + #proxy_set_header Upgrade $websocket_upgrade; + proxy_set_header Connection "upgrade"; + } +} +``` + +And then save a new one, getting the volume from the name. + +```bash +# Create the snapshot +aws ec2 create-snapshot --volume-id vol-0db075348c995f4c1 --description "HPCIC Flux Tutorial 2025 JupyterHub EC2 Spawner (save August 5, 2025)" --region us-east-2 + +# and the image +aws ec2 register-image --name "hpcic-flux-tutorial-2025" --description "HPCIC Flux Framework Tutorial (JupyterHub Spawner EC2) 2025" --root-device-name /dev/sda1 --block-device-mappings "DeviceName=/dev/sda1,Ebs={SnapshotId=snap-0786282c76b84f55e}" --region us-east-2 +``` diff --git a/2026/HPCIC-AWS/build/build-ubuntu.sh b/2026/HPCIC-AWS/build/build-ubuntu.sh new file mode 100644 index 00000000..f3fe29fc --- /dev/null +++ b/2026/HPCIC-AWS/build/build-ubuntu.sh @@ -0,0 +1,429 @@ +#!/bin/bash + +set -euo pipefail + +################################################################ +# +# Flux, Usernetes, Jupyter, and EFA +# I started with ubuntu 24.04 ARM server edition + +sudo apt-get update && sudo apt-get install -y build-essential \ + tar \ + autoconf \ + automake \ + make \ + wget \ + git \ + gcc \ + g++ \ + zip \ + libblas-dev \ + liblapack-dev \ + libfftw3-dev libfftw3-bin \ + libxml2-16 libxml2-dev \ + hdf5-tools \ + libhdf5-dev \ + cmake \ + libboost-all-dev \ + && sudo apt-get clean + +# Utilities +sudo apt-get update && \ + sudo apt-get -qq install -y --no-install-recommends \ + apt-utils \ + locales \ + ca-certificates \ + wget \ + man \ + git \ + flex \ + ssh \ + sudo \ + vim \ + luarocks \ + munge \ + lcov \ + ccache \ + lua5.4 \ + python3-dev \ + python3-pip \ + valgrind \ + jq && \ + sudo rm -rf /var/lib/apt/lists/* + +# Compilers, autotools +sudo apt-get update && \ + sudo apt-get -qq install -y --no-install-recommends \ + build-essential \ + pkg-config \ + autotools-dev \ + libtool \ + libffi-dev \ + autoconf \ + automake \ + make \ + clang \ + clang-tidy \ + gcc \ + g++ && \ + sudo rm -rf /var/lib/apt/lists/* + +sudo pip install --upgrade --ignore-installed --break-system-packages \ + "markupsafe==2.0.0" \ + coverage cffi ply six pyyaml "jsonschema>=2.6,<4.0" \ + sphinx sphinx-rtd-theme sphinxcontrib-spelling + +sudo apt-get update && \ + sudo apt-get -qq install -y --no-install-recommends \ + libsodium-dev \ + libzmq3-dev \ + certbot \ + nginx \ + libczmq-dev \ + libjansson-dev \ + libmunge-dev \ + libncursesw5-dev \ + liblua5.2-dev \ + liblz4-dev \ + libsqlite3-dev \ + uuid-dev \ + libhwloc-dev \ + libs3-dev \ + libevent-dev \ + libarchive-dev \ + libpam-dev && \ + sudo rm -rf /var/lib/apt/lists/* + +# Testing utils and libs +sudo apt-get update && \ + sudo apt-get -qq install -y --no-install-recommends \ + faketime \ + libfaketime \ + pylint \ + cppcheck \ + enchant-2 \ + aspell \ + aspell-en && \ + sudo rm -rf /var/lib/apt/lists/* + +sudo locale-gen en_US.UTF-8 + +# NOTE: luaposix installed by rocks due to Ubuntu bug: #1752082 https://bugs.launchpad.net/ubuntu/+source/lua-posix/+bug/1752082 +sudo luarocks install luaposix + +# Install openpmix, prrte. Openpmix (pmix2) seems to be installed by hwloc, v5.x +sudo mkdir -p /opt/flux +sudo chown -R ubuntu /opt/flux +cd /opt/flux +git clone --recurse-submodules https://github.com/openpmix/prrte.git && \ + cd prrte && \ + git checkout v3.0.1 && \ + ./autogen.pl && \ + ./configure --prefix=/opt/prrte && \ + make -j && sudo make -j all install + +export LANG=C.UTF-8 +export FLUX_SECURITY_VERSION=0.15.0 + +cd /opt/flux +CCACHE_DISABLE=1 && \ + V=$FLUX_SECURITY_VERSION && \ + PKG=flux-security-$V && \ + URL=https://github.com/flux-framework/flux-security/releases/download && \ + wget ${URL}/v${V}/${PKG}.tar.gz && \ + tar xvfz ${PKG}.tar.gz && \ + cd ${PKG} && \ + ./configure --prefix=/usr --sysconfdir=/etc || cat config.log && \ + make -j 4 && \ + sudo make install && \ + cd .. && \ + rm -rf flux-security-* + +# Setup MUNGE directories & key +sudo mkdir -p /var/run/munge && \ + dd if=/dev/urandom bs=1 count=1024 > munge.key && sudo mv munge.key /etc/munge/munge.key && \ + sudo chown -R munge /etc/munge/munge.key /var/run/munge && \ + sudo chmod 600 /etc/munge/munge.key + +cd /opt/flux +export FLUX_CORE_VERSION=0.86.0 +wget https://github.com/flux-framework/flux-core/releases/download/v${FLUX_CORE_VERSION}/flux-core-${FLUX_CORE_VERSION}.tar.gz && \ + tar xzvf flux-core-${FLUX_CORE_VERSION}.tar.gz && \ + cd flux-core-${FLUX_CORE_VERSION} && \ + ./configure --prefix=/usr --sysconfdir=/etc && \ + make clean && \ + make -j && \ + sudo make install + +sudo apt-get update +sudo apt-get -qq install -y --no-install-recommends \ + libboost-graph-dev \ + libboost-system-dev \ + libboost-filesystem-dev \ + libboost-regex-dev \ + libyaml-cpp-dev \ + libedit-dev \ + libboost-dev \ + libyaml-cpp-dev \ + curl + +export FLUX_SCHED_VERSION=0.52.0 +cd /opt/flux +wget https://github.com/flux-framework/flux-sched/releases/download/v${FLUX_SCHED_VERSION}/flux-sched-${FLUX_SCHED_VERSION}.tar.gz && \ + tar -xzvf flux-sched-${FLUX_SCHED_VERSION}.tar.gz && \ + cd flux-sched-${FLUX_SCHED_VERSION} && \ + ./configure --prefix=/usr --sysconfdir=/etc && \ + make -j && \ + sudo make install && \ + sudo ldconfig + +sudo apt-get update && \ + sudo apt-get install -y libfftw3-dev libfftw3-bin pdsh libfabric-dev libfabric1 \ + openssh-client openssh-server \ + dnsutils telnet strace git g++ \ + unzip bzip2 + +# Additional debugging +sudo apt-get update && \ + sudo apt-get install -y pdsh \ + openssh-client openssh-server \ + dnsutils telnet strace \ + unzip bzip2 + +# Install oras for saving artifacts +export VERSION="1.2.0" && \ + curl -LO "https://github.com/oras-project/oras/releases/download/v${VERSION}/oras_${VERSION}_linux_arm64.tar.gz" && \ + mkdir -p oras-install/ && \ + tar -zxf oras_${VERSION}_*.tar.gz -C oras-install/ && \ + sudo mv oras-install/oras /usr/local/bin/ && \ + rm -rf oras_${VERSION}_*.tar.gz oras-install/ + +# Additional packages +sudo apt-get update && sudo apt-get install -y ibverbs-utils libibverbs-dev libibverbs1 && sudo apt-get clean + +sudo curl -O https://efa-installer.amazonaws.com/aws-efa-installer-1.42.0.tar.gz && \ + tar -xzvf aws-efa-installer-1.42.0.tar.gz && \ + rm -rf aws-efa-installer-1.42.0.tar.gz && \ + cd aws-efa-installer && \ + sudo ./efa_installer.sh --skip-kmod --yes + +export PATH=/opt/amazon/openmpi/bin/:$PATH +cd /opt/ +sudo git clone https://github.com/lammps/lammps.git && \ + sudo chown -R ubuntu /opt/lammps && \ + cd /opt/lammps && \ + git fetch --depth 1 origin a8687b53724b630fb5f454c8d7be9f9370f8bb3b && \ + git checkout FETCH_HEAD && \ + mkdir build && \ + cd build && \ + cmake ../cmake -D PKG_REAXFF=yes -D BUILD_MPI=yes -D PKG_OPT=yes -D FFT=FFTW3 -D MPI_CXX_COMPILER=mpicxx \ + -D CMAKE_INSTALL_PREFIX=/usr \ + && make && sudo make install + +# Flux curve.cert +# Ensure we have a shared curve certificate +flux keygen /tmp/curve.cert && \ + sudo mkdir -p /etc/flux/system && \ + sudo cp /tmp/curve.cert /etc/flux/system/curve.cert && \ + sudo chown ubuntu /etc/flux/system/curve.cert && \ + sudo chmod o-r /etc/flux/system/curve.cert && \ + sudo chmod g-r /etc/flux/system/curve.cert && \ + # Permissions for imp + sudo chmod u+s /usr/libexec/flux/flux-imp && \ + sudo chmod 4755 /usr/libexec/flux/flux-imp && \ + # /var/lib/flux needs to be owned by the instance owner + sudo mkdir -p /var/lib/flux && \ + sudo chown ubuntu -R /var/lib/flux + +# Install Usernetes +cd /opt +echo "START updating cgroups2" +cat /etc/default/grub | grep GRUB_CMDLINE_LINUX= +GRUB_CMDLINE_LINUX="" +sudo sed -i -e 's/^GRUB_CMDLINE_LINUX=""/GRUB_CMDLINE_LINUX="systemd.unified_cgroup_hierarchy=1"/' /etc/default/grub +sudo update-grub +sudo mkdir -p /etc/systemd/system/user@.service.d + +cd /opt/lammps +cat </dev/null +br_netfilter +vxlan +EOF + +sudo mv ./usernetes.conf /etc/modules-load.d/usernetes.conf +sudo systemctl restart systemd-modules-load.service +echo "DONE updating kernel modules" + +echo "START 99-usernetes.conf" +echo "net.ipv4.conf.default.rp_filter = 2" > /tmp/99-usernetes.conf +sudo mv /tmp/99-usernetes.conf /etc/sysctl.d/99-usernetes.conf +sudo sysctl --system +echo "DONE 99-usernetes.conf" + +echo "START modprobe" +sudo modprobe vxlan +sudo systemctl daemon-reload + +# https://github.com/rootless-containers/rootlesskit/blob/master/docs/port.md#exposing-privileged-ports +cp /etc/sysctl.conf ./sysctl.conf +echo "net.ipv4.ip_unprivileged_port_start=0" | tee -a ./sysctl.conf +echo "net.ipv4.conf.default.rp_filter=2" | tee -a ./sysctl.conf +sudo mv ./sysctl.conf /etc/sysctl.conf + +sudo sysctl -p +sudo systemctl daemon-reload +echo "DONE modprobe" + +echo "START kubectl" +cd /tmp +curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/arm64/kubectl" +chmod +x ./kubectl +sudo mv ./kubectl /usr/bin/kubectl +echo "DONE kubectl" + +echo "Installing docker" +curl -o install.sh -fsSL https://get.docker.com +chmod +x install.sh +sudo ./install.sh +echo "done installing docker" + +# https://github.com/docker/docs/issues/14491 +sudo apt install -y systemd-container + +sudo chown -R ubuntu /home/ubuntu +echo "Setting up usernetes" +echo "export PATH=/usr/bin:$PATH" >> ~/.bashrc +echo "export XDG_RUNTIME_DIR=/home/ubuntu/.docker/run" >> ~/.bashrc +# This wants to write into run, which is probably OK (under userid) +echo "export DOCKER_HOST=unix:///home/ubuntu/.docker/run/docker.sock" >> ~/.bashrc + +echo "Installing docker user" +sudo loginctl enable-linger ubuntu +ls /var/lib/systemd/linger +mkdir -p /home/ubuntu/.docker/run + +# This might show failure because it creates the docker.sock in /run/user/UID/docker.sock +# but then we link to the expected path below +sudo apt-get install -y uidmap +dockerd-rootless-setuptool.sh install +sleep 10 +systemctl --user enable docker.service +systemctl --user start docker.service + +# Not sure why this is happening, but it's starting here +# As long as docker run hello world works we are good! +ln -s /run/user/1000/docker.sock /home/ubuntu/.docker/run/docker.sock +docker run hello-world + +sudo sysctl -w net.ipv4.ip_unprivileged_port_start=0 +sysctl net.ipv4.ip_unprivileged_port_start # confirm it reads 0 + +# Write scripts to start control plane and worker nodes +# Clone usernetes and usernetes-python +git clone https://github.com/rootless-containers/usernetes ~/usernetes + +echo "Done installing docker user" +sudo chown ubuntu /etc/flux/system/curve.cert +sudo chown -R ubuntu /home/ubuntu + +# Additional requirements for Jupyter +export NB_USER=ubuntu +export NB_UID=1000 + +sudo apt-get install -y dnsutils \ + iputils-ping \ + python3 \ + python3-dev \ + python3-pip \ + python3-venv \ + tini + +sudo apt-get install -y python3-greenlet +pip3 install ruamel.yaml.clib --break-system-packages +wget https://gist.githubusercontent.com/vsoch/2b66b6f2b3885fc9c747e38cc73b78e2/raw/7d596101d87e20e55b5cac589acadb232b29ed3c/requirements.txt +sudo python3 -m pip install -r requirements.txt --break-system-packages +sudo python3 -m pip install ipykernel --break-system-packages +sudo python3 -m pip install pycurl --break-system-packages +sudo python3 -m pip install ipython --break-system-packages && sudo python3 -m IPython kernel install + +# Flux accounting +git clone https://github.com/flux-framework/flux-accounting && \ + cd flux-accounting && \ + ./autogen.sh && \ + ./configure --prefix=/usr && \ + make -j && sudo make install + +# This is for arm +wget https://nodejs.org/dist/v20.15.0/node-v20.15.0-linux-arm64.tar.xz && \ + sudo apt-get update && sudo apt-get install -y xz-utils && sudo rm -rf /var/lib/apt/lists/* && \ + xz -d -v node-v20.15.0-linux-arm64.tar.xz && \ + sudo tar -C /usr/local --strip-components=1 -xvf node-v20.15.0-linux-arm64.tar + +# sudo apt-get purge -y nodejs npm +# sudo apt-get autoremove -y +# curl -fsSL https://deb.nodesource.com/setup_20.x | sudo -E bash - +# sudo apt-get install -y nodejs +# sudo rm -rf /usr/local/bin/node + +# This customizes the launcher UI +# https://jupyter-app-launcher.readthedocs.io/en/latest/usage.html +sudo python3 -m pip install jupyter_app_launcher --break-system-packages && \ +sudo python3 -m pip install --upgrade jupyter-server --break-system-packages && \ +sudo python3 -m pip install jupyter-launcher-shortcuts --break-system-packages && \ +sudo python3 -m pip install jupyterhub_idle_culler --break-system-packages +sudo mkdir -p /usr/local/share/jupyter/lab/jupyter_app_launcher +python3 -m pip install jupyterhub boto3 --break-system-packages +python3 -m pip install git+https://github.com/kubeflow/sdk.git@main#subdirectory=python --break-system-packages +sudo python3 -m pip install river riverapi --break-system-packages +sudo python3 -m pip install ipywidgets --break-system-packages + +# For Chapter 5: hybrid quantum-classical QAOA on AWS Braket SV1 +sudo python3 -m pip install amazon-braket-sdk scipy --break-system-packages + +sudo apt-get install -y bash-completion + +# This is for riverML +sudo python3 -m pip install river riverapi --break-system-packages + +# COPY ./tutorial /home/jovyan/ +# COPY ./docker/jupyter-launcher.yaml /usr/local/share/jupyter/lab/jupyter_app_launcher/jp_app_launcher.yaml +# ENV JUPYTER_APP_LAUNCHER_PATH=/usr/local/share/jupyter/lab/jupyter_app_launcher/ +# Give jovyan user permissions to tutorial materials +# RUN chmod -R 777 ~/ /home/jovyan +# COPY ./docker/flux-icon.png $HOME/flux-icon.png +# note that previous examples are added via git volume in config.yaml +# ENV SHELL=/usr/bin/bash +# ENV FLUX_URI_RESOLVE_LOCAL=t +# This is for JupyterHub +# COPY ./docker/entrypoint.sh /entrypoint.sh +# This is for a local start +# COPY ./docker/start.sh /start.sh +mkdir -p $HOME/.local/share && \ +chmod 777 $HOME/.local/share + +# Quick setup of flux-accounting (not working due to needing system service) +# RUN flux start /bin/bash -c "nohup flux account create-db && flux account-service & flux account add-bank root 1" && \ +# flux start flux account add-bank --parent-bank=root default 1 && \ +# flux start flux account add-user --username=jovyan --bank=default && \ +# flux start flux jobtap load mf_priority.so && \ +# flux start flux account-update-db + +# sudo rm /usr/local/bin/node +sudo npm install -g configurable-http-proxy +# CMD ["flux", "start", "--test-size=4", "jupyter", "lab"] +# +# Build usernetes image +cd /home/ubuntu/usernetes +docker build -t usernetes_node . +# At this point we have what we need! diff --git a/2026/HPCIC-AWS/ec2/docker-compose.yaml b/2026/HPCIC-AWS/ec2/docker-compose.yaml new file mode 100644 index 00000000..c4394bfa --- /dev/null +++ b/2026/HPCIC-AWS/ec2/docker-compose.yaml @@ -0,0 +1,61 @@ +# Use `make up`, not `docker compose up`, +# as this YAML requires ${...} variables to be set. +--- +services: + node: + image: usernetes_node + #build: . + hostname: ${NODE_NAME} + privileged: true + restart: always + networks: + default: + ipv4_address: ${NODE_IP} + ports: + # This maps Host Port 80 to Container Port 80 + - "80:80" + # : + # etcd (default: 2379) + - ${PORT_ETCD}:${PORT_ETCD} + # kube-apiserver (default: 6443) + - ${PORT_KUBE_APISERVER}:${PORT_KUBE_APISERVER} + # kubelet (default: 10250) + - ${PORT_KUBELET}:${PORT_KUBELET} + # flannel (default: 8472) + - ${PORT_FLANNEL}:${PORT_FLANNEL}/udp + volumes: + - .:/usernetes:ro + - /boot:/boot:ro + - /lib/modules:/lib/modules:ro + - node-var:/var + - node-opt:/opt + - node-etc:/etc + - type: tmpfs + target: /run + - type: tmpfs + target: /tmp + working_dir: /usernetes + environment: + KUBECONFIG: /etc/kubernetes/admin.conf + HOST_IP: ${HOST_IP} + sysctls: + - net.ipv4.ip_forward=1 + # In addition, `net.ipv4.conf.default.rp_filter` + # has to be set to 0 (disabled) or 2 (loose) + # in the daemon's network namespace. + annotations: + # Accelerate network for nerdctl >= 2.0.0-beta.4 with bypass4netns >= 0.4.1 + "nerdctl/bypass4netns": "${BYPASS4NETNS:-false}" + "nerdctl/bypass4netns-ignore-bind": "true" + "nerdctl/bypass4netns-ignore-subnets": "${BYPASS4NETNS_IGNORE_SUBNETS:-}" +networks: + default: + ipam: + config: + # Each of the nodes has to have a different IP. + # The node IP here is not accessible from other nodes. + - subnet: ${NODE_SUBNET} +volumes: + node-var: {} + node-opt: {} + node-etc: {} diff --git a/2026/HPCIC-AWS/ec2/ec2_spawner.py b/2026/HPCIC-AWS/ec2/ec2_spawner.py new file mode 100644 index 00000000..9826ab05 --- /dev/null +++ b/2026/HPCIC-AWS/ec2/ec2_spawner.py @@ -0,0 +1,313 @@ +import asyncio + +import boto3 +from jupyterhub.spawner import Spawner +from traitlets import Bool, Dict, List, Unicode + +# The default AMI disk size is small and we get disk pressure +# So let's try increasing. +desired_root_volume_size_gb = 256 + +# Define the block device mappings - note that lsblk will show /dev/nvme0n1 but it's this one +block_device_mappings = [ + { + "DeviceName": "/dev/sda1", + "Ebs": { + "VolumeSize": desired_root_volume_size_gb, + "DeleteOnTermination": True, + "VolumeType": "gp2", + }, + } +] + + +class EC2Spawner(Spawner): + """ + A Spawner for JupyterHub that launches a user's server on a new EC2 instance. + """ + + ami = Unicode( + "ami-06cebbfb446ee0ceb", + config=True, + help="The AMI ID for the Flux + Usernetes VM.", + ) + + # The EC2 instance type + instance_type = Unicode( + "hpc7g.16xlarge", config=True, help="The EC2 instance type." + ) + + # Always need the region + region = Unicode("us-east-1", config=True, help="Region to deploy instance to") + + # The name of the EC2 key pair (leaving empty to not allow access) + key_name = Unicode( + "", config=True, help="The name of the EC2 key pair for ssh access." + ) + + # List of security group IDs + security_group_ids = List( + [], config=True, help="List of security group IDs for the instance." + ) + + # The IAM instance profile to attach to the instance + iam_instance_profile_arn = Unicode( + "", + config=True, + help="The ARN of the IAM instance profile to attach to the instance.", + ) + + # Tags to apply to the instance + instance_tags = Dict( + {"ManagedBy": "JupyterHub"}, + config=True, + help="Tags to apply to the spawned EC2 instance.", + ) + + # The subnet to launch the instance in + subnet_id = Unicode( + "", config=True, help="The subnet ID to launch the instance in." + ) + + # We need to save the instance ID to be able to stop and poll it later. + instance_id = Unicode() + + def _get_user_data(self): + """ + Generates the UserData script to start the jupyter-lab server. + This version correctly handles single quotes in environment variables. + """ + user = "ubuntu" + + # Get environment variables from JupyterHub + env_vars = self.get_env() + + # Properly escape single quotes within environment variable values + # before wrapping them in single quotes for the export command. + env_export_cmds = [] + for key, value in env_vars.items(): + # 1. Convert value to string + value_str = str(value) + # 2. Escape any single quotes by replacing ' with '\'' + escaped_value = value_str.replace("'", r"'\''") + # 3. Create the export command + env_export_cmds.append(f"export {key}='{escaped_value}'") + + # The user under which the jupyter-lab server will run + # Script is written to: /var/lib/cloud/instance/scripts/part-001 + script = f"""#!/bin/bash +exec >> /var/log/user-data.log 2>&1 +date +# This should be the script path +echo "$0" + +# Load modules for usernetes +sudo modprobe ip6_tables +sudo modprobe ip6table_nat +sudo modprobe iptable_nat +sudo chown -R ubuntu /home/ubuntu + +# Just in case... +rm -rf ~/.aws + +# Someone is going to type this +# sudo ln -s $(which python3) /usr/bin/python + +# Clone repository tutorials, expose tutorial notebooks +mkdir -p /home/ubuntu/.local/share/jupyter/jupyter_app_launcher /home/ubuntu/.jupyter/lab/static /srv/jupyterlab/static +git clone --depth 1 -b hpcic-2026 https://github.com/flux-framework/Tutorials /tmp/tutorials +cp -R /tmp/tutorials/2026/HPCIC-AWS/tutorial /home/ubuntu/tutorial +cp /tmp/tutorials/2026/HPCIC-AWS/ec2/jupyter-launcher.yaml /home/ubuntu/.local/share/jupyter/jupyter_app_launcher/jp_app_launcher.yaml +sudo chown -R ubuntu /home/ubuntu/.local/share/jupyter + +cp /tmp/tutorials/2026/HPCIC-AWS/ec2/start-usernetes.sh /home/ubuntu/start-usernetes.sh +chmod +x /home/ubuntu/start-usernetes.sh +sudo chown -R ubuntu /home/ubuntu/ + +# We need to stop nginx if it's running +sudo systemctl stop nginx || true +sudo systemctl disable nginx || true + +sudo rm -rf /home/ubuntu/.jupyter/lab/workspaces +sudo mkdir -p /home/ubuntu/.jupyter/lab/workspaces +sudo chown -R ubuntu /home/ubuntu + +# Use sudo to switch to the user and bash to execute the script +sudo -i -u {user} bash << 'EOF' +#!/bin/bash +set -euo pipefail + +echo "Running as user $(whoami) in $(pwd)" + +# Export all the necessary environment variables for this shell +{chr(10).join(env_export_cmds)} + +echo "Starting JupyterLab..." + +export PYTHONPATH=/usr/lib/python3.12/site-packages +cd /home/ubuntu + +flux start --test-size=4 /usr/local/bin/jupyter-lab \\ + --ip=0.0.0.0 \\ + --port=8888 \\ + --notebook-dir=/home/ubuntu \\ + --preferred-dir=/home/ubuntu \\ + --IdentityProvider.token="" \\ + --ServerApp.password="" \\ + --ServerApp.base_url=$JUPYTERHUB_SERVICE_PREFIX \\ + --JupyterLabApp.hub_api_url=$JUPYTERHUB_API_URL \\ + --JupyterLabApp.hub_activity_url=$JUPYTERHUB_ACTIVITY_URL \\ + --JupyterLabApp.hub_prefix=$JUPYTERHUB_BASE_URL + +echo "JupyterLab command finished." +EOF + +echo "UserData Script Finished ---" +date +""" + return script + + async def start(self): + """ + Start the user's EC2 instance. + """ + self.log.info(f"User {self.user.name}: Requesting to start EC2 instance.") + + ec2 = boto3.resource("ec2", region_name=self.region) + + # Create a unique tag for this user's instance + tags = self.instance_tags.copy() + tags["jupyterhub-user"] = self.user.name + + tag_spec = [ + { + "ResourceType": "instance", + "Tags": [{"Key": k, "Value": v} for k, v in tags.items()], + } + ] + try: + instance = ec2.create_instances( + ImageId=self.ami, + InstanceType=self.instance_type, + KeyName=self.key_name, + BlockDeviceMappings=block_device_mappings, + SecurityGroupIds=self.security_group_ids, + SubnetId=self.subnet_id, + IamInstanceProfile={"Arn": self.iam_instance_profile_arn}, + TagSpecifications=tag_spec, + UserData=self._get_user_data(), + MinCount=1, + MaxCount=1, + )[0] + except Exception as e: + self.log.error( + f"Failed to create EC2 instance for user {self.user.name}: {e}" + ) + raise e + + self.instance_id = instance.id + self.log.info(f"User {self.user.name}: Started instance {self.instance_id}") + + # Wait for the instance to be in 'running' state + self.log.info( + f"User {self.user.name}: Waiting for instance {self.instance_id} to be running..." + ) + + # Poll until the instance is running and has an IP + while True: + await asyncio.sleep(5) + instance.reload() + + if instance.state["Name"] == "running": + if instance.public_ip_address: + self.log.info( + f"User {self.user.name}: Instance {self.instance_id} is running at IP {instance.public_ip_address}" + ) + # Return the IP and port for JupyterHub to proxy to + return (instance.public_ip_address, 8888) + else: + self.log.info( + f"User {self.user.name}: Instance running but waiting for public IP..." + ) + elif instance.state["Name"] in [ + "shutting-down", + "terminated", + "stopping", + "stopped", + ]: + error_msg = f"Instance {self.instance_id} entered state '{instance.state['Name']}' unexpectedly." + self.log.error(error_msg) + raise RuntimeError(error_msg) + + async def stop(self, now=False): + """ + Stop and terminate the user's EC2 instance. + """ + if not self.instance_id: + self.log.info(f"User {self.user.name}: No instance ID found to stop.") + return + + self.log.info( + f"User {self.user.name}: Requesting to terminate instance {self.instance_id}" + ) + ec2 = boto3.resource("ec2", region_name=self.region) + instance = ec2.Instance(self.instance_id) + + try: + instance.terminate() + self.log.info( + f"User {self.user.name}: Successfully terminated instance {self.instance_id}" + ) + except Exception as e: + # Handle cases where instance might already be terminated + if "InvalidInstanceID.NotFound" in str(e): + self.log.warning(f"Instance {self.instance_id} already terminated.") + else: + self.log.error(f"Failed to terminate instance {self.instance_id}: {e}") + raise e + + self.clear_state() + + async def poll(self): + """ + Check if the EC2 instance is still running. + Returns: + None if the instance is running. + 0 if the instance is stopped or terminated. + """ + if not self.instance_id: + return 0 # Not running + + ec2 = boto3.resource("ec2", region_name=self.region) + instance = ec2.Instance(self.instance_id) + + try: + instance.reload() + state = instance.state["Name"] + self.log.debug(f"Polling instance {self.instance_id}, state is: {state}") + + if state == "running": + return None # Still running + else: + return 0 # Not running (stopped, terminated, etc.) + except Exception: + # Instance not found, so it's not running + self.log.warning( + f"Polling failed for instance {self.instance_id}. Assuming it's terminated." + ) + return 0 + + # JupyterHub can remember the instance ID across restarts. + def load_state(self, state): + super().load_state(state) + self.instance_id = state.get("instance_id", "") + + def get_state(self): + state = super().get_state() + if self.instance_id: + state["instance_id"] = self.instance_id + return state + + def clear_state(self): + super().clear_state() + self.instance_id = "" diff --git a/2026/HPCIC-AWS/ec2/jupyter-launcher.yaml b/2026/HPCIC-AWS/ec2/jupyter-launcher.yaml new file mode 100644 index 00000000..da9d3740 --- /dev/null +++ b/2026/HPCIC-AWS/ec2/jupyter-launcher.yaml @@ -0,0 +1,112 @@ +- title: Flux Tutorial Notebook + description: This is the main Flux Framework Tutorial + type: jupyterlab-commands + icon: ./static/flux-icon.png + source: + - label: Flux Tutorial + id: 'filebrowser:open-path' + args: + path: ./tutorial/ch1/01_flux_tutorial.ipynb + icon: ./static/flux-icon.png + catalog: Notebook + +- title: Flux Python SDK Tutorial + description: This notebook will teach you about the Flux Python SDK + type: jupyterlab-commands + icon: ./static/flux-icon.png + source: + - label: Flux Tutorial + id: 'filebrowser:open-path' + args: + path: ./tutorial/ch2/02_flux_framework.ipynb + icon: ./static/flux-icon.png + catalog: Notebook + +- title: Flux Process, Monitoring, Utilities Tutorial + description: This will teach you about Flux utilities + type: jupyterlab-commands + icon: ./static/flux-icon.png + source: + - label: Flux Tutorial + id: 'filebrowser:open-path' + args: + path: ./tutorial/ch3/03_flux_tutorial.ipynb + icon: ./static/flux-icon.png + catalog: Notebook + +- title: Flux Framework and User-space Kubernetes + description: This will teach you about running AI/ML workloads with Flux and Kubernetes + type: jupyterlab-commands + icon: ./static/flux-icon.png + source: + - label: Flux Tutorial + id: 'filebrowser:open-path' + args: + path: ./tutorial/ch4/04_flux_framework_usernetes.ipynb + icon: ./static/flux-icon.png + catalog: Notebook + +- title: Flux Framework and Quantum Computing with AWS Braket + description: This will teach you about running hybrid quantum-classical workflows with Flux and AWS Braket + type: jupyterlab-commands + icon: ./static/flux-icon.png + source: + - label: Flux Tutorial + id: 'filebrowser:open-path' + args: + path: ./tutorial/ch5/05_flux_quantum_braket.ipynb + icon: ./static/flux-icon.png + catalog: Notebook + +- title: Flux Framework Portal + description: Flux Framework portal for projects, releases, and publication. + source: https://flux-framework.org/ + type: url + catalog: Flux Resources + args: + sandbox: [ 'allow-same-origin', 'allow-scripts', 'allow-downloads', 'allow-modals', 'allow-popups'] +- title: Flux Documentation + source: https://flux-framework.readthedocs.io/en/latest/ + type: url + catalog: Flux Resources + args: + sandbox: [ 'allow-same-origin', 'allow-scripts', 'allow-downloads', 'allow-modals', 'allow-popups'] +- title: Flux Cheat Sheet + source: https://flux-framework.org/cheat-sheet/ + type: url + catalog: Flux Resources + args: + sandbox: [ 'allow-same-origin', 'allow-scripts', 'allow-downloads', 'allow-modals', 'allow-popups'] +- title: Flux Glossary of Terms + source: https://flux-framework.readthedocs.io/en/latest/glossary.html + type: url + catalog: Flux Resources + args: + sandbox: [ 'allow-same-origin', 'allow-scripts', 'allow-downloads', 'allow-modals', 'allow-popups'] +- title: Flux Comics + source: https://flux-framework.readthedocs.io/en/latest/comics/fluxonomicon.html + description: come and meet FluxBird - the pink bird who knows things! + type: url + catalog: Flux Resources + args: + sandbox: [ 'allow-same-origin', 'allow-scripts', 'allow-downloads', 'allow-modals', 'allow-popups'] +- title: Flux Learning Guide + source: https://flux-framework.readthedocs.io/en/latest/guides/learning_guide.html + description: learn about what Flux does, how it works, and real research applications + type: url + catalog: Flux Resources + args: + sandbox: [ 'allow-same-origin', 'allow-scripts', 'allow-downloads', 'allow-modals', 'allow-popups'] +- title: Getting Started with Flux and Go + source: https://converged-computing.github.io/flux-go + type: url + catalog: Flux Resources + args: + sandbox: [ 'allow-same-origin', 'allow-scripts', 'allow-downloads', 'allow-modals', 'allow-popups'] +- title: Getting Started with Flux in C + source: https://converged-computing.github.io/flux-c-examples/ + description: ...looking for contributors! + type: url + catalog: Flux Resources + args: + sandbox: [ 'allow-same-origin', 'allow-scripts', 'allow-downloads', 'allow-modals', 'allow-popups'] diff --git a/2026/HPCIC-AWS/ec2/jupyterhub-braket-policy.json b/2026/HPCIC-AWS/ec2/jupyterhub-braket-policy.json new file mode 100644 index 00000000..806852c6 --- /dev/null +++ b/2026/HPCIC-AWS/ec2/jupyterhub-braket-policy.json @@ -0,0 +1,42 @@ +{ + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "BraketSV1DeviceScoped", + "Effect": "Allow", + "Action": [ + "braket:GetDevice", + "braket:CreateQuantumTask" + ], + "Resource": "arn:aws:braket:::device/quantum-simulator/amazon/sv1" + }, + { + "Sid": "BraketTaskPollingAndSearch", + "Effect": "Allow", + "Action": [ + "braket:GetQuantumTask", + "braket:SearchQuantumTasks", + "braket:CancelQuantumTask", + "braket:SearchDevices" + ], + "Resource": "*" + }, + { + "Sid": "BraketResultsBucket", + "Effect": "Allow", + "Action": [ + "s3:CreateBucket", + "s3:ListBucket", + "s3:GetBucketLocation", + "s3:PutObject", + "s3:GetObject", + "s3:PutBucketPublicAccessBlock", + "s3:PutBucketPolicy" + ], + "Resource": [ + "arn:aws:s3:::amazon-braket-*", + "arn:aws:s3:::amazon-braket-*/*" + ] + } + ] +} diff --git a/2026/HPCIC-AWS/ec2/jupyterhub-ec2-policy.json b/2026/HPCIC-AWS/ec2/jupyterhub-ec2-policy.json new file mode 100644 index 00000000..cd347f1c --- /dev/null +++ b/2026/HPCIC-AWS/ec2/jupyterhub-ec2-policy.json @@ -0,0 +1,16 @@ +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "ec2:RunInstances", + "ec2:TerminateInstances", + "ec2:DescribeInstances", + "ec2:DescribeInstanceStatus", + "ec2:CreateTags" + ], + "Resource": "*" + } + ] +} diff --git a/2026/HPCIC-AWS/ec2/jupyterhub-passrole-policy.json b/2026/HPCIC-AWS/ec2/jupyterhub-passrole-policy.json new file mode 100644 index 00000000..133d0c76 --- /dev/null +++ b/2026/HPCIC-AWS/ec2/jupyterhub-passrole-policy.json @@ -0,0 +1,10 @@ +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": "iam:PassRole", + "Resource": "arn:aws:iam::633731392008:role/JupyterHub-EC2-Manager-Role" + } + ] +} diff --git a/2026/HPCIC-AWS/ec2/jupyterhub_config.py b/2026/HPCIC-AWS/ec2/jupyterhub_config.py new file mode 100644 index 00000000..b1a535cc --- /dev/null +++ b/2026/HPCIC-AWS/ec2/jupyterhub_config.py @@ -0,0 +1,173 @@ +# jupyterhub_config.py +# +# Configuration for JupyterHub running on a dedicated EC2 instance, +# spawning single-user servers on separate EC2 instances. +# +# This file should be placed in the same directory as ec2_spawner.py +# and where you run the 'jupyterhub' command. +# +# Assumed Directory Structure: +# /srv/jupyterhub/ +# ├── jupyterhub_config.py (this file) +# └── ec2_spawner.py (the custom spawner class file) + +import os +import sys + +# c is a global traitlets config object that JupyterHub provides. +c = get_config() + +# ----------------------------------------------------------------------------- +# Core JupyterHub Settings +# ----------------------------------------------------------------------------- + +# Tell JupyterHub to use your custom spawner class. +# We add '.' to the path to make sure Python can find ec2_spawner.py +# in the current directory. +sys.path.insert(0, os.path.dirname(__file__)) +c.JupyterHub.spawner_class = "ec2_spawner.EC2Spawner" + +# Custom login page +c.JupyterHub.template_paths = ["/srv/jupyterhub/templates"] +c.JupyterHub.static_paths = ["/srv/jupyterhub/static"] + +# IP and Port for the Hub to listen on. +# '0.0.0.0' makes it listen on all network interfaces. +c.JupyterHub.hub_ip = "127.0.0.1" +c.JupyterHub.hub_port = 8081 +c.JupyterHub.ip = "0.0.0.0" +c.JupyterHub.port = 443 +c.JupyterHub.hub_connect_url = "http://127.0.0.1:8081" + +# Hub and proxy are on the same machine +c.JupyterHub.internal_ssl = False + +# When you add ssl, put the certs here and uncomment the below lines. +# Also comment out he proxy api ip and port above. +c.JupyterHub.ssl_cert = ( + "/etc/letsencrypt/live/tutorial.flux-framework.org/fullchain.pem" +) +c.JupyterHub.ssl_key = ( + "/etc/letsencrypt/live/tutorial.flux-framework.org/privkey.pem" +) + +# The public IP of this Hub machine. +hub_connect_ip = os.environ.get("HUB_CONNECT_IP") +if not hub_connect_ip: + raise RuntimeError( + "HUB_CONNECT_IP environment variable must be set to the private IP of the Hub instance." + ) +c.JupyterHub.hub_connect_ip = hub_connect_ip + + +# ----------------------------------------------------------------------------- +# Authenticator Settings +# ----------------------------------------------------------------------------- +# For a tutorial, DummyAuthenticator is simple. Users can enter any username +# and the password specified below. +# For production, you would switch to OAuthenticator (e.g., with GitHub or Google). +c.JupyterHub.authenticator_class = "dummy" +c.DummyAuthenticator.password = "REPLACEME" + +# Grant admin rights to a specific user for monitoring and management. +c.Authenticator.admin_users = {"admin"} + +# We need to make this longer for usernetes to start +c.Spawner.start_timeout = 300 +c.Spawner.http_timeout = 300 +c.EC2Spawner.http_timeout = 300 +c.EC2Spawner.start_timeout = 300 + +# ----------------------------------------------------------------------------- +# Custom EC2Spawner Settings +# ----------------------------------------------------------------------------- +# These are the settings for your custom ec2_spawner.py class. + +# The AMI ID for the tutorial user VM. This AMI should have Python, JupyterLab, +# Usernetes, and all tutorial dependencies pre-installed. +c.EC2Spawner.ami = "ami-04fde965376f2d904" + +# Instance type for each user's server. +# 't4g.2xlarge' is what had a maximum fom per core ratio in my experiments +c.EC2Spawner.instance_type = "hpc7g.16xlarge" + +# The name of the EC2 key pair for SSH access (for debugging). +# We should remove this for actual tutorial. +c.EC2Spawner.key_name = "dinosaur" +c.Spawner.notebook_dir = "/home/ubuntu" + +# Security Group for the spawned instances. Must allow port 8888 ingress +# from this Hub's security group, and port 22 for SSH access. +c.EC2Spawner.security_group_ids = ["sg-0ddfbe3cd0f9253d3"] + +# The VPC subnet to launch the instances in. Must have internet access. +c.EC2Spawner.subnet_id = "subnet-0b0e2408f9960d3f6" + +# The IAM role for the *spawned* user instances. This role can grant permissions +# to S3, etc., if the tutorial needs it. This is attached to the user VM. +c.EC2Spawner.iam_instance_profile_arn = ( + "arn:aws:iam::633731392008:instance-profile/JupyterHub-EC2-Manager-Profile" +) + +# Custom tags to apply to each spawned EC2 instance for tracking. +c.EC2Spawner.instance_tags = { + "ManagedBy": "JupyterHub", + "Project": "HPCIC-2025-Tutorial", + "Purpose": "User-Server", +} + + +# ----------------------------------------------------------------------------- +# Service: Idle Culler +# ----------------------------------------------------------------------------- +# Shut down (and terminate, via our spawner's stop() method) idle user servers. + +c.JupyterHub.load_roles = [ + { + "name": "jupyterhub-idle-culler-role", + "scopes": [ + "list:users", + "read:users:activity", + "delete:servers", + # The admin:servers scope is required for the culler to see and stop all servers. + "admin:servers", + ], + "services": ["jupyterhub-idle-culler-service"], + } +] + +c.JupyterHub.services = [ + { + "name": "jupyterhub-idle-culler-service", + "admin": True, # The service needs admin permissions to stop other user's servers + "command": [ + sys.executable, + "-m", + "jupyterhub_idle_culler", + # Timeout in seconds. 3600 = 1 hour. + # Set this to a value appropriate for your tutorial. + "--timeout=3600", + # Check every 5 minutes + "--cull-every=300", + ], + } +] + +# ----------------------------------------------------------------------------- +# Persistence +# ----------------------------------------------------------------------------- +# Store the JupyterHub database in a persistent location on the Hub VM. +# This ensures that if you restart JupyterHub, it remembers active users and their +# associated EC2 instance IDs. +c.JupyterHub.db_url = "sqlite:////srv/jupyterhub/jupyterhub.sqlite" + +# ----------------------------------------------------------------------------- +# Other Optional Settings +# ----------------------------------------------------------------------------- + +# Concurrent spawn limit to prevent runaway costs if many people log in at once +c.JupyterHub.concurrent_spawn_limit = 120 + +# Memory limit for the Hub process itself (not the user servers). +# The default is usually fine. +# c.JupyterHub.cookie_secret_file = '/srv/jupyterhub/jupyterhub_cookie_secret' # Optional, but good practice diff --git a/2026/HPCIC-AWS/ec2/login.html b/2026/HPCIC-AWS/ec2/login.html new file mode 100644 index 00000000..d997a0f5 --- /dev/null +++ b/2026/HPCIC-AWS/ec2/login.html @@ -0,0 +1,168 @@ +{% extends "page.html" %} +{% if announcement_login is string %} + {% set announcement = announcement_login %} +{% endif %} + +{% block login_widget %} +{% endblock %} + +{% block stylesheet %} +{{ super() }} + +{% endblock %} + +{% block main %} + +{% block login %} +
+{% block login_container %} + +
+ + + +
+

Flux Tutorial running on AWS

+ +
+
+
+

Sign in

+
+
+ + + + {% if login_error %} + + {% endif %} + + + + + + + + + {% block login_terms %} + {% if login_term_url %} + + {% endif %} + {% endblock login_terms %} + +
+
+
+{% endblock login_container %} +
+{% endblock login %} + +{% endblock %} + +{% block script %} +{{ super() }} + +{% endblock %} diff --git a/2026/HPCIC-AWS/ec2/start-usernetes.sh b/2026/HPCIC-AWS/ec2/start-usernetes.sh new file mode 100644 index 00000000..ab8d78e8 --- /dev/null +++ b/2026/HPCIC-AWS/ec2/start-usernetes.sh @@ -0,0 +1,32 @@ +#!/bin/bash + +cd /home/ubuntu/usernetes +docker rmi usernetes_node +docker build --no-cache -t usernetes_node . +cd - + +make -C /home/ubuntu/usernetes up +sleep 10 +make -C /home/ubuntu/usernetes kubeadm-init +sleep 10 +make -C /home/ubuntu/usernetes install-flannel +make -C /home/ubuntu/usernetes kubeconfig +export KUBECONFIG=/home/ubuntu/usernetes/kubeconfig + +# Untaint the control plane to schedule jobs, etc. on +echo " 🍑 Untainting control plane and labeling node" +control_plane_node=$(kubectl get nodes -l node-role.kubernetes.io/control-plane -o jsonpath='{.items[0].metadata.name}' 2>/dev/null) + +# Ensure the user interacts with default kubeconfig +sudo cp /home/ubuntu/usernetes/kubeconfig /home/ubuntu/.kube/config + +# Taint away! +kubectl taint node "${control_plane_node}" node-role.kubernetes.io/control-plane:NoSchedule- +kubectl label node "${control_plane_node}" node.kubernetes.io/exclude-from-external-load-balancers- + +# This ensures the node is ingress ready +kubectl label node u7s-$(hostname) ingress-ready=true + +echo "Cluster is ready. 🤓" +# echo "export KUBECONFIG=/home/ubuntu/usernetes/kubeconfig" + diff --git a/2026/HPCIC-AWS/tutorial/.gitignore b/2026/HPCIC-AWS/tutorial/.gitignore new file mode 100644 index 00000000..3ebb2aab --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/.gitignore @@ -0,0 +1,2 @@ +flux*.out +.ipynb_checkpoints diff --git a/2026/HPCIC-AWS/tutorial/README.md b/2026/HPCIC-AWS/tutorial/README.md new file mode 100644 index 00000000..9001bfd9 --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/README.md @@ -0,0 +1,3 @@ +# Final Survey + +Please go to: https://converged-computing.org/community-survey-2025 for our 2025 survey. diff --git a/2026/HPCIC-AWS/tutorial/assets/Flux-logo.svg b/2026/HPCIC-AWS/tutorial/assets/Flux-logo.svg new file mode 100644 index 00000000..f2d126bb --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/assets/Flux-logo.svg @@ -0,0 +1 @@ +Flux-logo-3 \ No newline at end of file diff --git a/2026/HPCIC-AWS/tutorial/assets/flux-icon.png b/2026/HPCIC-AWS/tutorial/assets/flux-icon.png new file mode 100644 index 00000000..d50aa52d Binary files /dev/null and b/2026/HPCIC-AWS/tutorial/assets/flux-icon.png differ diff --git a/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/.mergify.yml b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/.mergify.yml new file mode 100644 index 00000000..65c63410 --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/.mergify.yml @@ -0,0 +1,18 @@ +pull_request_rules: + - name: rebase and merge when passing all checks + conditions: + - base=master + - status-success="check formatting (3.6)" + - status-success="check formatting (3.7)" + - status-success="check formatting (3.8)" + - label="merge-when-passing" + - label!="work-in-progress" + - "approved-reviews-by=@flux-framework/core" + - "#approved-reviews-by>0" + - "#changes-requested-reviews-by=0" + - -title~=^\[*[Ww][Ii][Pp] + actions: + merge: + method: merge + strict: smart + strict_method: rebase diff --git a/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/Makefile b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/Makefile new file mode 100644 index 00000000..f2199057 --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/Makefile @@ -0,0 +1,25 @@ +# Minimal makefile for Sphinx documentation +# + +# You can set these variables from the command line, and also +# from the environment for the first two. +SPHINXOPTS ?= +SPHINXBUILD ?= sphinx-build +SOURCEDIR = . +BUILDDIR = _build + +# Put it first so that "make" without argument is like "make help". +help: + @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) + +.PHONY: help Makefile check spelling $(SCHEMA_DIRS) + +# Catch-all target: route all unknown targets to Sphinx using the new +# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). +%: Makefile + @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) + +check: spelling + +spelling: + @$(SPHINXBUILD) -W -b spelling "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) diff --git a/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/README.md b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/README.md new file mode 100644 index 00000000..892bcb04 --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/README.md @@ -0,0 +1,72 @@ +# Flux Workflow Examples + +This contents used to be hosted at [flux-framework/flux-workflow-examples](https://github.com/flux-framework/flux-workflow-examples) and has been moved here for annual updates paired with the Flux Tutorials. + +The examples contained here demonstrate and explain some simple use-cases with Flux, +and make use of Flux's command-line interface (CLI), Flux's C library, +and the Python and Lua bindings to the C library. + +## Requirements + +The examples assume that you have installed: + +1. A recent version of Flux +2. Python 3.6+ +3. Lua 5.1+ + +You can also use an interactive container locally, binding this directory to the container: + +```bash +docker run -it -v $(pwd):/home/fluxuser/flux-workflow-examples fluxrm/flux-sched:jammy +cd /home/fluxuser/flux-workflow-examples/ +``` + +**_1. [CLI: Job Submission](job-submit-cli)_** + +Launch a flux instance and schedule/launch compute and io-forwarding jobs on +separate nodes using the CLI + +**_2. [Python: Job Submission](job-submit-api)_** + +Schedule/launch compute and io-forwarding jobs on separate nodes using the Python bindings + +**_3. [Python: Job Submit/Wait](job-submit-wait)_** + +Submit jobs and wait for them to complete using the Flux Python bindings + +**_4. [Python: Asynchronous Bulk Job Submission](async-bulk-job-submit)_** + +Asynchronously submit jobspec files from a directory and wait for them to complete in any order + +**_5. [Python: Tracking Job Status and Events](job-status-control)_** + +Submit job bundles, get event updates, and wait until all jobs complete + +**_6. [Python: Job Cancellation](job-cancel)_** + +Cancel a running job + +**_7. [Lua: Use Events](synchronize-events)_** + +Use events to synchronize compute and io-forwarding jobs running on separate +nodes + +**_8. [Python: Simple KVS Example](kvs-python-bindings)_** + +Use KVS Python interfaces to store user data into KVS + +**_9. [CLI/Lua: Job Ensemble Submitted with a New Flux Instance](job-ensemble)_** + +Submit job bundles, print live job events, and exit when all jobs are complete + +**_10. [CLI: Hierarchical Launching](hierarchical-launching)_** + +Launch a large number of sleep 0 jobs + +**_11. [C/Lua: Use a Flux Comms Module](comms-module)_** + +Use a Flux Comms Module to communicate with job elements + +**_12. [C/Python: A Data Conduit Strategy](data-conduit)_** + +Attach to a job that receives OS time data from compute jobs diff --git a/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/async-bulk-job-submit/README.md b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/async-bulk-job-submit/README.md new file mode 100644 index 00000000..c612e517 --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/async-bulk-job-submit/README.md @@ -0,0 +1,113 @@ +# Python Asynchronous Bulk Job Submission + +Parts (a) and (b) demonstrate different implementations of the same basic use-case---submitting large numbers of jobs to Flux. For simplicity, in these examples all of the jobs are identical. + +In part (a), we use the `flux.job.submit_async` and `flux.job.wait` functions to submit jobs and wait for them. +In part (b), we use the `FluxExecutor` class, which offers a higher-level interface. It is important to note that +these two different implementations deal with very different kinds of futures. +The executor's futures fulfill in the background and callbacks added to the futures may +be invoked by different threads; the `submit_async` futures do not fulfill in the background, callbacks are always +invoked by the same thread that added them, and sharing the futures among threads is not supported. + +## Setup - Downloading the Files + +If you haven't already, download the files and change your working directory: + +```bash +$ cd flux-workflow-examples/async-bulk-job-submit +``` + +## Part (a) - Using `submit_async` + +### Description: Asynchronously submit jobspec files from a directory and wait for them to complete in any order + +1. Allocate three nodes from a resource manager: + +```bash +salloc -N3 -ppdebug +``` + +2. Make a **jobs** directory: + +```bash +mkdir /tmp/jobs +``` + +3. If you are running Slurm, launch a Flux instance on the current allocation by running `flux start` once per node, redirecting log messages to the file `out` in the current directory: + +```bash +srun --pty --mpi=none -N3 flux start -o,-S,log-filename=out +``` + +4. Store the jobspec of a `sleep 0` job in the **jobs** directory: + +```bash +flux run --dry-run -n1 sleep 0 > /tmp/jobs/0.json +``` + +5. Copy the jobspec of **job0** 1024 times to create a directory of 1025 `sleep 0` jobs: + +```bash +for i in `seq 1 1024`; do cp /tmp/jobs/0.json /tmp/jobs/${i}.json; done +``` + +6. Run the **bulksubmit.py** script and pass all jobspec in the **jobs** directory as an argument with a shell glob `jobs/*.json`: + +```bash +./bulksubmit.py /tmp/jobs/*.json +``` +```console +bulksubmit: Starting... +bulksubmit: submitted 1025 jobs in 0.43s. 2392.93job/s +bulksubmit: First job finished in about 0.521s +|██████████████████████████████████████████████████████████| 100.0% (274.3 job/s) +bulksubmit: Ran 1025 jobs in 3.7s. 274.3 job/s +``` + +### Notes to Part (a) + +- `h = flux.Flux()` creates a new Flux handle which can be used to connect to and interact with a Flux instance. + +- `job_submit_async(h, jobspec.read(), waitable=True).then(submit_cb)` submits a jobspec, returning a future which will be fulfilled when the submission of this job is complete. + +`.then(submit_cb)`, called on the returned future, will cause our callback `submit_cb()` to be invoked when the submission of this job is complete and a jobid is available. To process job submission RPC responses and invoke callbacks, the flux reactor for handle `h` must be run: + +```python +if h.reactor_run() < 0: + h.fatal_error("reactor start failed") +``` + +The reactor will return automatically when there are no more outstanding RPC responses, i.e., all jobs have been submitted. + +- `job.wait(h)` waits for any job submitted with the `FLUX_JOB_WAITABLE` flag to transition to the **INACTIVE** state. + + +### Part (b) - Using FluxExecutor + +#### Description: Asynchronously submit a single command repeatedly + +If continuing from part (a), skip to step 3. + +1. Allocate three nodes from a resource manager: + +```bash +salloc -N3 -ppdebug +``` + +2. Launch a Flux instance on the current allocation by running `flux start` once per node, redirecting log messages to the file `out` in the current directory: + +```bash +srun --pty --mpi=none -N3 flux start -o,-S,log-filename=out +``` + +3. Run the **bulksubmit_executor.py** script and pass the command (`/bin/sleep 0` in this example) and the number of times to run it (default is 100): + +```bash +./bulksubmit_executor.py -n200 /bin/sleep 0 +``` +```console +bulksubmit_executor: submitted 200 jobs in 0.18s. 1087.27job/s +bulksubmit_executor: First job finished in about 0.248s +|██████████████████████████████████████████████████████████| 100.0% (229.8 job/s) +bulksubmit_executor: Ran 200 jobs in 1.0s. 199.6 job/s +``` diff --git a/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/async-bulk-job-submit/bulksubmit.py b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/async-bulk-job-submit/bulksubmit.py new file mode 100755 index 00000000..c1a2e9a5 --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/async-bulk-job-submit/bulksubmit.py @@ -0,0 +1,62 @@ +#!/usr/bin/env python3 + +import time +import sys +import flux + +from flux import job +from flux import constants + +t0 = time.time() +jobs = [] +label = "bulksubmit" + +# open connection to broker +h = flux.Flux() + + +def log(s): + print(label + ": " + s) + + +def progress(fraction, length=72, suffix=""): + fill = int(round(length * fraction)) + bar = "\u2588" * fill + "-" * (length - fill) + s = "\r|{0}| {1:.1f}% {2}".format(bar, 100 * fraction, suffix) + sys.stdout.write(s) + if fraction == 1.0: + sys.stdout.write("\n") + + +def submit_cb(f): + jobs.append(job.submit_get_id(f)) + + +# asynchronously submit jobspec files from a directory +log("Starting...") +for file in sys.argv[1:]: + with open(file) as jobspec: + job.submit_async(h, jobspec.read(), waitable=True).then(submit_cb) + +if h.reactor_run() < 0: + h.fatal_error("reactor start failed") + +total = len(jobs) +dt = time.time() - t0 +jps = len(jobs) / dt +log("submitted {0} jobs in {1:.2f}s. {2:.2f}job/s".format(total, dt, jps)) + +count = 0 +while count < total: + # wait for jobs to complete in any order + job.wait(h) + count = count + 1 + if count == 1: + log("First job finished in about {0:.3f}s".format(time.time() - t0)) + suffix = "({0:.1f} job/s)".format(count / (time.time() - t0)) + progress(count / total, length=58, suffix=suffix) + +dt = time.time() - t0 +log("Ran {0} jobs in {1:.1f}s. {2:.1f} job/s".format(total, dt, total / dt)) + +# vi: ts=4 sw=4 expandtab diff --git a/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/async-bulk-job-submit/bulksubmit_executor.py b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/async-bulk-job-submit/bulksubmit_executor.py new file mode 100755 index 00000000..52808633 --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/async-bulk-job-submit/bulksubmit_executor.py @@ -0,0 +1,67 @@ +#!/usr/bin/env python3 + +import time +import sys +import argparse +import concurrent.futures as cf + +from flux.job import FluxExecutor, JobspecV1 + + +def log(label, s): + print(label + ": " + s) + + +def progress(fraction, length=72, suffix=""): + fill = int(round(length * fraction)) + bar = "\u2588" * fill + "-" * (length - fill) + s = f"\r|{bar}| {100 * fraction:.1f}% {suffix}" + sys.stdout.write(s) + if fraction == 1.0: + sys.stdout.write("\n") + + +def main(): + parser = argparse.ArgumentParser( + description="Submit a command repeatedly using FluxExecutor" + ) + parser.add_argument( + "-n", + "--njobs", + type=int, + metavar="N", + help="Set the total number of jobs to run", + default=100, + ) + parser.add_argument("command", nargs=argparse.REMAINDER) + args = parser.parse_args() + if not args.command: + args.command = ["true"] + t0 = time.perf_counter() + label = "bulksubmit_executor" + with FluxExecutor() as executor: + compute_jobspec = JobspecV1.from_command(args.command) + futures = [executor.submit(compute_jobspec) for _ in range(args.njobs)] + # wait for the jobid for each job, as a proxy for the job being submitted + for fut in futures: + fut.jobid() + # all jobs submitted - print timings + dt = time.perf_counter() - t0 + jps = args.njobs / dt + log(label, f"submitted {args.njobs} jobs in {dt:.2f}s. {jps:.2f}job/s") + # wait for jobs to complete + for i, _ in enumerate(cf.as_completed(futures)): + if i == 0: + log( + label, + f"First job finished in about {time.perf_counter() - t0:.3f}s", + ) + jps = (i + 1) / (time.perf_counter() - t0) + progress((i + 1) / args.njobs, length=58, suffix=f"({jps:.1f} job/s)") + # print time summary + dt = time.perf_counter() - t0 + log(label, f"Ran {args.njobs} jobs in {dt:.1f}s. {args.njobs / dt:.1f} job/s") + + +if __name__ == "__main__": + main() diff --git a/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/bulksubmit/0.sh b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/bulksubmit/0.sh new file mode 100755 index 00000000..a11e2318 --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/bulksubmit/0.sh @@ -0,0 +1,3 @@ +#!/bin/bash + +echo "Once upon a time... 📗️" diff --git a/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/bulksubmit/1.sh b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/bulksubmit/1.sh new file mode 100755 index 00000000..f958b58f --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/bulksubmit/1.sh @@ -0,0 +1,3 @@ +#!/bin/bash + +echo "There was a little duck 🦆️" diff --git a/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/bulksubmit/2.sh b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/bulksubmit/2.sh new file mode 100755 index 00000000..fe6930b6 --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/bulksubmit/2.sh @@ -0,0 +1,3 @@ +#!/bin/bash + +echo "Her name was pizzaquack 🍕️" diff --git a/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/bulksubmit/3.sh b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/bulksubmit/3.sh new file mode 100755 index 00000000..7ba6b823 --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/bulksubmit/3.sh @@ -0,0 +1,3 @@ +#!/bin/bash + +echo "She was very fond of cheese 🧀️" diff --git a/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/bulksubmit/4.sh b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/bulksubmit/4.sh new file mode 100755 index 00000000..089c949a --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/bulksubmit/4.sh @@ -0,0 +1,3 @@ +#!/bin/bash + +echo "And running Flux 🌀️" diff --git a/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/bulksubmit/5.sh b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/bulksubmit/5.sh new file mode 100755 index 00000000..3c009208 --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/bulksubmit/5.sh @@ -0,0 +1,3 @@ +#!/bin/bash + +echo "And so she ran Flux, while she ate her cheese 😋️" diff --git a/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/bulksubmit/6.sh b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/bulksubmit/6.sh new file mode 100755 index 00000000..9233634e --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/bulksubmit/6.sh @@ -0,0 +1,3 @@ +#!/bin/bash + +echo "And was so happy! The end. 🌈️" diff --git a/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/comms-module/Makefile b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/comms-module/Makefile new file mode 100644 index 00000000..ccc018db --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/comms-module/Makefile @@ -0,0 +1,19 @@ +all: capp.so ioapp.so + +FLUX_CORE_LIBS = $(shell pkg-config --libs flux-core) +FLUX_CORE_INCLUDES = $(shell pkg-config --cflags flux-core) + +ioapp.so: ioapp.o + gcc -Wl,--no-undefined --disable-static -shared -export-dynamic $^ -o $@ $(FLUX_CORE_LIBS) + +ioapp.o: app.c + gcc $(FLUX_CORE_INCLUDES) $^ -DIO_SERVICE=1 -fPIC -c -o $@ + +capp.so: capp.o + gcc -Wl,--no-undefined --disable-static -shared -export-dynamic $^ -o $@ $(FLUX_CORE_LIBS) + +capp.o: app.c + gcc $(FLUX_CORE_INCLUDES) $^ -DCOMP_SERVICE=1 -fPIC -c -o $@ + +clean: + rm *.o *.so diff --git a/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/comms-module/README.md b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/comms-module/README.md new file mode 100644 index 00000000..6f1456ca --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/comms-module/README.md @@ -0,0 +1,93 @@ +# Using a Flux Comms Module + +## Description: Use a Flux comms module to communicate with job elements + +### Setup + +If you haven't already, download the files and change your working directory: + +```bash +$ cd flux-workflow-examples/comms-module +``` + +### Execution + +If you need to get an allocation on Slurm: + +```bash +salloc -N3 -ppdebug +``` + +Point to `flux-core`'s `pkgconfig` directory: + +| Shell | Command | +| ----- | ---------- | +| tcsh | `setenv PKG_CONFIG_PATH /lib/pkgconfig` | +| bash/zsh | `export PKG_CONFIG_PATH='/lib/pkgconfig'` | + +This might look like this in the container: + +```bash +export PKG_CONFIG_PATH=/usr/lib/pkgconfig +``` + +Then build the module (if you don't have permission, copy to /tmp) + +```bash +cp -R ./comms-module /tmp/comms-module +cd /tmp/comms-module +make +``` + +Add the directory of the modules to `FLUX_MODULE_PATH`; if the module was +built in the current dir: + +```bash +flux module load ioapp.so +flux module load capp.so +export FLUX_MODULE_PATH=${FLUX_MODULE_PATH}:$(pwd) +``` + +Now let's try it! If you need to run flux start under Slurm: + +```bash +srun --pty --mpi=none -N3 flux start -o,-S,log-filename=out +``` + +Try running flux with the module on the path. + +```bash +flux run -N 1 -n 2 ./compute.lua 120 +flux run -N 1 -n 2 ./io-forwarding.lua 120 +``` +Notice that the module is loaded (at the bottom): + +```console +Try `flux-module load --help' for more information. +Module Idle S Sendq Recvq Service +heartbeat 1 R 0 0 +resource 0 R 0 0 +job-ingest 0 R 0 0 +kvs-watch 0 R 0 0 +sched-fluxion-resource 0 R 0 0 +cron idle R 0 0 +barrier idle R 0 0 +job-exec 0 R 0 0 +job-list idle R 0 0 +kvs 0 R 0 0 +content-sqlite 0 R 0 0 content-backing +job-info 0 R 0 0 +job-manager 0 R 0 0 +sched-fluxion-qmanager 0 R 0 0 sched +content 0 R 0 0 +connector-local 0 R 0 0 1002-shell-f3Lv2Zd3tj,1002-shell-f3N2WmZB5H +ioapp 83 R 0 0 +Block until we hear go message from the an io forwarder +``` + +If you run them together, they work together: + +```bash +flux submit -N 1 -n 2 ./compute.lua 120 +flux run -N 1 -n 2 ./io-forwarding.lua 120 +``` diff --git a/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/comms-module/app.c b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/comms-module/app.c new file mode 100644 index 00000000..336ecfbb --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/comms-module/app.c @@ -0,0 +1,129 @@ +#include +#include +#include + +#if !defined (IO_SERVICE) && !defined (COMP_SERVICE) +# error "Either IO_SERVICE or COMP_SERVICE macro is needed" +#endif + +struct app_ctx { + flux_t *h; + int count; + flux_msg_handler_t **handlers; +}; + +static void freectx (void *arg) +{ + struct app_ctx *ctx = (struct app_ctx *)arg; + flux_msg_handler_delvec (ctx->handlers); + free (ctx); +} + +static struct app_ctx *getctx (flux_t *h) +{ +#if IO_SERVICE + struct app_ctx *ctx = flux_aux_get (h, "ioapp"); +#elif COMP_SERVICE + struct app_ctx *ctx = flux_aux_get (h, "capp"); +#endif + if (!ctx) { + ctx = malloc (sizeof (*ctx)); + ctx->count = 0; + ctx->handlers = NULL; +#if IO_SERVICE + flux_aux_set (h, "ioapp", ctx, freectx); +#elif COMP_SERVICE + flux_aux_set (h, "capp", ctx, freectx); +#endif + } + return ctx; +} + +#if IO_SERVICE +static void io_request_cb (flux_t *h, flux_msg_handler_t *w, + const flux_msg_t *msg, void *arg) +{ + const char *topic = NULL; + struct app_ctx *ctx = getctx (h); + int data = 0; + + if (flux_request_unpack (msg, &topic, "{s:i}", "data", &data)) + goto error; + ctx->count++; + if (flux_respond_pack (h, msg, "{s:i}", "count", ctx->count) < 0) + flux_log_error (h, "%s", __FUNCTION__); + flux_log (h, LOG_DEBUG, "count: %d", ctx->count); + return; + +error: + flux_log_error (h, "%s", __FUNCTION__); + if (flux_respond (h, msg, NULL) < 0) + flux_log_error (h, "%s: flux_respond", __FUNCTION__); +} +#endif + +#if COMP_SERVICE +static void comp_request_cb (flux_t *h, flux_msg_handler_t *w, + const flux_msg_t *msg, void *arg) +{ + const char *topic = NULL; + struct app_ctx *ctx = getctx (h); + int data = 0; + + flux_log (h, LOG_INFO, "comp_request_cb:"); + if (flux_request_unpack (msg, &topic, "{s:i}", "data", &data)) + goto error; + + ctx->count++; + if (flux_respond_pack (h, msg, "{s:i}", "count", ctx->count) < 0) + flux_log_error (h, "%s", __FUNCTION__); + return; + +error: + flux_log_error (h, "%s", __FUNCTION__); + if (flux_respond (h, msg, NULL) < 0) + flux_log_error (h, "%s: flux_respond", __FUNCTION__); +} +#endif + +static struct flux_msg_handler_spec htab[] = { +#if IO_SERVICE + { FLUX_MSGTYPE_REQUEST, "ioapp.io", io_request_cb, 0 }, +#endif + +#if COMP_SERVICE + { FLUX_MSGTYPE_REQUEST, "capp.comp", comp_request_cb, 0 }, +#endif + + FLUX_MSGHANDLER_TABLE_END +}; + + +int mod_main (flux_t *h, int argc, char **argv) +{ + + struct app_ctx *ctx = getctx (h); + if (flux_msg_handler_addvec (h, htab, (void *)h, + &ctx->handlers) < 0) { + flux_log (ctx->h, LOG_ERR, "flux_msg_handler_addvec: %s", strerror (errno)); + goto done; + } + + if (flux_reactor_run (flux_get_reactor (h), 0) < 0) { + flux_log (h, LOG_ERR, "flux_reactor_run: %s", strerror (errno)); + goto done; + } + +done: + return 0; +} + +#if IO_SERVICE +MOD_NAME ("ioapp"); +#elif COMP_SERVICE +MOD_NAME ("capp"); +#endif + +/* + * vi:tabstop=4 shiftwidth=4 expandtab + */ diff --git a/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/comms-module/compute.lua b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/comms-module/compute.lua new file mode 100755 index 00000000..f505f54d --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/comms-module/compute.lua @@ -0,0 +1,62 @@ +#!/usr/bin/env lua + +local f, err = require 'flux' .new () +local amount = tonumber (arg[1]) or 120 +local rank = tonumber (os.getenv('FLUX_TASK_RANK')) or 0 +local frank = tonumber (os.getenv('FLUX_LOCAL_RANKS')) or 0 +io.stdout:setvbuf ("no") + +local function sleep (n) + os.execute ("sleep " .. n) +end + +if #arg ~= 1 then + print ("Usage: compute.lua seconds") + print (" Compute for seconds") + os.exit (1) +end + +-- subscribe app.io.go event +local rc, err = f:subscribe ("app.io.go") +if not rc then + print ("Failed to subscribe an event, %s", err) + os.exit (1) +end + +-- the leader rank of compute job installs app module +if rank == 0 then + os.execute ("flux module load -r " .. 0 .. " capp") + os.execute ("flux module list") +end + +-- wait for an event sent from the leader of io-forwarding job to sync +-- between io job's installing the app module and sending a request later +print ("Block until we hear go message from the an io forwarder") +local rc, err = f:recv_event () +if not rc then + print ("Failed to receive an event, %s", err) + os.exit (1) +end + +if rank == 0 then + local rc, err = f:sendevent ({ data = "please proceed" }, "app.comp.go") + if not rc then error (err) end + print ("Sent a go event") +end + +local resp, err = f:rpc ("ioapp.io", { data = rank }) +if not resp then + if err == "Function not implemented" then + print ("ioapp.io request handler isn't loaded") + else + print (err) + end +else + print ("Count so far: " .. resp.count) +end + +print ("Will compute for " .. amount .. " seconds") +sleep (amount) +f:unsubscribe ("app.io.go") + +-- vi: ts=4 sw=4 expandtab diff --git a/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/comms-module/io-forwarding.lua b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/comms-module/io-forwarding.lua new file mode 100755 index 00000000..0f9f78f9 --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/comms-module/io-forwarding.lua @@ -0,0 +1,57 @@ +#!/usr/bin/env lua + +local flux = require 'flux' +local f = flux.new () +local amount = tonumber (arg[1]) or 120 +local rank = tonumber (os.getenv('FLUX_TASK_RANK')) or 0 +local frank = tonumber (os.getenv('FLUX_LOCAL_RANKS')) or 0 +io.stdout:setvbuf ("no") + +local function sleep (n) + os.execute ("sleep " .. n) +end + +if #arg ~= 1 then + print ("Usage: io-forward.lua seconds") + print (" Forward I/O requests for seconds") + os.exit (1) +end + +-- subscribe app.comp.go event +local rc, err = f:subscribe ("app.comp.go") +if not rc then + print ("Failed to subscribe an event, %s", err) + os.exit (1) +end + +if rank == 0 then + os.execute ("flux module load -r " .. 0 .. " ioapp") + os.execute ("flux module list") + local rc, err = f:sendevent ({ data = "please proceed" }, "app.io.go") + if not rc then error (err) end + print ("Sent a go event") +end + +-- Wait for an event sent from the leader of compute job to sync +-- between compute job's installing the app module and sending a request later +print ("Block until we hear go message from the a leader compute process") +local rc, err = f:recv_event () +if not rc then + print ("Failed to receive an, %s", err) + os.exit (1) +end + +local resp, err = f:rpc ("capp.comp", { data = rank }) +if not resp then + if err == "Function not implemented" then + print ("capp.comp request handler isn't loaded") + else + print (err) + end +end + +print ("Will forward IO requests for " .. amount .. " seconds") +sleep (amount) +f:unsubscribe ("app.comp.go") + +-- vi: ts=4 sw=4 expandtab diff --git a/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/conf.py b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/conf.py new file mode 100644 index 00000000..75f3e5ce --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/conf.py @@ -0,0 +1,83 @@ +############################################################### +# Copyright 2020 Lawrence Livermore National Security, LLC +# (c.f. AUTHORS, NOTICE.LLNS, COPYING) +# +# This file is part of the Flux resource manager framework. +# For details, see https://github.com/flux-framework. +# +# SPDX-License-Identifier: LGPL-3.0 +############################################################### + +# Configuration file for the Sphinx documentation builder. +# +# This file only contains a selection of the most common options. For a full +# list see the documentation: +# https://www.sphinx-doc.org/en/master/usage/configuration.html + +# -- Path setup -------------------------------------------------------------- + +# If extensions (or modules to document with autodoc) are in another directory, +# add these directories to sys.path here. If the directory is relative to the +# documentation root, use os.path.abspath to make it absolute, like shown here. +# +# import os +# import sys +# sys.path.insert(0, os.path.abspath('.')) + + +# -- Project information ----------------------------------------------------- + +project = 'Flux' +copyright = '''Copyright 2020 Lawrence Livermore National Security, LLC and Flux developers. + +SPDX-License-Identifier: LGPL-3.0''' +author = 'This page is maintained by the Flux community.' + +# The full version, including alpha/beta/rc tags +release = '0.1.0' + + +# -- General configuration --------------------------------------------------- + +# Add any Sphinx extension module names here, as strings. They can be +# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom +# ones. +extensions = [ + 'sphinx.ext.intersphinx', + 'sphinxcontrib.spelling', + 'recommonmark', +] + +# sphinxcontrib.spelling settings +spelling_word_list_filename = [ + 'spell.en.pws' +] + +# Add any paths that contain templates here, relative to this directory. +templates_path = ['_templates'] + +# List of patterns, relative to source directory, that match files and +# directories to ignore when looking for source files. +# This pattern also affects html_static_path and html_extra_path. +exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store', 'README.md'] + +master_doc = 'index' +source_suffix = ['.rst', '.md'] + +# -- Options for HTML output ------------------------------------------------- + +# The theme to use for HTML and HTML Help pages. See the documentation for +# a list of builtin themes. +# +html_theme = 'sphinx_rtd_theme' + +# Add any paths that contain custom static files (such as style sheets) here, +# relative to this directory. They are copied after the builtin static files, +# so a file named "default.css" will overwrite the builtin "default.css". +html_static_path = [ +] + +# -- Options for man output ------------------------------------------------- + +man_pages = [ +] diff --git a/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/data-conduit/Makefile b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/data-conduit/Makefile new file mode 100644 index 00000000..56abc33b --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/data-conduit/Makefile @@ -0,0 +1,13 @@ +all: conduit.so + +FLUX_CORE_LIBS = $(shell pkg-config --libs flux-core) +FLUX_CORE_INCLUDES = $(shell pkg-config --cflags flux-core) + +conduit.so: conduit.o + gcc -Wl,--no-undefined --disable-static -shared -export-dynamic $^ -o $@ $(FLUX_CORE_LIBS) + +conduit.o: conduit.c + gcc $(FLUX_CORE_INCLUDES) $^ -fPIC -c -o $@ + +clean: + rm *.o *.so diff --git a/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/data-conduit/README.md b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/data-conduit/README.md new file mode 100644 index 00000000..3a9a927c --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/data-conduit/README.md @@ -0,0 +1,102 @@ +# A Data Conduit Strategy + +**Note that this module script does not compile and needs an update** + + +## Description: Use a data stream to send packets through + +### Setup + +If you haven't already, download the files and change your working directory: + +```bash +$ cd flux-workflow-examples/data-conduit +``` + +### Execution + +If you are using Slurm, allocate three nodes from a resource manager: + +```bash +salloc -N3 -ppdebug +``` + +Point to `flux-core`'s `pkgconfig` directory: + +| Shell | Command | +| ----- | ---------- | +| tcsh | `setenv PKG_CONFIG_PATH /lib/pkgconfig` | +| bash/zsh | `export PKG_CONFIG_PATH='/lib/pkgconfig'` | + +This might look like this in the container: + +```bash +export PKG_CONFIG_PATH=/usr/lib/pkgconfig +``` + +Then build the module (if you don't have permission, copy to /tmp) + +```bash +cp -R ./data-conduit /tmp/data-conduit +cd /tmp/data-conduit +make +``` + +3. `make` + +4. Add the directory of the modules to `FLUX_MODULE_PATH`, if the module was built in the current directory: + +`export FLUX_MODULE_PATH=${FLUX_MODULE_PATH}:$(pwd)` + +5. Launch a Flux instance on the current allocation by running `flux start` once per node, redirecting log messages to the file `out` in the current directory: + +`srun --pty --mpi=none -N3 flux start -o,-S,log-filename=out` + +6. Submit the **datastore** script: + +`flux submit -N 1 -n 1 ./datastore.py` + +7. Submit and resubmit five **compute** scripts to send time data to **datastore**: + +`flux submit -N 1 -n 1 ./compute.py 1` + +`flux submit -N 1 -n 1 ./compute.py 1` + +`flux submit -N 1 -n 1 ./compute.py 1` + +`flux submit -N 1 -n 1 ./compute.py 1` + +`flux submit -N 1 -n 1 ./compute.py 1` + +8. Attach to the **datastore** job to see the data sent by the **compute.py** scripts: + +`flux job attach 1900070043648` + +``` +Starting.... +Module was loaded successfully... +finished initialize... +starting run() +Waiting for a packet +{u'test': 101} +Waiting for a packet +{u'test': 101, u'1578431137': u'os.time'} +Waiting for a packet +{u'test': 101, u'1578431137': u'os.time', u'1578431139': u'os.time'} +Waiting for a packet +{u'test': 101, u'1578431140': u'os.time', u'1578431137': u'os.time', u'1578431139': u'os.time'} +Waiting for a packet +{u'test': 101, u'1578431140': u'os.time', u'1578431137': u'os.time', u'1578431139': u'os.time', u'1578431141': u'os.time'} +Bye bye! +run finished... +``` + +--- + +### Notes + +- `f = flux.Flux()` creates a new Flux handle which can be used to connect to and interact with a Flux instance. + +- `kvs.put()` places the value of _udata_ under the key **"conduit"**. Once the key-value pair is put, the change must be committed with `kvs.commit()`. The value can then be retrieved with `kvs.get()`. + +- `f.rpc()` creates a new RPC object consisting of a specified topic and payload (along with additional flags) that are exchanged with a Flux service. diff --git a/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/data-conduit/compute.py b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/data-conduit/compute.py new file mode 100644 index 00000000..d03f8716 --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/data-conduit/compute.py @@ -0,0 +1,39 @@ +#!/usr/bin/env python3 + +import argparse +import time +import os +import re +import flux +import json +from flux import kvs +from flux.message import Message + +parser = argparse.ArgumentParser(description="compute for seconds") +parser.add_argument( + "integer", + metavar="S", + type=int, + help="an integer for the number of seconds to compute", +) + +args = parser.parse_args() + +f = flux.Flux() +udata = "conduit" +kvs.put(f, "conduit", udata) +kvs.commit(f) + +cr = kvs.get(f, "conduit") +print(cr) + +os_time = int(time.time()) +payload = {str(os_time): "os.time"} +new_payload = {"data": json.dumps(payload)} +print("Sending ", json.dumps(new_payload)) + +# this data is ultimately flowed into the data store +f.rpc("conduit.put", new_payload, 0) + + +time.sleep(args.integer) diff --git a/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/data-conduit/conduit.c b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/data-conduit/conduit.c new file mode 100644 index 00000000..9e6f446e --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/data-conduit/conduit.c @@ -0,0 +1,182 @@ +#include +#include +#include +#include +#include +#include +#include +#include +#include + +struct conduit_ctx { + flux_t *h; + struct sockaddr_un server_sockaddr; + struct sockaddr_un client_sockaddr; + int client_sock; + bool connected; + char *sockname; + char *csockname; + flux_msg_handler_t **handlers; +}; + +static void freectx (void *arg) +{ + struct conduit_ctx *ctx = (struct conduit_ctx *)arg; + flux_msg_handler_delvec (ctx->handlers); + free (ctx->sockname); + free (ctx->csockname); + if (ctx->connected) + close (ctx->client_sock); + free (ctx); +} + +static struct conduit_ctx *getctx (flux_t *h) +{ + struct conduit_ctx *ctx = flux_aux_get (h, "conduit"); + if (!ctx) { + char *user = getenv ("USER"); + ctx = malloc (sizeof (*ctx)); + ctx->connected = false; + ctx->handlers = NULL; + asprintf (&(ctx->sockname), "/tmp/%s/mysock", user? user : ""); + asprintf (&(ctx->csockname),"/tmp/%s/mycsock", user? user : ""); + flux_aux_set (h, "conduit", ctx, freectx); + } + return ctx; +} + +/* Forward the received JSON string to the datastore.py */ +static int conduit_send (flux_t *h, const char *json_str) +{ + int rc = -1; + int n = 0; + struct conduit_ctx *ctx = getctx (h); + + n = (int) strlen (json_str); + if ((rc = send (ctx->client_sock, (void *)&n, sizeof (n), 0)) == -1) { + flux_log_error (h, "send error %s", __FUNCTION__); + return rc; + } + if ((rc = send (ctx->client_sock, (void *)json_str, n, 0)) == -1) { + flux_log_error (h, "send error %s", __FUNCTION__); + return rc; + } + flux_log (h, LOG_INFO, "conduit_send succeed"); + return 0; +} + +/* request callback called when conduit.put request is invoked */ +static void conduit_put_request_cb (flux_t *h, flux_msg_handler_t *w, + const flux_msg_t *msg, void *arg) +{ + int rc = -1; + const char *topic = NULL; + struct conduit_ctx *ctx = getctx (h); + const char *data = NULL; + + flux_log (h, LOG_INFO, "conduit_put_request_cb:"); + if (ctx->connected == false) { + flux_log (h, LOG_INFO, "conduit not connected"); + errno = ENOTCONN; + goto done; + } + if (flux_request_unpack (msg, &topic, "{s:s}", "data", &data)) { + flux_log_error (h, "%s", __FUNCTION__); + goto done; + } + if (conduit_send (h, data) < 0) + errno = EPROTO; +done: + if (flux_respond (h, msg, errno, NULL) < 0) + flux_log_error (h, "%s: flux_respond", __FUNCTION__); +} + +/* open the Unix domain socket to talk to datastore.py */ +static int conduit_open (flux_t *h) +{ + struct conduit_ctx *ctx = getctx (h); + int rc = -1; + int len = 0; + char buf[256]; + memset(&(ctx->server_sockaddr), 0, sizeof(struct sockaddr_un)); + memset(&(ctx->client_sockaddr), 0, sizeof(struct sockaddr_un)); + + if ((ctx->client_sock = socket(AF_UNIX, SOCK_STREAM, 0)) == -1) { + flux_log (h, LOG_ERR, "SOCKET ERROR = %d\n", errno); + goto done; + } + + ctx->client_sockaddr.sun_family = AF_UNIX; + strcpy(ctx->client_sockaddr.sun_path, ctx->csockname); + len = sizeof(ctx->client_sockaddr); + unlink (ctx->csockname); + if ((rc = bind(ctx->client_sock, + (struct sockaddr *)&ctx->client_sockaddr, len)) == -1) { + flux_log (h, LOG_ERR, "BIND ERROR: %d\n", errno); + close(ctx->client_sock); + goto done; + } + flux_log (h, LOG_INFO, "Conduit client socket bound\n"); + + ctx->server_sockaddr.sun_family = AF_UNIX; + strcpy(ctx->server_sockaddr.sun_path, ctx->sockname); + if ((rc = connect(ctx->client_sock, + (struct sockaddr *)&ctx->server_sockaddr, len)) == -1) { + flux_log (h, LOG_ERR, "CONNECT ERROR = %d\n", errno); + close(ctx->client_sock); + goto done; + } + + ctx->connected = true; + flux_log (h, LOG_INFO, "Conduit socket connected\n"); + conduit_send (h, "{\"test\":101}"); + rc = 0; +done: + return rc; +} + + +static struct flux_msg_handler_spec htab[] = { + { FLUX_MSGTYPE_REQUEST, "conduit.put", conduit_put_request_cb, 0 }, + FLUX_MSGHANDLER_TABLE_END +}; + +int mod_main (flux_t *h, int argc, char **argv) +{ + uint32_t rank = 0; + struct conduit_ctx *ctx = getctx (h); + + if (conduit_open (h) < 0) { + flux_log (ctx->h, LOG_ERR, "conduit_open failed"); + goto done; + } + if (flux_get_rank (h, &rank) < 0) { + flux_log (ctx->h, LOG_ERR, "flux_get_rank failed"); + goto done; + } + + /* Put the rank where this module is loaded into conduit key + */ + flux_kvs_txn_t *txn = flux_kvs_txn_create (); + flux_kvs_txn_pack (txn, 0, "conduit", "i", rank); + flux_kvs_commit (h, 0, txn); + flux_kvs_txn_destroy (txn); + if (flux_msg_handler_addvec (h, htab, (void *)h, + &ctx->handlers) < 0) { + flux_log (ctx->h, LOG_ERR, "flux_msg_handler_addvec: %s", strerror (errno)); + goto done; + } + if (flux_reactor_run (flux_get_reactor (h), 0) < 0) { + flux_log (h, LOG_ERR, "flux_reactor_run: %s", strerror (errno)); + goto done; + } + +done: + return 0; +} + +MOD_NAME ("conduit"); + +/* + * vi:tabstop=4 shiftwidth=4 expandtab + */ diff --git a/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/data-conduit/datastore.py b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/data-conduit/datastore.py new file mode 100755 index 00000000..d5fcc488 --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/data-conduit/datastore.py @@ -0,0 +1,68 @@ +#!/usr/bin/env python3 + +import socket +import struct +import json +import sys +import os + +sockdir = os.path.join("/tmp", os.environ["USER"]) +sockname = os.path.join(sockdir, "mysock") + +store = {} +sock = "" + + +def initialize(): + global sock + if not os.path.exists(sockdir): + os.mkdir(sockdir) + if os.path.exists(sockname): + os.remove(sockname) + sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) + sock.bind(sockname) + sock.listen(1) + cmd = "flux module load ./conduit.so" + os.system(cmd) + + +def run(): + global sock + global store + connection, client_address = sock.accept() + for x in range(5): + print("Waiting for a packet") + mybytes = bytearray(4) + nbytes, address = connection.recvfrom_into(mybytes, 4) + if nbytes == 0: + break + size = ( + mybytes[0] * 1 + + mybytes[1] * 256 + + mybytes[2] * 65536 + + mybytes[3] * 16777216 + ) + data = bytearray(size) + nbytes, address = connection.recvfrom_into(data, size) + dict_blob = json.loads(data.decode("ascii")) + + if dict_blob is not None: + store.update(dict_blob) + print(store) + else: + print("Mallformed data, discarding") + + connection.close() + cmd = "flux module remove conduit" + os.system(cmd) + print("Bye bye!") + + +def main(): + print("Starting....") + initialize() + run() + + +if __name__ == "__main__": + main() diff --git a/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/hierarchical-launching/README.md b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/hierarchical-launching/README.md new file mode 100644 index 00000000..ff33747a --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/hierarchical-launching/README.md @@ -0,0 +1,39 @@ +# Hierarchical Launching + +## Description: Launch an ensemble of sleep 0 tasks + +### Setup + +If you haven't already, download the files and change your working directory: + +```bash +$ cd flux-workflow-examples/hierarchical-launching +``` + +### Execution + +If you need to start flux on a Slurm cluster: + +```bash +salloc -N3 -ppdebug +srun --pty --mpi=none -N3 flux start -o,-S,log-filename=out +``` + +Start the parent instance + +```bash +./parent.sh +``` +```console +Mon Nov 18 15:31:08 PST 2019 +13363018989568 +13365166473216 +13367095853056 +First Level Done +Mon Nov 18 15:34:13 PST 2019 +``` + +### Notes + +- You can increase the number of jobs by increasing `NCORES` in `parent.sh` and +`NJOBS` in `ensemble.sh`. diff --git a/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/hierarchical-launching/ensemble.sh b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/hierarchical-launching/ensemble.sh new file mode 100755 index 00000000..efd987b2 --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/hierarchical-launching/ensemble.sh @@ -0,0 +1,12 @@ +#!/usr/bin/env sh + +NJOBS=750 +MAXTIME=$(expr ${NJOBS} + 2) + +for i in `seq 1 ${NJOBS}`; do + flux submit --nodes=1 --ntasks=1 --cores-per-task=1 sleep 0 +done + +flux jobs +flux queue drain +echo "Final Level Done" diff --git a/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/hierarchical-launching/parent.sh b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/hierarchical-launching/parent.sh new file mode 100755 index 00000000..19d74e3f --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/hierarchical-launching/parent.sh @@ -0,0 +1,14 @@ +#!/usr/bin/env sh + +NCORES=3 + +date + +for i in `seq 1 ${NCORES}`; do + flux submit -N 1 -n 1 flux start ./ensemble.sh +done + +flux queue drain +echo "First Level Done" + +date diff --git a/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/index.rst b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/index.rst new file mode 100644 index 00000000..aa335e9b --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/index.rst @@ -0,0 +1,95 @@ +Flux Workflow Examples +---------------------- + +The examples contained here demonstrate and explain some simple use-cases with Flux, +and make use of Flux's command-line interface (CLI), Flux's C library, and the Python and Lua bindings to the C library. +The entire set of examples can be downloaded by cloning the `Github repo `_. + +The examples assume that you have installed: + +#. A recent version of Flux + +#. Python 3.6+ + +#. Lua 5.1+ + +:doc:`CLI: Job Submission ` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Launch a flux instance and schedule/launch compute and io-forwarding +jobs on separate nodes using the CLI + +:doc:`Python: Job Submission ` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Schedule/launch compute and io-forwarding jobs on separate nodes using +the Python bindings + +:doc:`Python: Job Submit/Wait ` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Submit jobs and wait for them to complete using the Flux Python bindings + +:doc:`Python: Asynchronous Bulk Job Submission ` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Asynchronously submit jobspec files from a directory and wait for them +to complete in any order + +:doc:`Python: Tracking Job Status and Events ` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Submit job bundles and wait until all jobs complete + +:doc:`Python: Job Cancellation ` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Cancel a running job + +:doc:`Lua: Use Events ` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Use events to synchronize compute and io-forwarding jobs running on +separate nodes + +:doc:`Python: Simple KVS Example ` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Use KVS Python interfaces to store user data into KVS + +:doc:`CLI/Lua: Job Ensemble Submitted with a New Flux Instance ` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Submit job bundles, print live job events, and exit when all jobs are +complete + +:doc:`CLI: Hierarchical Launching ` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Launch a large number of sleep 0 jobs + +:doc:`C/Lua: Use a Flux Comms Module ` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Use a Flux Comms Module to communicate with job elements + +:doc:`C/Python: A Data Conduit Strategy ` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Attach to a job that receives OS time data from compute jobs + +.. toctree:: + :hidden: + + job-submit-cli/README + job-submit-api/README + job-submit-wait/README + async-bulk-job-submit/README + job-status-control/README + job-cancel/README + synchronize-events/README + kvs-python-bindings/README + job-ensemble/README + hierarchical-launching/README + comms-module/README + data-conduit/README diff --git a/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-cancel/README.md b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-cancel/README.md new file mode 100644 index 00000000..2c9c0cf7 --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-cancel/README.md @@ -0,0 +1,42 @@ +# Job Cancellation + +## Description: Cancel a running job + +### Setup + +If you haven't already, download the files and change your working directory: + +```bash +$ cd flux-workflow-examples/job-cancel +``` + +### Execution + +Launch the submitter script: + +```bash +python3 ./submitter.py $(flux resource list -no {ncores} --state=up) +``` + +```console +Submitted 1st job: 2241905819648 +Submitted 2nd job: 2258951471104 + +First submitted job status (2241905819648) - RUNNING +Second submitted job status (2258951471104) - PENDING + +Canceled first job: 2241905819648 + +First submitted job status (2241905819648) - CANCELED +Second submitted job status (2258951471104) - RUNNING +``` + +### Notes + +- `f = flux.Flux()` creates a new Flux handle which can be used to connect to and interact with a Flux instance. + +- `flux.job.submit(f, sleep_jobspec, waitable=True)` submits a jobspec, returning a job ID that can be used to interact with the submitted job. + +- `flux.job.cancel(f, jobid)` cancels the job. + +- `flux.job.wait_async(f, jobid)` will wait for the job to complete (or in this case, be canceled). It returns a Flux future, which can be used to process the result later. Only jobs submitted with `waitable=True` can be waited for. diff --git a/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-cancel/submitter.py b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-cancel/submitter.py new file mode 100644 index 00000000..b95584fd --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-cancel/submitter.py @@ -0,0 +1,51 @@ +#!/usr/bin/env python3 + +import time +import argparse + +import flux +from flux.job import JobspecV1 + +f = flux.Flux() + +parser = argparse.ArgumentParser( + description=""" + Description: Submit two 'sleep 60' jobs that take up + all resources on a node. + """ +) +parser.add_argument(dest="cores", help="number of cores on the node") +args = parser.parse_args() + +# submit a sleep job that takes up all resources +sleep_jobspec = JobspecV1.from_command( + ["sleep", "60"], num_tasks=1, cores_per_task=int(args.cores) +) +first_jobid = flux.job.submit(f, sleep_jobspec, waitable=True) +print("Submitted 1st job: %d" % (int(first_jobid))) +time.sleep(1) + +# submit a second sleep job - will be scheduled, but not run +sleep_jobspec = JobspecV1.from_command( + ["sleep", "60"], num_tasks=1, cores_per_task=int(args.cores) +) +second_jobid = flux.job.submit(f, sleep_jobspec, waitable=True) +print("Submitted 2nd job: %d\n" % (int(second_jobid))) +time.sleep(1) + +# get list of JobInfo objects - fetch their ID's and current status +jobs = flux.job.JobList(f, max_entries=2).jobs() +print("First submitted job status (%d) - %s" % (int(jobs[1].id.dec), jobs[1].status)) +print("Second submitted job status (%d) - %s\n" % (int(jobs[0].id.dec), jobs[0].status)) + +# cancel the first job +flux.job.cancel(f, first_jobid) +future = flux.job.wait_async(f, first_jobid).wait_for(5.0) +return_id, success, errmsg = future.get_status() +print("Canceled first job: %d\n" % (int(return_id))) +time.sleep(1) + +# the second job should now run since the first was canceled +jobs = flux.job.JobList(f, max_entries=2).jobs() +print("First submitted job status (%d) - %s" % (int(jobs[1].id.dec), jobs[1].status)) +print("Second submitted job status (%d) - %s" % (int(jobs[0].id.dec), jobs[0].status)) diff --git a/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-ensemble/README.md b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-ensemble/README.md new file mode 100644 index 00000000..99361f5b --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-ensemble/README.md @@ -0,0 +1,94 @@ +# Job Ensemble Submitted with a New Flux Instance + +## Description: Launch a flux instance and submit one instance of an io-forwarding job and 50 compute jobs, each spanning the entire set of nodes. + +### Setup + +If you haven't already, download the files and change your working directory: + +``` +$ git clone https://github.com/flux-framework/flux-workflow-examples.git +$ cd flux-workflow-examples/job-ensemble +``` + +### Execution + +If you need a Slurm allocation: + +```bash +salloc -N3 -ppdebug + +# Take a look at the script first +cat ensemble.sh +``` +Here is how to run under Slurm: + +```bash +srun --pty --mpi=none -N3 flux start -o,-S,log-filename=out ./ensemble.sh +``` + +Or without: + +```bash +flux start -o,-S,log-filename=out ./ensemble.sh +``` + +``` +JOBID USER NAME STATE NTASKS NNODES RUNTIME +1721426247680 fluxuser compute.lu RUN 4 2 0.122s +1718322462720 fluxuser compute.lu RUN 4 2 0.293s +1715201900544 fluxuser compute.lu RUN 4 2 0.481s +1712299442176 fluxuser compute.lu RUN 4 2 0.626s +1709296320512 fluxuser compute.lu RUN 4 2 0.885s +1706293198848 fluxuser compute.lu RUN 4 2 1.064s +1691378253824 fluxuser io-forward RUN 1 1 1.951s +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Jobid: 1691378253824 +{ + "version": 1, + "execution": { + "R_lite": [ + { + "rank": "0", + "children": { + "core": "0-1" + } + } + ] + } +} +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Jobid: 1694414929920 +{ + "version": 1, + "execution": { + "R_lite": [ + { + "rank": "1-2", + "children": { + "core": "0-3" + } + } + ] + } +} +. +. +. +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Jobid: 1721426247680 +{ + "version": 1, + "execution": { + "R_lite": [ + { + "rank": "1-2", + "children": { + "core": "8-11" + } + } + ] + } +} + +``` diff --git a/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-ensemble/compute.lua b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-ensemble/compute.lua new file mode 100755 index 00000000..e5159fd1 --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-ensemble/compute.lua @@ -0,0 +1,18 @@ +#!/usr/bin/env lua + +local amount = tonumber (arg[1]) or 120 + +local function sleep (n) + os.execute ("sleep " .. n) +end + +if #arg ~= 1 then + print ("Usage: compute.lua seconds") + print (" Compute for seconds") + os.exit (1) +end + +print ("Will compute for " .. amount .. " seconds") +sleep (amount) + +-- vi: ts=4 sw=4 expandtab diff --git a/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-ensemble/ensemble.sh b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-ensemble/ensemble.sh new file mode 100755 index 00000000..9468ec6a --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-ensemble/ensemble.sh @@ -0,0 +1,21 @@ +#!/usr/bin/env sh + +NJOBS=10 +MAXTIME=$(expr ${NJOBS} + 2) +JOBIDS="" + +JOBIDS=$(flux submit --nodes=1 --ntasks=1 --cores-per-task=2 ./io-forwarding.lua ${MAXTIME}) +for i in `seq 1 ${NJOBS}`; do + JOBIDS="${JOBIDS} $(flux submit --nodes=1 --ntasks=1 --cores-per-task=2 ./compute.lua 1)" +done + +flux jobs +flux queue drain + +# print mock-up prevenance data +for i in ${JOBIDS}; do + echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~" + echo "Jobid: ${i}" + KVSJOBID=$(flux job id --to=kvs ${i}) + flux kvs get ${KVSJOBID}.R | jq +done diff --git a/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-ensemble/io-forwarding.lua b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-ensemble/io-forwarding.lua new file mode 100755 index 00000000..3427b1ed --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-ensemble/io-forwarding.lua @@ -0,0 +1,18 @@ +#!/usr/bin/env lua + +local amount = tonumber (arg[1]) or 120 + +local function sleep (n) + os.execute ("sleep " .. n) +end + +if #arg ~= 1 then + print ("Usage: io-forward.lua seconds") + print (" Forward I/O requests for seconds") + os.exit (1) +end + +print ("Will forward IO requests for " .. amount .. " seconds") +sleep (amount) + +-- vi: ts=4 sw=4 expandtab diff --git a/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-ensemble/kvs-watch-until.lua b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-ensemble/kvs-watch-until.lua new file mode 100755 index 00000000..16e63aed --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-ensemble/kvs-watch-until.lua @@ -0,0 +1,82 @@ +#!/usr/bin/env lua +-- +-- Exit only if/when all ranks have exited 'unknown' state +-- +local usage = [[ +Usage: kvs-wait-until [OPTIONS] KEY CODE +Watch kvs KEY until Lua code CODE returns true. +(CODE is supplied key value in variable 'v') +If -t, --timeout is provided, and the timeout expires, then +exit with non-zero exit status. + -h, --help Display this message + -v, --verbose Print value on each watch callback + -t, --timeout=T Wait at most T seconds (before exiting +]] + +local getopt = require 'flux.alt_getopt' .get_opts +local timer = require 'flux.timer'.new() +local f = require 'flux' .new() + +local function printf (...) + io.stdout:write (string.format (...)) +end +local function log_err (...) + io.stdout:write (string.format (...)) +end + +local opts, optind = getopt (arg, "hvt:", + { verbose = 'v', + timeout = 't', + help = 'h' + } + ) +if opts.h then print (usage); os.exit (0) end + +local key = arg [optind] +local callback = arg [optind+1] + +if not key or not callback then + log_err ("KVS key and callback code required\n") + print (usage) + os.exit (1) +end + +callback = "return function (v) return "..callback.." end" +local fn, err = loadstring (callback, "callback") +if not fn then + log_err ("code compile error: %s", err) + os.exit (1) +end +local cb = fn () + +local kw, err = f:kvswatcher { + key = key, + handler = function (kw, result) + if opts.v then + printf ("%4.03fs: %s = %s\n", + timer:get0(), + key, tostring (result)) + end + -- Do not pass nil result to callback: + if result == nil then return end + local ok, rv = pcall (cb, result) + if not ok then error (rv) end + if ok and rv then + os.exit (0) + end + end +} + +if opts.t then + local tw, err = f:timer { + timeout = opts.t * 1000, + handler = function (f, to) + log_err ("%4.03fs: Timeout expired!\n", timer:get0()) + os.exit (1) + end + } +end + +timer:set () +f:reactor () +-- vi: ts=4 sw=4 expandtab diff --git a/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-status-control/README.md b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-status-control/README.md new file mode 100644 index 00000000..02ad14c1 --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-status-control/README.md @@ -0,0 +1,71 @@ +# Using Flux Job Status and Control API + +## Description: Submit job bundles, get event updates, and wait until all jobs complete + +### Setup + +If you haven't already, download the files and change your working directory: + +```bash +$ cd flux-workflow-examples/job-status-control +``` + +### Execution + +1. Allocate three nodes from a resource manager: + +```bash +salloc -N3 -p pdebug +``` + +2. If needed, launch a Flux instance on the current allocation by running `flux start` once per node, redirecting log messages to the file `out` in the current directory: + +```bash +srun --pty --mpi=none -N3 flux start -o,-S,log-filename=out +``` + +3. Run the bookkeeper executable along with the number of jobs to be submitted (if no size is specified, 6 jobs are submitted: 3 instances of **compute.py**, and 3 instances of **io-forwarding,py**): + +```bash +python3 ./bookkeeper.py 2 +``` +```console +bookkeeper: all jobs submitted +bookkeeper: waiting until all jobs complete +job 39040581632 triggered event 'submit' +job 39040581633 triggered event 'submit' +job 39040581632 triggered event 'depend' +job 39040581632 triggered event 'priority' +job 39040581632 triggered event 'alloc' +job 39040581633 triggered event 'depend' +job 39040581633 triggered event 'priority' +job 39040581633 triggered event 'alloc' +job 39040581632 triggered event 'start' +job 39040581633 triggered event 'start' +job 39040581632 triggered event 'finish' +job 39040581633 triggered event 'finish' +job 39040581633 triggered event 'release' +job 39040581633 triggered event 'free' +job 39040581633 triggered event 'clean' +job 39040581632 triggered event 'release' +job 39040581632 triggered event 'free' +job 39040581632 triggered event 'clean' +bookkeeper: all jobs completed +``` + +### Notes + +- The following constructs a job request using the **JobspecV1** class with customizable parameters for how you want to utilize the resources allocated for your job: +```python +compute_jobreq = JobspecV1.from_command( + command=["./compute.py", "10"], num_tasks=4, num_nodes=2, cores_per_task=2 +) +compute_jobreq.cwd = os.getcwd() +compute_jobreq.environment = dict(os.environ) +``` + +- `with FluxExecutor() as executor:` creates a new `FluxExecutor` which can be used to submit jobs, wait for them to complete, and get event updates. Using the executor as a context manager (`with ... as ...:`) ensures it is shut down properly. + +- `executor.submit(compute_jobreq)` returns a `concurrent.futures.Future` subclass which completes when the underlying job is done. The jobid of the underlying job can be fetched with the `.jobid([timeout])` method (which waits until the jobid is ready). + +- Throughout the course of a job, various events will occur to it. `future.add_event_callback(event, event_callback)` adds a callback which will be invoked when the given event occurs. diff --git a/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-status-control/bookkeeper.py b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-status-control/bookkeeper.py new file mode 100755 index 00000000..a7cef195 --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-status-control/bookkeeper.py @@ -0,0 +1,54 @@ +#!/usr/bin/env python3 + +import os +import argparse + +from flux.job import JobspecV1, FluxExecutor + + +def event_callback(future, event): + print(f"job {future.jobid()} triggered event {event.name!r}") + + +# main +def main(): + # set up command-line parser + parser = argparse.ArgumentParser( + description="submit and wait for the completion of " + "N bundles, each consisting of compute " + "and io-forwarding jobs" + ) + parser.add_argument( + "njobs", metavar="N", type=int, help="the number of bundles to submit and wait", + ) + args = parser.parse_args() + # set up jobspecs + compute_jobreq = JobspecV1.from_command( + command=["./compute.py", "10"], num_tasks=6, num_nodes=3, cores_per_task=2 + ) + compute_jobreq.cwd = os.getcwd() + compute_jobreq.environment = dict(os.environ) + io_jobreq = JobspecV1.from_command( + command=["./io-forwarding.py", "10"], num_tasks=3, num_nodes=3, cores_per_task=1 + ) + io_jobreq.cwd = os.getcwd() + io_jobreq.environment = dict(os.environ) + # submit jobs and register event callbacks for all events + with FluxExecutor() as executor: + futures = [executor.submit(compute_jobreq) for _ in range(args.njobs // 2)] + futures.extend( + executor.submit(io_jobreq) for _ in range(args.njobs // 2, args.njobs) + ) + print("bookkeeper: all jobs submitted") + for fut in futures: + # each event can have a different callback + for event in executor.EVENTS: + fut.add_event_callback(event, event_callback) + print("bookkeeper: waiting until all jobs complete") + # exiting the context manager waits for the executor to complete all futures + print("bookkeeper: all jobs completed") + + +main() + +# vi: ts=4 sw=4 expandtab diff --git a/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-status-control/compute.py b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-status-control/compute.py new file mode 100755 index 00000000..1f860f2a --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-status-control/compute.py @@ -0,0 +1,17 @@ +#!/usr/bin/env python3 + +import argparse +import time + +parser = argparse.ArgumentParser(description="compute for seconds") +parser.add_argument( + "integer", + metavar="S", + type=int, + help="an integer for the number of seconds to compute", +) + +args = parser.parse_args() + +print("Will compute for " + str(args.integer) + " seconds.") +time.sleep(args.integer) diff --git a/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-status-control/io-forwarding.py b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-status-control/io-forwarding.py new file mode 100755 index 00000000..217ed0e0 --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-status-control/io-forwarding.py @@ -0,0 +1,17 @@ +#!/usr/bin/env python3 + +import argparse +import time + +parser = argparse.ArgumentParser(description="forward I/O requests for seconds") +parser.add_argument( + "integer", + metavar="S", + type=int, + help="an integer for the number of seconds to compute", +) + +args = parser.parse_args() + +print("Will forward I/O requests for " + str(args.integer) + " seconds.") +time.sleep(args.integer) diff --git a/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-submit-api/README.md b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-submit-api/README.md new file mode 100644 index 00000000..cfcd17e6 --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-submit-api/README.md @@ -0,0 +1,115 @@ +# Job Submit API + +To run the following examples, download the files and change your working directory: + +```bash +$ cd flux-workflow-examples/job-submit-api +``` + +## Part(a) - Using a direct job.submit RPC + +### Description: Schedule and launch compute and io-forwarding jobs on separate nodes + +1. Allocate three nodes from a resource manager: + +```bash +salloc -N3 -p pdebug +``` + +2. Launch a Flux instance on the current allocation by running `flux start` once per node, redirecting log messages to the file `out` in the current directory: + +```bash +srun --pty --mpi=none -N3 flux start -o,-S,log-filename=out +``` + +3. Run the submitter executable: + +```bash +python3 ./submitter.py +``` + +4. List currently running jobs: + +```bash +flux jobs +``` +```console +JOBID USER NAME ST NTASKS NNODES RUNTIME +ƒ5W8gVwm fluxuser io-forward R 1 1 19.15s +ƒ5Vd2kJs fluxuser compute.py R 4 2 19.18s +``` + +5. Information about jobs, such as the submitted job specification, an eventlog, and the resource description format **R** are stored in the KVS. The data can be queried via the `job-info` module via the `flux job info` command. For example, to fetch **R** for a job which has been allocated resources: + +```bash +flux job info ƒ5W8gVwm R +``` +```console +{"version":1,"execution":{"R_lite":[{"rank":"2","children":{"core":"0"}}]}} +``` +```bash +flux job info ƒ5Vd2kJs R +``` +```console +{"version":1,"execution":{"R_lite":[{"rank":"0-1","children":{"core":"0-3"}}]}} +``` + +## Part(b) - Using a direct job.submit RPC + +### Schedule and launch both compute and io-forwarding jobs across all nodes + +1. Allocate three nodes from a resource manager: + +```bash +salloc -N3 -p pdebug +``` + +2. Launch another Flux instance on the current allocation: + +```bash +srun --pty --mpi=none -N3 flux start -o,-S,log-filename=out +``` + +3. Run the second submitter executable: + +```bash +python3 ./submitter2.py +``` + +4. List currently running jobs: + +```bash +flux jobs +``` +```console +JOBID USER NAME ST NTASKS NNODES RUNTIME +ƒctYadhh fluxuser io-forward R 3 3 3.058s +ƒct1StnT fluxuser compute.py R 6 3 3.086s +``` + +5. Fetch **R** for the jobs that have been allocated resources: + +```bash +flux job info $(flux job last) jobspec +``` +```console +{"version":1,"execution":{"R_lite":[{"rank":"0-2","children":{"core":"0-3"}}]}} +``` +```console +{"resources": [{"type": "node", "count": 3, "with": [{"type": "slot", "count": 1, "with": [{"type": "core", "count": 1}], "label": "task"}]}], "tasks": [{"command": ["./io-forwarding.py", "120"], "slot": "task", "count": {"per_slot": 1}}], "attributes": {"system": {"duration": 0, "cwd": "/home/fluxuser/flux-workflow-examples/job-submit-api"}}, "version": 1} +``` + +### Notes + +- `f = flux.Flux()` creates a new Flux handle which can be used to connect to and interact with a Flux instance. + +- The following constructs a job request using the **JobspecV1** class with customizable parameters for how you want to utilize the resources allocated for your job: +```python +compute_jobreq = JobspecV1.from_command( + command=["./compute.py", "120"], num_tasks=4, num_nodes=2, cores_per_task=2 +) +compute_jobreq.cwd = os.getcwd() +compute_jobreq.environment = dict(os.environ) +``` + +- `flux.job.submit(f, compute_jobreq)` submits the job to be run, and returns a job ID once it begins running. diff --git a/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-submit-api/compute.py b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-submit-api/compute.py new file mode 100755 index 00000000..1f860f2a --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-submit-api/compute.py @@ -0,0 +1,17 @@ +#!/usr/bin/env python3 + +import argparse +import time + +parser = argparse.ArgumentParser(description="compute for seconds") +parser.add_argument( + "integer", + metavar="S", + type=int, + help="an integer for the number of seconds to compute", +) + +args = parser.parse_args() + +print("Will compute for " + str(args.integer) + " seconds.") +time.sleep(args.integer) diff --git a/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-submit-api/io-forwarding.py b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-submit-api/io-forwarding.py new file mode 100755 index 00000000..217ed0e0 --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-submit-api/io-forwarding.py @@ -0,0 +1,17 @@ +#!/usr/bin/env python3 + +import argparse +import time + +parser = argparse.ArgumentParser(description="forward I/O requests for seconds") +parser.add_argument( + "integer", + metavar="S", + type=int, + help="an integer for the number of seconds to compute", +) + +args = parser.parse_args() + +print("Will forward I/O requests for " + str(args.integer) + " seconds.") +time.sleep(args.integer) diff --git a/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-submit-api/submitter.py b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-submit-api/submitter.py new file mode 100755 index 00000000..51f24089 --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-submit-api/submitter.py @@ -0,0 +1,23 @@ +#!/usr/bin/env python3 + +import json +import os +import re +import flux +from flux.job import JobspecV1 + +f = flux.Flux() + +compute_jobreq = JobspecV1.from_command( + command=["./compute.py", "120"], num_tasks=4, num_nodes=2, cores_per_task=2 +) +compute_jobreq.cwd = os.getcwd() +compute_jobreq.environment = dict(os.environ) +print(flux.job.submit(f, compute_jobreq)) + +io_jobreq = JobspecV1.from_command( + command=["./io-forwarding.py", "120"], num_tasks=1, num_nodes=1, cores_per_task=1 +) +io_jobreq.cwd = os.getcwd() +io_jobreq.environment = dict(os.environ) +print(flux.job.submit(f, io_jobreq)) diff --git a/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-submit-api/submitter2.py b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-submit-api/submitter2.py new file mode 100755 index 00000000..670acff9 --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-submit-api/submitter2.py @@ -0,0 +1,23 @@ +#!/usr/bin/env python3 + +import json +import os +import re +import flux +from flux.job import JobspecV1 + +f = flux.Flux() + +compute_jobreq = JobspecV1.from_command( + command=["./compute.py", "120"], num_tasks=6, num_nodes=3, cores_per_task=2 +) +compute_jobreq.cwd = os.getcwd() +compute_jobreq.environment = dict(os.environ) +print(flux.job.submit(f, compute_jobreq)) + +io_jobreq = JobspecV1.from_command( + command=["./io-forwarding.py", "120"], num_tasks=3, num_nodes=3, cores_per_task=1 +) +io_jobreq.cwd = os.getcwd() +io_jobreq.environment = dict(os.environ) +print(flux.job.submit(f, io_jobreq)) diff --git a/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-submit-cli/README.md b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-submit-cli/README.md new file mode 100644 index 00000000..0a349793 --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-submit-cli/README.md @@ -0,0 +1,61 @@ +# Job Submit CLI + +To run the following examples, download the files and change your working directory: + +```console +$ cd flux-workflow-examples/job-submit-cli +``` + +## Example + +### Launch a flux instance and submit compute and io-forwarding jobs + +If you need an allocation: + +```bash +salloc -N3 -ppdebug +srun --pty --mpi=none -N3 flux start -o,-S,log-filename=out +``` + +To submit + +```bash +# if you have more than one node... +flux submit --nodes=2 --ntasks=4 --cores-per-task=2 ./compute.lua 120 + +# and if not! +flux submit --nodes=1 --ntasks=1 --cores-per-task=2 ./io-forwarding.lua 120 +``` + +Attach to watch output: + +```bash +# Control +C then Control+Z to detach +flux job attach $(flux job last) +``` + +List running jobs: + +```bash +flux jobs +``` +```console +JOBID USER NAME ST NTASKS NNODES RUNTIME +ƒ3ETxsR9H fluxuser io-forward R 1 1 2.858s +ƒ38rBqEWT fluxuser compute.lu R 4 2 15.6s +``` + +Get information about job: + +```bash +flux job info $(flux job last) R +flux job info $(flux job last) jobspec +flux job info $(flux job last) eventlog +flux job info $(flux job last) guest.output + +# Example with flux job id +flux job info ƒ3ETxsR9H R +``` +```console +{"version": 1, "execution": {"R_lite": [{"rank": "0", "children": {"core": "5-7"}}], "nodelist": ["674f16a501e5"], "starttime": 1723225494, "expiration": 4876808372}} +``` diff --git a/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-submit-cli/compute.lua b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-submit-cli/compute.lua new file mode 100755 index 00000000..4fbccc87 --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-submit-cli/compute.lua @@ -0,0 +1,17 @@ +#!/usr/bin/env lua + +local amount = tonumber (arg[1]) or 120 + +local function sleep (n) + os.execute ("sleep " .. n) +end + +if #arg ~= 1 then + print ("Usage: compute.lua seconds") + print (" Compute for seconds") + os.exit (1) +end + +print ("Will compute for " .. amount .. " seconds") +sleep (amount) + diff --git a/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-submit-cli/compute.py b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-submit-cli/compute.py new file mode 100755 index 00000000..1f860f2a --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-submit-cli/compute.py @@ -0,0 +1,17 @@ +#!/usr/bin/env python3 + +import argparse +import time + +parser = argparse.ArgumentParser(description="compute for seconds") +parser.add_argument( + "integer", + metavar="S", + type=int, + help="an integer for the number of seconds to compute", +) + +args = parser.parse_args() + +print("Will compute for " + str(args.integer) + " seconds.") +time.sleep(args.integer) diff --git a/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-submit-cli/io-forwarding.lua b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-submit-cli/io-forwarding.lua new file mode 100755 index 00000000..46ccda0f --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-submit-cli/io-forwarding.lua @@ -0,0 +1,17 @@ +#!/usr/bin/env lua + +local amount = tonumber (arg[1]) or 120 + +local function sleep (n) + os.execute ("sleep " .. n) +end + +if #arg ~= 1 then + print ("Usage: io-forward.lua seconds") + print (" Forward I/O requests for seconds") + os.exit (1) +end + +print ("Will forward IO requests for " .. amount .. " seconds") +sleep (amount) + diff --git a/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-submit-cli/io-forwarding.py b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-submit-cli/io-forwarding.py new file mode 100755 index 00000000..217ed0e0 --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-submit-cli/io-forwarding.py @@ -0,0 +1,17 @@ +#!/usr/bin/env python3 + +import argparse +import time + +parser = argparse.ArgumentParser(description="forward I/O requests for seconds") +parser.add_argument( + "integer", + metavar="S", + type=int, + help="an integer for the number of seconds to compute", +) + +args = parser.parse_args() + +print("Will forward I/O requests for " + str(args.integer) + " seconds.") +time.sleep(args.integer) diff --git a/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-submit-wait/README.md b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-submit-wait/README.md new file mode 100644 index 00000000..c4fbd5d0 --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-submit-wait/README.md @@ -0,0 +1,159 @@ +# Python Job Submit/Wait + +To run the following examples, download the files and change your working directory: + +```bash +$ cd flux-workflow-examples/job-submit-wait +``` + +## Part(a) - Python Job Submit/Wait + +### Description: Submit jobs asynchronously and wait for them to complete in any order + +1. If needed, allocate three nodes from a resource manager: + +```bash +salloc -N3 -ppdebug +``` + +2. Launch a Flux instance on the current allocation by running `flux start` once per node, redirecting log messages to the file `out` in the current directory: + +```bash +srun --pty --mpi=none -N3 flux start -o,-S,log-filename=out +``` + +3. Submit the **submitter_wait_any.py** script, along with the number of jobs you want to run (if no argument is passed, 10 jobs are submitted): + +```bash +python3 ./submitter_wait_any.py 10 +``` +```console +submit: 46912591450240 compute_jobspec +submit: 46912591450912 compute_jobspec +submit: 46912591451080 compute_jobspec +submit: 46912591363152 compute_jobspec +submit: 46912591362984 compute_jobspec +submit: 46912591451360 bad_jobspec +submit: 46912591451528 bad_jobspec +submit: 46912591451696 bad_jobspec +submit: 46912591451864 bad_jobspec +submit: 46912591452032 bad_jobspec +wait: 46912591451528 Error: job returned exit code 1 +wait: 46912591451864 Error: job returned exit code 1 +wait: 46912591451360 Error: job returned exit code 1 +wait: 46912591451696 Error: job returned exit code 1 +wait: 46912591452032 Error: job returned exit code 1 +wait: 46912591450240 Success +wait: 46912591363152 Success +wait: 46912591450912 Success +wait: 46912591451080 Success +wait: 46912591362984 Success +``` + +## Part(b) - Python Job Submit/Wait (Sliding Window) + +### Description: Asynchronously submit jobs and keep at most a number of those jobs active + +1. Allocate three nodes from a resource manager: + +```bash +salloc -N3 -ppdebug +``` + +2. Launch a Flux instance on the current allocation by running `flux start` once per node, redirecting log messages to the file `out` in the current directory: + +```bash +srun --pty --mpi=none -N3 flux start -o,-S,log-filename=out +``` + +3. Submit the **submitter_sliding_window.py** script, along with the number of jobs you want to run and the size of the window (if no argument is passed, 10 jobs are submitted and the window size is 2 jobs): + +```bash +python3 ./submitter_sliding_window.py 10 3 +``` +```console +submit: 5624175788032 +submit: 5624611995648 +submit: 5625014648832 +wait: 5624175788032 Success +submit: 5804329533440 +wait: 5624611995648 Success +submit: 5804648300544 +wait: 5625014648832 Success +submit: 5805084508160 +wait: 5804329533440 Success +submit: 5986144223232 +wait: 5804648300544 Success +submit: 5986462990336 +wait: 5805084508160 Success +submit: 5986882420736 +wait: 5986144223232 Success +submit: 6164435697664 +wait: 5986462990336 Success +wait: 5986882420736 Success +wait: 6164435697664 Success +``` + + +## Part(c) - Python Job Submit/Wait (Specific Job ID) + +### Description: Asynchronously submit jobs, block/wait for specific jobs to complete + +1. Allocate three nodes from a resource manager: + +```bash +salloc -N3 -ppdebug +``` + +2. Launch a Flux instance on the current allocation by running `flux start` once per node, redirecting log messages to the file `out` in the current directory: + +```bash +srun --pty --mpi=none -N3 flux start -o,-S,log-filename=out +``` + +3. Submit the **submitter_wait_in_order.py** script, along with the number of jobs you want to run (if no argument is passed, 10 jobs are submitted): + +```bash +python3 ./submitter_wait_in_order.py 10 +``` +```console +submit: 46912593818008 compute_jobspec +submit: 46912593818176 compute_jobspec +submit: 46912593818344 compute_jobspec +submit: 46912593818512 compute_jobspec +submit: 46912593738048 compute_jobspec +submit: 46912519873816 bad_jobspec +submit: 46912593818792 bad_jobspec +submit: 46912593818960 bad_jobspec +submit: 46912593819128 bad_jobspec +submit: 46912593819296 bad_jobspec +wait: 46912593818008 Success +wait: 46912593818176 Success +wait: 46912593818344 Success +wait: 46912593818512 Success +wait: 46912593738048 Success +wait: 46912519873816 Error: job returned exit code 1 +wait: 46912593818792 Error: job returned exit code 1 +wait: 46912593818960 Error: job returned exit code 1 +wait: 46912593819128 Error: job returned exit code 1 +wait: 46912593819296 Error: job returned exit code 1 +``` + +### Notes + +- The following constructs a job request using the **JobspecV1** class with customizable parameters for how you want to utilize the resources allocated for your job: + +```python +# create jobspec for compute.py +compute_jobspec = JobspecV1.from_command(command=["./compute.py", "15"], num_tasks=4, num_nodes=2, cores_per_task=2) +compute_jobspec.cwd = os.getcwd() +compute_jobspec.environment = dict(os.environ) +``` + +- Using the executor as a context manager (`with FluxExecutor() as executor`) ensures it shuts down properly. + +- `executor.submit(jobspec)` returns a future which completes when the job is done. + +- `future.exception()` blocks until the future is complete and returns (not raises) an exception if the job was canceled or was otherwise prevented from execution. Otherwise the method returns ``None``. + +- `future.result()` blocks until the future is complete and returns the return code of the job. If the job succeeded, the return code will be 0. diff --git a/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-submit-wait/compute.py b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-submit-wait/compute.py new file mode 100755 index 00000000..1f860f2a --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-submit-wait/compute.py @@ -0,0 +1,17 @@ +#!/usr/bin/env python3 + +import argparse +import time + +parser = argparse.ArgumentParser(description="compute for seconds") +parser.add_argument( + "integer", + metavar="S", + type=int, + help="an integer for the number of seconds to compute", +) + +args = parser.parse_args() + +print("Will compute for " + str(args.integer) + " seconds.") +time.sleep(args.integer) diff --git a/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-submit-wait/submitter_sliding_window.py b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-submit-wait/submitter_sliding_window.py new file mode 100755 index 00000000..cfec3111 --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-submit-wait/submitter_sliding_window.py @@ -0,0 +1,54 @@ +#!/usr/bin/env python3 + +import os +import argparse +import collections +import concurrent.futures as cf + +from flux.job import JobspecV1, FluxExecutor + + +def main(): + # parse command line + parser = argparse.ArgumentParser() + parser.add_argument("njobs", nargs="?", type=int, default=10) + parser.add_argument("window_size", nargs="?", type=int, default=2) + args = parser.parse_args() + print(args) + # create jobspec for compute.py + compute_jobspec = JobspecV1.from_command( + command=["./compute.py", "5"], num_tasks=4, num_nodes=2, cores_per_task=2 + ) + compute_jobspec.cwd = os.getcwd() + compute_jobspec.environment = dict(os.environ) + # create a queue of the jobspecs to submit + jobspec_queue = collections.deque(compute_jobspec for _ in range(args.njobs)) + futures = [] # holds incomplete futures + with FluxExecutor() as executor: + while jobspec_queue or futures: + if len(futures) < args.window_size and jobspec_queue: + fut = executor.submit(jobspec_queue.popleft()) + print(f"submit: {id(fut)}") + futures.append(fut) + else: + done, not_done = cf.wait(futures, return_when=cf.FIRST_COMPLETED) + futures = list(not_done) + for fut in done: + if fut.exception() is not None: + print( + f"wait: {id(fut)} Error: job raised error " + f"{fut.exception()}" + ) + elif fut.result() == 0: + print(f"wait: {id(fut)} Success") + else: + print( + f"wait: {id(fut)} Error: job returned " + f"exit code {fut.result()}" + ) + + +if __name__ == "__main__": + main() + +# vim: tabstop=4 shiftwidth=4 expandtab diff --git a/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-submit-wait/submitter_wait_any.py b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-submit-wait/submitter_wait_any.py new file mode 100755 index 00000000..890a1f0a --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-submit-wait/submitter_wait_any.py @@ -0,0 +1,45 @@ +#!/usr/bin/env python3 + +import os +import argparse +import concurrent.futures + +from flux.job import JobspecV1, FluxExecutor + + +def main(): + # parse command line + parser = argparse.ArgumentParser() + parser.add_argument("njobs", nargs="?", type=int, default=10) + args = parser.parse_args() + # create jobspec for compute.py + compute_jobspec = JobspecV1.from_command( + command=["./compute.py", "10"], num_tasks=4, num_nodes=2, cores_per_task=2 + ) + compute_jobspec.cwd = os.getcwd() + compute_jobspec.environment = dict(os.environ) + # create bad jobspec that will fail + bad_jobspec = JobspecV1.from_command(["/bin/false"]) + # create an executor to submit jobs + with FluxExecutor() as executor: + futures = [] + # submit half successful jobs and half failures + for _ in range(args.njobs // 2): + futures.append(executor.submit(compute_jobspec)) + print(f"submit: {id(futures[-1])} compute_jobspec") + for _ in range(args.njobs // 2, args.njobs): + futures.append(executor.submit(bad_jobspec)) + print(f"submit: {id(futures[-1])} bad_jobspec") + for fut in concurrent.futures.as_completed(futures): + if fut.exception() is not None: + print(f"wait: {id(fut)} Error: job raised error {fut.exception()}") + elif fut.result() == 0: + print(f"wait: {id(fut)} Success") + else: + print(f"wait: {id(fut)} Error: job returned exit code {fut.result()}") + + +if __name__ == "__main__": + main() + +# vim: tabstop=4 shiftwidth=4 expandtab diff --git a/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-submit-wait/submitter_wait_in_order.py b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-submit-wait/submitter_wait_in_order.py new file mode 100755 index 00000000..cad64918 --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-submit-wait/submitter_wait_in_order.py @@ -0,0 +1,44 @@ +#!/usr/bin/env python3 + +import argparse +import os + +from flux.job import JobspecV1, FluxExecutor + + +def main(): + # parse command line + parser = argparse.ArgumentParser() + parser.add_argument("njobs", nargs="?", type=int, default=10) + args = parser.parse_args() + # create jobspec for compute.py + compute_jobspec = JobspecV1.from_command( + command=["./compute.py", "10"], num_tasks=4, num_nodes=2, cores_per_task=2 + ) + compute_jobspec.cwd = os.getcwd() + compute_jobspec.environment = dict(os.environ) + bad_jobspec = JobspecV1.from_command(["/bin/false"]) + # create an executor to submit jobs + with FluxExecutor() as executor: + futures = [] + # submit half successful jobs and half failures + for _ in range(args.njobs // 2): + futures.append(executor.submit(compute_jobspec)) + print(f"submit: {id(futures[-1])} compute_jobspec") + for _ in range(args.njobs // 2, args.njobs): + futures.append(executor.submit(bad_jobspec)) + print(f"submit: {id(futures[-1])} bad_jobspec") + # wait for each future in turn + for fut in futures: + if fut.exception() is not None: + print(f"wait: {id(fut)} Error: job raised error {fut.exception()}") + elif fut.result() == 0: + print(f"wait: {id(fut)} Success") + else: + print(f"wait: {id(fut)} Error: job returned exit code {fut.result()}") + + +if __name__ == "__main__": + main() + +# vim: tabstop=4 shiftwidth=4 expandtab diff --git a/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-watch/job-watch.sh b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-watch/job-watch.sh new file mode 100755 index 00000000..df379bc6 --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/job-watch/job-watch.sh @@ -0,0 +1,11 @@ +#!/bin/bash + +echo "25 chocolate chip pancakes on the table... 25 chocolate chip pancakes! 🥞️" +sleep 3 +echo "Eat a stack, for a snack, 15 chocolate chip pancakes on the table! 🥄️" +sleep 3 +echo "15 chocolate chip pancakes on the table... 15 chocolate chip pancakes! 🥞️" +sleep 2 +echo "Throw a stack... it makes a smack! 15 chocolate chip pancakes on the wall! 🥞️" +sleep 2 +echo "You got some cleaning to do 🧽️" diff --git a/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/kvs-python-bindings/README.md b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/kvs-python-bindings/README.md new file mode 100644 index 00000000..5c3aa227 --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/kvs-python-bindings/README.md @@ -0,0 +1,69 @@ +# KVS Python Binding Example + +## Description: Use the KVS Python interface to store user data into KVS + +If you haven't already, download the files and change your working directory: + +```console +$ cd flux-workflow-examples/kvs-python-bindings +``` + +1. Launch a Flux instance by running `flux start`, redirecting log messages to the file `out` in the current directory: + +```bash +flux start -s 1 -o,-S,log-filename=out +``` + +2. Submit the Python script: + +```bash +flux submit -N 1 -n 1 ./kvsput-usrdata.py +``` +```console +6705031151616 +``` + +3. Attach to the job and view output: + +```bash +flux job attach $(flux job last) +``` +```console +hello world +hello world again +``` + +4. Each job is run within a KVS namespace. `FLUX_KVS_NAMESPACE` is set, which is automatically read and used by the KVS operations in the handle. To take a look at the job's KVS, convert its job ID to KVS: + +```bash +flux job id --to=kvs $(flux job last) +``` +```console +job.0000.0619.2300.0000 +``` + +5. The keys for this job will be put at the root of the namespace, which is mounted under "guest". To get the value stored under the first key "usrdata": + +```bash +flux kvs get job.0000.0619.2300.0000.guest.usrdata +``` +```bash +"hello world" +``` + +6. Get the value stored under the second key "usrdata2": + +```bash +flux kvs get job.0000.0619.2300.0000.guest.usrdata2 +``` +```console +"hello world again" +``` + +### Notes + +- `f = flux.Flux()` creates a new Flux handle which can be used to connect to and interact with a Flux instance. + +- `kvs.put()` places the value of _udata_ under the key **"usrdata"**. Once the key-value pair is put, the change must be committed with `kvs.commit()`. The value can then be retrieved with `kvs.get()` + +- `kvs.get()` on a directory will return a KVSDir object which supports the `with` compound statement. `with` guarantees a commit is called on the directory. diff --git a/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/kvs-python-bindings/kvsput-usrdata.py b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/kvs-python-bindings/kvsput-usrdata.py new file mode 100755 index 00000000..0a5cb770 --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/kvs-python-bindings/kvsput-usrdata.py @@ -0,0 +1,24 @@ +#!/usr/bin/env python3 + +import sys +import flux +import os +from flux import kvs + +f = flux.Flux() +udata = "hello world" +# using function interface +kvs.put(f, "usrdata", udata) +# commit is required to effect the above put op to the server +kvs.commit(f) +print(kvs.get(f, "usrdata")) + +# get() on a directory will return a KVSDir object which supports +# the "with" compound statement. "with" guarantees a commit is called +# on the directory. +with kvs.get(f, ".") as kd: + kd["usrdata2"] = "hello world again" + +print(kvs.get(f, "usrdata2")) + +# vi: ts=4 sw=4 expandtab diff --git a/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/requirements.txt b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/requirements.txt new file mode 100644 index 00000000..1463f8f1 --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/requirements.txt @@ -0,0 +1,3 @@ +sphinx-rtd-theme +sphinxcontrib-spelling +recommonmark diff --git a/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/synchronize-events/README.md b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/synchronize-events/README.md new file mode 100644 index 00000000..641be098 --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/synchronize-events/README.md @@ -0,0 +1,54 @@ +# Using Events with Separate Nodes + +## Description: Using events to synchronize compute and io-forwarding jobs running on separate nodes + +If you haven't already, download the files and change your working directory: + +```console +$ cd flux-workflow-examples/synchronize-events +``` + +Ask for a Slurm allocation, if relevant: + +```bash +salloc -N3 -ppdebug +srun --pty --mpi=none -N3 flux start -o,-S,log-filename=out +flux submit --nodes=1 --ntasks=4 --cores-per-task=2 ./compute.lua 120 +``` + +And: + +```bash +flux submit --nodes=1 --ntasks=1 --cores-per-task=2 ./io-forwarding.lua 120 +``` + +5. List running jobs: + +```bash +flux jobs +``` +``` +JOBID USER NAME ST NTASKS NNODES RUNTIME RANKS +ƒA4TgT7d fluxuser io-forward R 1 1 4.376s 2 +ƒ6vEcj7M fluxuser compute.lu R 4 2 11.51s [0-1] +``` + +6. Attach to running or completed job output: + +```bash +flux job attach ƒ6vEcj7M +``` +```console +Block until we hear go message from the an io forwarder +Block until we hear go message from the an io forwarder +Recv an event: please proceed +Recv an event: please proceed +Will compute for 120 seconds +Will compute for 120 seconds +Block until we hear go message from the an io forwarder +Block until we hear go message from the an io forwarder +Recv an event: please proceed +Recv an event: please proceed +Will compute for 120 seconds +Will compute for 120 seconds +``` diff --git a/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/synchronize-events/compute.lua b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/synchronize-events/compute.lua new file mode 100755 index 00000000..925be4cb --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/synchronize-events/compute.lua @@ -0,0 +1,23 @@ +#!/usr/bin/env lua + +local f, err = require 'flux' .new () + +local amount = tonumber (arg[1]) or 120 + +local function sleep (n) + os.execute ("sleep " .. n) +end + +if #arg ~= 1 then + print ("Usage: compute.lua seconds") + print (" Compute for seconds") + os.exit (1) +end + +print ("Block until we hear go message from the an io forwarder") +f:subscribe ("app.iof.go") +local t, tag = f:recv_event () +print ("Recv an event: " .. t.data ) +print ("Will compute for " .. amount .. " seconds") +sleep (amount) + diff --git a/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/synchronize-events/io-forwarding.lua b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/synchronize-events/io-forwarding.lua new file mode 100755 index 00000000..bad77f8b --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/assets/flux-workflow-examples/synchronize-events/io-forwarding.lua @@ -0,0 +1,23 @@ +#!/usr/bin/env lua + +local flux = require 'flux' +local f = flux.new () +local amount = tonumber (arg[1]) or 120 + +local function sleep (n) + os.execute ("sleep " .. n) +end + +if #arg ~= 1 then + print ("Usage: io-forward.lua seconds") + print (" Forward I/O requests for seconds") + os.exit (1) +end + +local rc, err = f:sendevent ({ data = "please proceed" }, "app.iof.go") +if not rc then error (err) end +print ("Sent a go event") + +print ("Will forward IO requests for " .. amount .. " seconds") +sleep (amount) + diff --git a/2026/HPCIC-AWS/tutorial/ch1/01_flux_tutorial.ipynb b/2026/HPCIC-AWS/tutorial/ch1/01_flux_tutorial.ipynb new file mode 100644 index 00000000..f8727dbb --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/ch1/01_flux_tutorial.ipynb @@ -0,0 +1,726 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "2507d149-dcab-458a-a554-37388e0ee13a", + "metadata": { + "tags": [] + }, + "source": [ + "
\n", + "
\n", + "
" + ] + }, + { + "cell_type": "markdown", + "id": "40e867ba-f689-4301-bb60-9a448556bb84", + "metadata": { + "tags": [] + }, + "source": [ + "# Welcome to the Flux Tutorial\n", + "\n", + "> What is Flux Framework? 🤔️\n", + " \n", + "Flux is a flexible framework for resource management, built for your site. The framework consists of a suite of projects, tools, and libraries that may be used to build site-custom resource managers for High Performance Computing centers and cloud environments. Flux is a next-generation resource manager and scheduler with many transformative capabilities like hierarchical scheduling and resource management (you can think of it as \"fractal scheduling\") and directed-graph based resource representations.\n", + "\n", + "## I'm ready! How do I do this tutorial? 😁️\n", + "\n", + "This tutorial is split into 3 chapters, each of which has a notebook. Let's first setup your workspace. Click on to create a Terminal. Drag it to be alongside this notebook so you have a terminal on the left alongside this text on the right (you should see both at once).\n", + "\n", + "When you are done, let's get started!" + ] + }, + { + "cell_type": "markdown", + "id": "15e82c38-8465-49ac-ae2b-b0bb56a79ec9", + "metadata": { + "tags": [] + }, + "source": [ + "\n", + "# Getting started with Flux\n", + "\n", + "The code and examples that this tutorial is based on can be found at [flux-framework/Tutorials](https://github.com/flux-framework/Tutorials/tree/master/2024-HPCIC-AWS). You can also find python examples in the `assets/flux-workflow-examples` directory from the sidebar navigation in this JupyterLab instance. " + ] + }, + { + "cell_type": "markdown", + "id": "ae33fef6-278c-4996-8534-fd15e548b338", + "metadata": { + "tags": [] + }, + "source": [ + "
\n", + "Tip: Did you know you can get help for flux or a flux command? For example, try \"flux help\" and \"flux help jobs\"\n", + "
\n", + "\n", + "Try asking for help first across top level commands.\n", + "\n", + "```bash\n", + "flux help\n", + "```\n", + "\n", + "Now try for a specific command\n", + "\n", + "```bash\n", + "flux help jobs\n", + "```\n", + "The second will open a manpage, and you can press \"Q\" for quit to exit. Now let's get help for a specific subcommand. This works too:\n", + "\n", + "```bash\n", + "flux submit --help\n", + "```\n" + ] + }, + { + "cell_type": "markdown", + "id": "ec052119", + "metadata": {}, + "source": [ + "## Flux Resources\n", + "\n", + "When you are interacting with Flux, you will commonly want to know what resources are available to you. Flux uses [hwloc](https://github.com/open-mpi/hwloc) to detect the resources on each node and then to populate its resource graph.\n", + "\n", + "You can access the topology information that Flux collects with the `flux resource` subcommand. Let's run `flux resource list` to see the resources available to us in this notebook.\n", + "\n", + "```bash\n", + "flux resource list\n", + " STATE NNODES NCORES NGPUS NODELIST\n", + " free 1 64 0 ip-10-0-25-10\n", + " allocated 0 0 0 \n", + " down 0 0 0 \n", + "```" + ] + }, + { + "cell_type": "markdown", + "id": "0086e47e", + "metadata": {}, + "source": [ + "Flux can also bootstrap its resource graph based on static input files, like in the case of a multi-user system instance setup by site administrators. [More information on Flux's static resource configuration files](https://flux-framework.readthedocs.io/projects/flux-core/en/latest/guide/admin.html#configuration). Flux provides a more standard interface to listing available resources that works regardless of the resource input source: `flux resource`.\n", + "\n", + "```bash\n", + "# To view status of resources\n", + "flux resource status\n", + " STATE UP NNODES NODELIST\n", + " avail ✔ 1 ip-10-0-25-10\n", + "```\n", + "The output above may vary based on your resources. It might also be the case that you need to see queues. Here is how to do that:\n", + "\n", + "```bash\n", + "flux queue list\n", + " EN ST TDEFAULT TLIMIT NNODES NCORES NGPUS\n", + " ✔ ✔ inf inf 0-1 0-64 0-0\n", + "```" + ] + }, + { + "cell_type": "markdown", + "id": "dee2d6af-43fa-490e-88e9-10f13e660125", + "metadata": { + "tags": [] + }, + "source": [ + "
\n", + "\n", + "# Flux Commands \n", + "\n", + "Here are how Flux commands map to a scheduler you are likely familiar with, Slurm. A larger table with similar mappings for LSF, Moab, and Slurm can be [viewed here](https://hpc.llnl.gov/banks-jobs/running-jobs/batch-system-cross-reference-guides). For submitting jobs, you can use the `flux` `submit`, `run`, `bulksubmit`, `batch`, and `alloc` commands.\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
OperationSlurmFlux
One-off run of a single job (blocking)srunflux run
One-off run of a single job (interactive)srun --ptyflux run -o pty.interactive
One-off run of a single job (not blocking)NAflux submit
Bulk submission of jobs (not blocking)NAflux bulksubmit
Watching jobsNAflux watch
Querying the status of jobssqueue/scontrol show job job_idflux jobs/flux job info job_id
Canceling running jobsscancelflux cancel
Allocation for an interactive instancesallocflux alloc
Submitting batch jobssbatchflux batch
" + ] + }, + { + "cell_type": "markdown", + "id": "ac798095", + "metadata": {}, + "source": [ + "## flux run\n", + "\n", + "
\n", + "Description: Running a single job (blocking)\n", + "
\n", + "\n", + "The `flux run` command submits a job to Flux (similar to `flux submit`) but then attaches to the job with `flux job attach`, printing the job's stdout/stderr to the terminal and exiting with the same exit code as the job. It's basically doing an interactive submit, because you will be able to watch the output in your terminal, and it will block your terminal until the job completes.\n", + "\n", + "```bash\n", + "flux run sh -c 'sleep 5 && echo hello'\n", + "hello\n", + "```\n", + "\n", + "The output from the previous command is the hostname (a container ID string in this case). If the job exits with a non-zero exit code this will be reported by `flux job attach` (occurs implicitly with `flux run`). For example, execute the following:\n", + "\n", + "```bash\n", + "flux run /bin/false\n", + "flux-job: task(s) exited with exit code 1\n", + "```\n", + "A job submitted with `run` can be canceled with two rapid `Cltr-C`s in succession, or a user can detach from the job with `Ctrl-C Ctrl-Z`. The user can then re-attach to the job by using `flux job attach JOBID`. `flux submit` and `flux run` also support many other useful flags:\n", + "\n", + "```bash\n", + "flux run -N1 -n1 -c1 hostname\n", + "ip-10-0-25-10\n", + "```\n", + "\n", + "```bash\n", + "flux run -n4 --label-io --time-limit=5s --env-remove=LD_LIBRARY_PATH hostname\n", + "3: ip-10-0-25-10\n", + "2: ip-10-0-25-10\n", + "1: ip-10-0-25-10\n", + "0: ip-10-0-25-10\n", + "```\n", + "\n", + "To see all flags for a `flux run`:\n", + "\n", + "```bash\n", + "flux run --help\n", + "```\n", + "\n", + "Let's try compiling the Makefile in this directory.\n", + "\n", + "```bash\n", + "flux run make -j\n", + "mpicc -o hello hello.c\n", + "```\n", + "And run an exclusive job using all the processes on the node:\n", + "\n", + "```bash\n", + "flux run -N1 -n32 -c2 --exclusive ./hello\n", + "ƒ2261FtDpK: completed MPI_Init in 2.171s. There are 32 tasks\n", + "ƒ2261FtDpK: completed first barrier in 0.001s\n", + "ƒ2261FtDpK: completed MPI_Finalize in 0.021s\n", + "```" + ] + }, + { + "cell_type": "markdown", + "id": "7c09708a-74a1-4e61-b678-cb337b7df435", + "metadata": {}, + "source": [ + "## flux submit\n", + "\n", + "
\n", + "Description: Running a single job (not blocking)\n", + "
\n", + "\n", + "\n", + "The `flux submit` command submits a job to Flux and prints out the jobid. Let's peek at the help first.\n", + "\n", + "```bash\n", + "flux submit --help\n", + "```\n", + "\n", + "Let's test with a basic hostname.\n", + "\n", + "```bash\n", + "flux submit hostname\n", + "```\n", + "\n", + "But how does one get output? To quickly see output (which will block the terminal if the job is still running) after a submit, you can do:\n", + "\n", + "```bash\n", + "flux job attach $(flux job last)\n", + "```\n", + "\n", + "How does that work? The command `flux job last` provides the last job identifier.\n", + "\n", + "```bash\n", + "flux job last\n", + "```\n", + "\n", + "And when we attach to it, if the job is still running we will block the terminal. To provide a custom path to an output or error file, you can provide `--out` and `--err`, respectively. Let's try those both now. Now let's submit another one, and give it the same output and error file\n", + "\n", + "```bash\n", + "flux submit --out /tmp/harry-potter.txt --err /tmp/harry-potter.txt echo \"Yer a wizard, $(whoami)!\"\n", + "```\n", + "\n", + "Take a look!\n", + "\n", + "```bash\n", + "cat /tmp/harry-potter.txt\n", + "```\n", + "\n", + "The `submit` command supports common options like `--nnodes`, `--ntasks`, and `--cores-per-task`. There are short option equivalents (`-N`, `-n`, and `-c`, respectively) of these options as well. `--cores-per-task=1` is the default.\n", + "\n", + "```bash\n", + "flux submit -N1 -n2 sleep inf\n", + "```\n", + "\n", + "#### Some submission flags of note\n", + "\n", + "* `-N` specifies a number of nodes\n", + "* `-n` specifies a number of tasks for distributed applications, and cores for interactive allocations\n", + "* `-c` specifies a number of cores per task\n", + "* `--requires` constrains a job to run on a specific rank or hostname\n", + "* `--dependency` makes a job depend on another job\n", + "* `-cc` submits carbon copies of the same job many times\n", + "* `--out` and `--err` redirect output and error to files\n", + "\n", + "Let's work through some examples of these flags!" + ] + }, + { + "cell_type": "markdown", + "id": "91e9ed6c", + "metadata": {}, + "source": [ + "## flux bulksubmit\n", + "\n", + "
\n", + "Description: Submitting jobs in bulk (not blocking)\n", + "
\n", + "\n", + "The `flux bulksubmit` command enqueues jobs based on a set of inputs which are substituted on the command line, similar to `xargs` and the GNU `parallel` utility, except the jobs have access to the resources of an entire Flux instance instead of only the local system.\n", + "\n", + "```bash\n", + "flux bulksubmit --wait echo {} ::: harry ron hermione\n", + "ƒ3Z7z2ZZ6o\n", + "ƒ3Z7z2ZZ6p\n", + "ƒ3Z7z2ZZ6q\n", + "```\n", + "\n", + "### carbon copy\n", + "\n", + "The `--cc` option (akin to \"carbon copy\") to `submit` makes repeated submission even easier via, `flux submit --cc=IDSET`:\n", + "\n", + "```bash\n", + "flux submit --watch --cc=1-4 echo {cc}\n", + "ƒ24R3EgAAB\n", + "ƒ24R3EgAAC\n", + "ƒ24R3EgAAD\n", + "ƒ24R3EgAAE\n", + "1\n", + "2\n", + "3\n", + "4\n", + "```\n", + "\n", + "Try this to add a progress bar and see the jobs/s rate report:\n", + "\n", + "```bash\n", + "flux submit --cc=1-100 --watch --progress --jps hostname\n", + "```\n", + "\n", + "Note that `--wait` is implied by `--watch`, meaning that when you are watching jobs, you are also waiting for them to finish. Here are some other carbon copy commands that are useful to try:\n", + "\n", + "```bash\n", + "# Use flux carbon copy to submit identical jobs with different inputs\n", + "flux submit --cc=\"1-10\" echo \"Hello I am job {cc}\"\n", + "\n", + "# Submits scripts myscript1.sh through myscript10.sh\n", + "flux submit --cc=0-6 flux-workflow-examples/bulksubmit/{cc}.sh\n", + "\n", + "# Bypass the key value store and write output to file with jobid\n", + "flux submit --cc=1-10 --output=job-{{id}}.out echo \"This is job {cc}\"\n", + "\n", + "# Use carbon copy to submit identical jobs with different inputs\n", + "flux bulksubmit --dry-run --cc={1} echo {0} ::: a b c ::: 0-1 0-3 0-7\n", + "```\n", + "\n", + "Of course, Flux can launch more than just single-node, single-core jobs. We can submit multiple heterogeneous jobs and Flux will co-schedule the jobs while also ensuring no oversubscription of resources (e.g., cores). Let's run the second example here, and add a clever trick to ask for output as we submit the jobs. This is a fun one, I promise!\n", + "\n", + "```bash\n", + "for jobid in $(flux submit --cc=0-7 /bin/bash bulksubmit/{cc}.sh); \n", + " do \n", + " flux job attach ${jobid}; \n", + " sleep 1; \n", + "done\n", + "```\n", + "\n", + "Note: in this tutorial, we cannot assume that the host you are running on has multiple cores, thus the examples below only vary the number of nodes per job. Varying the `cores-per-task` is also possible on Flux when the underlying hardware supports it (e.g., a multi-core node). Here are two examples.\n", + "\n", + "```bash\n", + "# Two nodes, 2 tasks, and each task has one core\n", + "flux submit --nodes=2 --ntasks=2 --cores-per-task=1 --job-name magic sleep inf\n", + "\n", + "# One node, one task, and each task has one core\n", + "flux submit --nodes=1 --ntasks=1 --cores-per-task=1 --job-name moremagic sleep inf\n", + "```" + ] + }, + { + "cell_type": "markdown", + "id": "641f446c-b2e8-40d8-b6bd-eb6b9dba3c71", + "metadata": {}, + "source": [ + "## flux watch\n", + "\n", + "
\n", + "Description: 👀️ Watching jobs\n", + "
\n", + "\n", + "Wouldn't it be cool to submit a job and then watch it? Well, yeah! We can do this now with flux watch. Let's run a fun example, and then watch the output. We have sleeps in here interspersed with echos only to show you the live action! 🥞️\n", + "Also note a nice trick - you can always use `flux job last` to get the last JOBID.\n", + "Here is an example (not runnable, as notebooks don't support environment variables) for getting and saving a job id:\n", + "\n", + "```bash\n", + "flux submit hostname\n", + "JOBID=$(flux job last)\n", + "```\n", + "\n", + "And then you could use the variable `$JOBID` in your subsequent script or interactions with Flux! So what makes `flux watch` different from `flux job attach`? Aside from the fact that `flux watch` is read-only, `flux watch` can watch many (or even all (`flux watch --all`) jobs at once!\n", + "\n", + "```bash\n", + "flux submit ./job-watch.sh\n", + "flux watch $(flux job last)\n", + "```" + ] + }, + { + "cell_type": "markdown", + "id": "3f8c2af2", + "metadata": {}, + "source": [ + "## flux jobs\n", + "\n", + "
\n", + "Description: Querying the status of jobs\n", + "
\n", + "\n", + "We can now list the jobs in the queue with `flux jobs` and we should see both jobs that we just submitted. Jobs that are instances are colored blue in output, red jobs are failed jobs, and green jobs are those that completed successfully. Note that the JupyterLab notebook may not display these colors. You will be able to see them in the terminal.\n", + "\n", + "```bash\n", + "flux submit sleep inf\n", + "flux jobs\n", + "```\n", + "\n", + "You might also want to see all the jobs with `-a`:\n", + "\n", + "```bash\n", + "flux jobs -a\n", + "```" + ] + }, + { + "cell_type": "markdown", + "id": "77ca4277", + "metadata": {}, + "source": [ + "## flux cancel\n", + "\n", + "
\n", + "Description: Canceling running jobs\n", + "
\n", + "\n", + "Since some of the jobs we see in the table above won't ever exit (and we didn't specify a timelimit), let's cancel them all now and free up the resources.\n", + "\n", + "```bash\n", + "# This was previously flux cancelall -f\n", + "flux cancel --all\n", + "flux jobs\n", + "```" + ] + }, + { + "cell_type": "markdown", + "id": "2d3e314e-98eb-487a-ad8e-1442840e37d8", + "metadata": {}, + "source": [ + "## flux alloc\n", + "\n", + "
\n", + "Description: Allocation for an interactive instance\n", + "
\n", + "\n", + "You might want to request an allocation for a set of resources (an allocation) and then attach to them interactively. This is the goal of flux alloc. Try these commands in the terminal: \n", + "\n", + "```bash\n", + "# Look at the resources you have outside of the allocation\n", + "flux resource list\n", + "\n", + "# Request an allocation with 2 \"nodes\" - a subset of what you have in total\n", + "flux alloc -N 2\n", + "\n", + "# See the resources you are given\n", + "flux resource list\n", + "\n", + "# You can exit from the allocation like this!\n", + "exit\n", + "```\n", + "When you want to automate this, submitting work to an allocation, you would use `flux batch`." + ] + }, + { + "cell_type": "markdown", + "id": "544aa0a9", + "metadata": {}, + "source": [ + "## flux batch\n", + "\n", + "
\n", + "Description: Submitting batch jobs\n", + "
\n", + "\n", + "We can use the `flux batch` command to easily created nested flux instances. When `flux batch` is invoked, Flux will automatically create a nested instance that spans the resources allocated to the job, and then Flux runs the batch script passed to `flux batch` on rank 0 of the nested instance. \"Rank\" refers to the rank of the Tree-Based Overlay Network (TBON) used by the [Flux brokers](https://flux-framework.readthedocs.io/projects/flux-core/en/latest/man1/flux-broker.html).\n", + "\n", + "While a batch script is expected to launch parallel jobs using `flux run` or `flux submit` at this level, nothing prevents the script from further batching other sub-batch-jobs using the `flux batch` interface, if desired. Take a quick look at [sleep_batch.sh](sleep_batch.sh) to see what we are about to run, then run:\n", + "\n", + "```bash\n", + "flux batch --nslots=2 --cores-per-slot=1 --nodes=2 ./sleep_batch.sh\n", + "flux batch --nslots=2 --cores-per-slot=1 --nodes=2 ./sleep_batch.sh\n", + "```" + ] + }, + { + "cell_type": "markdown", + "id": "7f2b135c-ece7-45f7-b25d-dc90ba5f44f7", + "metadata": {}, + "source": [ + "### `flux job`\n", + "\n", + "Let's next inspect the last job we ran with `flux job info` and target the last job identifier with `flux job last`. \n", + "\n", + "```bash\n", + "# Note here we are using flux job last to see the last job id\n", + "# The \"R\" here asks for the resource spec\n", + "flux job info $(flux job last) R\n", + "\n", + "# When we attach it will direct us to our output file\n", + "flux job attach $(flux job last)\n", + "```\n", + "\n", + "We can again see a list all completed jobs with `flux jobs -a`. To restrict the output to failed (i.e., jobs that exit with nonzero exit code, time out, or are canceled or killed) jobs, run:\n", + "\n", + "```bash\n", + "flux jobs -f failed\n", + "```" + ] + }, + { + "cell_type": "markdown", + "id": "6bc17bac-2fc4-4418-8939-e930f9929976", + "metadata": {}, + "source": [ + "### flux submit from within a batch\n", + "\n", + "Next open up [hello-batch.sh](hello-batch.sh) to see an example of using `flux batch` to submit jobs within the instance, and then wait for them to finish. This script is going to:\n", + "\n", + "1. Create a flux instance with the top level resources you specify\n", + "2. Submit jobs to the scheduler controlled by the broker of that sub-instance\n", + "3. Run the four jobs, with `--flags=waitable` and `flux job wait --all` to wait for the output file\n", + "4. Within the batch script, you can add `--wait` or `--flags=waitable` to individual jobs, and use `flux queue drain` to wait for the queue to drain, _or_ `flux job wait --all` to wait for the jobs you flagged to finish. \n", + "\n", + "Note that when you submit a batch job, you'll get a job id back for the _batch job_, and usually when you look at the output of that with `flux job attach $jobid` you will see the output file(s) where the internal contents are written. Since we want to print the output file easily to the terminal, we are waiting for the batch job by adding the `--flags=waitable` and then waiting for it. Let's try to run our batch job now.\n", + "\n", + "```bash\n", + "flux batch --flags=waitable --out /tmp/flux-batch.out -N1 ./hello-batch.sh\n", + "flux job wait\n", + "cat /tmp/hello-batch-1.out\n", + "cat /tmp/hello-batch-2.out\n", + "cat /tmp/hello-batch-3.out\n", + "cat /tmp/hello-batch-4.out\n", + "```\n", + "```console\n", + "Hello job 1 from ip-10-0-25-10 💛️\n", + "Hello job 2 from ip-10-0-25-10 💚️\n", + "Hello job 3 from ip-10-0-25-10 💙️\n", + "Hello job 4 from ip-10-0-25-10 💜️\n", + "```" + ] + }, + { + "cell_type": "markdown", + "id": "04b405b1-219f-489c-abfc-e2983e82124a", + "metadata": {}, + "source": [ + "### The Flux Hierarchy 🍇️\n", + "\n", + "One feature of the Flux Framework scheduler that is unique is its ability to submit jobs within instances, where an instance can be thought of as a level in a graph. Let's start with a basic image - this is what it might look like to submit to a scheduler that is not graph-based (left), where all jobs go to a central job queue or database. Note that our maximum job throughput is one job per second. The throughput is limited by the workload manager's ability to process a single job. We can improve upon this by simply adding another level, perhaps with three instances. For example, let's say we create a flux allocation or batch that has control of some number of child nodes. We might launch three new instances (each with its own scheduler and queue, right image) at that level two, and all of a sudden, we get a throughput of 1x3, or three jobs per second.\n", + "\n", + "\n", + "\n", + " \n", + " \n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + "\n", + "All of a sudden, the throughout can increase exponentially because we are essentially submitting to different schedulers. The example above is not impressive, but our [learning guide](https://flux-framework.readthedocs.io/en/latest/guides/learning_guide.html#fully-hierarchical-resource-management-techniques) (Figure 10) has a beautiful example of how it can scale, done via an actual experiment. We were able to submit 500 jobs/second using only three levels, vs. close to 1 job/second with one level. For an interesting detail, you can vary the scheduler algorithm or topology within each sub-instance, meaning that you can do some fairly interesting things with scheduling work, and all without stressing the top level system instance. \n", + "\n", + "Now that we understand nested instances, let's look at another batch example that better uses them. Here we have two job scripts:\n", + "\n", + "- [sub_job1.sh](sub_job1.sh): Is going to be run with `flux batch` and submit sub_job2.sh\n", + "- [sub_job2.sh](sub_job2.sh): Is going to be submitted by sub_job1.sh.\n", + "\n", + "Take a look at each script to see how they work, and then submit it!\n", + "\n", + "```bash\n", + "flux batch ./sub_job1.sh\n", + "```\n", + "\n", + "And now that we've submitted, let's look at the hierarchy for all the jobs we just ran. Here is how to try flux pstree, which normally can show jobs in an instance, but it has limited functionality given we are in a notebook! So instead of just running the single command, let's add \"-a\" to indicate \"show me ALL jobs.\"\n", + "More complex jobs and in a different environment would have deeper nesting. You can [see examples here](https://flux-framework.readthedocs.io/en/latest/jobs/hierarchies.html?h=pstree#flux-pstree-command).\n", + "\n", + "```bash\n", + "flux pstree -a\n", + ".\n", + "├── ./sub_job1.sh:CD\n", + "├── ./hello-batch.sh:CD\n", + "├── ./hello-batch.sh:F\n", + "├── 11*[echo:CD]\n", + "├── 2*[./sleep_batch.sh:F]\n", + "├── sleep:CA\n", + "├── job-watch.sh:CD\n", + "├── 104*[hostname:CD]\n", + "├── moremagic:F\n", + "├── magic:F\n", + "├── 8*[bash:CD]\n", + "├── hello:CD\n", + "├── hello:F\n", + "├── 2*[make:CD]\n", + "├── make:F\n", + "├── false:F\n", + "└── sh:CD\n", + "```" + ] + }, + { + "cell_type": "markdown", + "id": "7724130f-b0db-4ccf-a01e-98907b9a27ca", + "metadata": {}, + "source": [ + "You can also try a more detailed view with `flux pstree -a -X`!" + ] + }, + { + "cell_type": "markdown", + "id": "70e3df1d-32c9-4996-b6f7-2fa85f4c02ad", + "metadata": { + "tags": [] + }, + "source": [ + "### flux start\n", + "\n", + "
\n", + "Description: Interactively starting a set of resources\n", + "
\n", + "\n", + "Sometimes you need to interactively start a set of compute resources. We call this subset a flux instance. You can launch jobs under this instance, akin to how you've done above! In fact, this entire tutorial is started (to give you 4 faux nodes) with a `flux start` command: \n", + "\n", + "```bash\n", + "flux start --test-size=4\n", + "```\n", + "\n", + "A Flux instance may be running as the default resource manager on a cluster, a job in a resource manager such as Slurm, LSF, or Flux itself, or as a test instance launched locally. This is really neat because it means you can launch Flux under other resource managers where it is not installed as the system workload manager. You can also execute \"one off\" commands to it, for example, to see the instance size:\n", + "\n", + "```bash\n", + "flux start --test-size=4 flux getattr size\n", + "4\n", + "```\n", + "\n", + "When you run `flux start` without a command, it will give you an interactive shell to the instance. When you provide a command (as we do above) it will run it and exit. This is what happens for the command above! The output indicates the number of brokers started successfully. As soon as we get and print the size, we exit." + ] + }, + { + "cell_type": "markdown", + "id": "926bd17b-b288-4b51-b984-1478dd382954", + "metadata": { + "tags": [] + }, + "source": [ + "#### **Wrap up: All the Different Ways to Do Work (from the CLI)**\n", + "Here's a basic table that shows the four submission commands we use in Flux. \n", + "\n", + "| | creates subinstance | runs distributed application |\n", + "|------------------------|-------------------------------|---------------------------------------|\n", + "| interactive | `flux alloc` | `flux run` |\n", + "| backgrounded | `flux batch` | `flux submit`👀 |\n", + "\n", + "* `flux alloc` will allocate resources and start an interactive Flux sub-instance underneath those resources. Within that subinstance, you can submit as many jobs as you like, with no worry about backing up the parent (usually system) instance.\n", + "* `flux batch` will also allocate resources and start a Flux sub-instance, but the job is not interactive, and thus `batch` requires a script outlining the work to do.\n", + "* `flux run` runs a program under a Flux instance. It does not create a new sub-instance, and will watch until the program completes.\n", + "* `flux submit` does not exist in other resource managers, notably Slurm. It does the same thing as `flux run`, but does not watch for job output, instead writing this to a file. " + ] + }, + { + "cell_type": "markdown", + "id": "c9c3e767-0459-4218-a8cf-0f98bd32d6bf", + "metadata": {}, + "source": [ + "# This concludes Chapter 1! 📗️\n", + "\n", + "In this module, we covered:\n", + "1. Submitting jobs with Flux\n", + "2. The differences in submission commands.\n", + "\n", + "To continue with the tutorial, open [Chapter 2](../ch2/02_flux_framework.ipynb)" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.12.3" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/2026/HPCIC-AWS/tutorial/ch1/Makefile b/2026/HPCIC-AWS/tutorial/ch1/Makefile new file mode 100644 index 00000000..c3530067 --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/ch1/Makefile @@ -0,0 +1,2 @@ +make: + mpicc -o hello hello.c diff --git a/2026/HPCIC-AWS/tutorial/ch1/bulksubmit/0.sh b/2026/HPCIC-AWS/tutorial/ch1/bulksubmit/0.sh new file mode 100755 index 00000000..6570ec41 --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/ch1/bulksubmit/0.sh @@ -0,0 +1,4 @@ +#!/bin/bash + +echo "Enter, stranger, but take heed" +sleep 1 diff --git a/2026/HPCIC-AWS/tutorial/ch1/bulksubmit/1.sh b/2026/HPCIC-AWS/tutorial/ch1/bulksubmit/1.sh new file mode 100755 index 00000000..dd6c458a --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/ch1/bulksubmit/1.sh @@ -0,0 +1,4 @@ +#!/bin/bash + +echo "Of what awaits the sin of greed," +sleep 1 diff --git a/2026/HPCIC-AWS/tutorial/ch1/bulksubmit/2.sh b/2026/HPCIC-AWS/tutorial/ch1/bulksubmit/2.sh new file mode 100755 index 00000000..cfbfa217 --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/ch1/bulksubmit/2.sh @@ -0,0 +1,4 @@ +#!/bin/bash + +echo "For those who take, but do not earn," +sleep 1 diff --git a/2026/HPCIC-AWS/tutorial/ch1/bulksubmit/3.sh b/2026/HPCIC-AWS/tutorial/ch1/bulksubmit/3.sh new file mode 100755 index 00000000..c1fea0ee --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/ch1/bulksubmit/3.sh @@ -0,0 +1,4 @@ +#!/bin/bash + +echo "Must pay most dearly in their turn," +sleep 1 diff --git a/2026/HPCIC-AWS/tutorial/ch1/bulksubmit/4.sh b/2026/HPCIC-AWS/tutorial/ch1/bulksubmit/4.sh new file mode 100755 index 00000000..3359c352 --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/ch1/bulksubmit/4.sh @@ -0,0 +1,4 @@ +#!/bin/bash + +echo "So if you seek beneath our floors" +sleep 1 diff --git a/2026/HPCIC-AWS/tutorial/ch1/bulksubmit/5.sh b/2026/HPCIC-AWS/tutorial/ch1/bulksubmit/5.sh new file mode 100755 index 00000000..9a07e711 --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/ch1/bulksubmit/5.sh @@ -0,0 +1,4 @@ +#!/bin/bash + +echo "A treasure that was never yours," +sleep 1 diff --git a/2026/HPCIC-AWS/tutorial/ch1/bulksubmit/6.sh b/2026/HPCIC-AWS/tutorial/ch1/bulksubmit/6.sh new file mode 100755 index 00000000..6eabff7c --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/ch1/bulksubmit/6.sh @@ -0,0 +1,4 @@ +#!/bin/bash + +echo "Thief, you have been warned, beware" +sleep 1 diff --git a/2026/HPCIC-AWS/tutorial/ch1/bulksubmit/7.sh b/2026/HPCIC-AWS/tutorial/ch1/bulksubmit/7.sh new file mode 100755 index 00000000..20f9e2f4 --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/ch1/bulksubmit/7.sh @@ -0,0 +1,4 @@ +#!/bin/bash + +echo "Of finding more than treasure there." +sleep 1 diff --git a/2026/HPCIC-AWS/tutorial/ch1/hello-batch.sh b/2026/HPCIC-AWS/tutorial/ch1/hello-batch.sh new file mode 100755 index 00000000..3c39dc02 --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/ch1/hello-batch.sh @@ -0,0 +1,8 @@ +#!/bin/bash + +flux submit --flags=waitable -N1 --output=/tmp/hello-batch-1.out echo "Hello job 1 from $(hostname) 💛️" +flux submit --flags=waitable -N1 --output=/tmp/hello-batch-2.out echo "Hello job 2 from $(hostname) 💚️" +flux submit --flags=waitable -N1 --output=/tmp/hello-batch-3.out echo "Hello job 3 from $(hostname) 💙️" +flux submit --flags=waitable -N1 --output=/tmp/hello-batch-4.out echo "Hello job 4 from $(hostname) 💜️" +# Wait for the jobs to finish +flux job wait --all \ No newline at end of file diff --git a/2026/HPCIC-AWS/tutorial/ch1/hello.c b/2026/HPCIC-AWS/tutorial/ch1/hello.c new file mode 100644 index 00000000..b43bfaa9 --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/ch1/hello.c @@ -0,0 +1,94 @@ +/************************************************************\ + * Copyright 2014 Lawrence Livermore National Security, LLC + * (c.f. AUTHORS, NOTICE.LLNS, COPYING) + * + * This file is part of the Flux resource manager framework. + * For details, see https://github.com/flux-framework. + * + * SPDX-License-Identifier: LGPL-3.0 +\************************************************************/ + +#if HAVE_CONFIG_H +#include "config.h" +#endif +#include +#include +#include +#include +#include +#include + +static struct timespec ts_diff (struct timespec start, struct timespec end) +{ + struct timespec temp; + if ((end.tv_nsec-start.tv_nsec)<0) { + temp.tv_sec = end.tv_sec-start.tv_sec-1; + temp.tv_nsec = 1000000000+end.tv_nsec-start.tv_nsec; + } else { + temp.tv_sec = end.tv_sec-start.tv_sec; + temp.tv_nsec = end.tv_nsec-start.tv_nsec; + } + return temp; +} + +double monotime_since (struct timespec t0) +{ + struct timespec ts, d; + clock_gettime (CLOCK_MONOTONIC, &ts); + + d = ts_diff (t0, ts); + + return ((double) d.tv_sec * 1000 + (double) d.tv_nsec / 1000000); +} + +void monotime (struct timespec *tp) +{ + clock_gettime (CLOCK_MONOTONIC, tp); +} + +bool monotime_isset (struct timespec t) +{ + return (t.tv_sec || t.tv_nsec); +} + +int main (int argc, char *argv[]) +{ + int id, ntasks; + struct timespec t; + const char *label; + + if (!(label = getenv ("FLUX_JOB_CC"))) + if (!(label = getenv ("FLUX_JOB_ID"))) + label = "0"; + + monotime (&t); + MPI_Init (&argc, &argv); + MPI_Comm_rank (MPI_COMM_WORLD, &id); + MPI_Comm_size (MPI_COMM_WORLD, &ntasks); + if (id == 0) { + printf ("%s: completed MPI_Init in %0.3fs. There are %d tasks\n", + label, + monotime_since (t) / 1000, ntasks); + fflush (stdout); + } + + monotime (&t); + MPI_Barrier (MPI_COMM_WORLD); + if (id == 0) { + printf ("%s: completed first barrier in %0.3fs\n", + label, + monotime_since (t) / 1000); + fflush (stdout); + } + + monotime (&t); + MPI_Finalize (); + if (id == 0) { + printf ("%s: completed MPI_Finalize in %0.3fs\n", + label, + monotime_since (t) / 1000); + fflush (stdout); + } + return 0; +} + diff --git a/2026/HPCIC-AWS/tutorial/ch1/job-watch.sh b/2026/HPCIC-AWS/tutorial/ch1/job-watch.sh new file mode 100755 index 00000000..9b4c08d1 --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/ch1/job-watch.sh @@ -0,0 +1,11 @@ +#!/bin/bash + +echo "Oh you may not think me pretty," +sleep 3 +echo "But don’t judge on what you see," +sleep 3 +echo "I’ll eat myself if you can find" +sleep 2 +echo "A smarter hat than me." +sleep 2 +echo "[A scheduler smarter than me]!" diff --git a/2026/HPCIC-AWS/tutorial/ch1/sleep_batch.sh b/2026/HPCIC-AWS/tutorial/ch1/sleep_batch.sh new file mode 100644 index 00000000..58496dae --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/ch1/sleep_batch.sh @@ -0,0 +1,15 @@ +#!/bin/bash +#FLUX: --nodes=2 +#FLUX: --nslots=2 +#FLUX: --cores-per-slot=1 + +echo "Starting my batch job" +echo "Print the resources allocated to this batch job" +flux resource list + +echo "Use sleep to emulate a parallel program" +echo "Run the program at a total of 2 processes each requiring" +echo "1 core. These processes are equally spread across 2 nodes." +flux run -N 2 -n 2 sleep 30 +flux run -N 2 -n 2 sleep 30 + diff --git a/2026/HPCIC-AWS/tutorial/ch1/sub_job1.sh b/2026/HPCIC-AWS/tutorial/ch1/sub_job1.sh new file mode 100755 index 00000000..5cf7ff47 --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/ch1/sub_job1.sh @@ -0,0 +1,6 @@ +#!/bin/bash +#FLUX: -N1 + +flux batch -N1 ./sub_job2.sh +flux queue drain + diff --git a/2026/HPCIC-AWS/tutorial/ch1/sub_job2.sh b/2026/HPCIC-AWS/tutorial/ch1/sub_job2.sh new file mode 100755 index 00000000..d947f191 --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/ch1/sub_job2.sh @@ -0,0 +1,4 @@ +#!/bin/bash + +flux run -N1 sleep 30 + diff --git a/2026/HPCIC-AWS/tutorial/ch2/02_flux_framework.ipynb b/2026/HPCIC-AWS/tutorial/ch2/02_flux_framework.ipynb new file mode 100644 index 00000000..e393320c --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/ch2/02_flux_framework.ipynb @@ -0,0 +1,11118 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "tags": [] + }, + "source": [ + "
\n", + "
\n", + "
\n", + "\n", + "# Chapter 2: Python Submission API 🐍️\n", + "\n", + "Flux also provides first-class python bindings which can be used to submit jobs programmatically. For this chapter in the tutorial, we will be using Python and you can interact with cells in the notebook." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Importing the flux package\n", + "\n", + "Flux requires Python to build, so if you have Flux installed, you have at least one Python installation that works. However, you can also `pip install flux-python` to get the `flux` package in a side-installation of Python. The Python SDK interacts with the current Flux instance via the Flux handle. Let's show how to test that." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": { + "tags": [] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n" + ] + } + ], + "source": [ + "import flux \n", + "print(flux.Flux())" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "import os\n", + "import json\n", + "import flux\n", + "import flux.job\n", + "from flux.job import JobspecV1\n", + "from flux.job.JobID import JobID" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "tags": [] + }, + "source": [ + "### `flux.job.JobspecV1`\n", + "\n", + "
\n", + "Description: The JobspecV1 class provides an easy means to create job specifications\n", + "
\n", + "\n", + "Flux represents work as a standard called the [Jobspec](https://flux-framework.readthedocs.io/projects/flux-rfc/en/latest/spec_25.html). While you could write YAML or JSON, it's much easier to use provided Python functions that take high level metadata (command, resources, etc) to generate them. We can then replicate our previous example of submitting multiple heterogeneous jobs using these Python helpers, and testing that Flux co-schedules them." + ] + }, + { + "cell_type": "code", + "execution_count": 33, + "metadata": { + "tags": [] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "jobid.f58 = 'ƒkLsZEaNK'\n" + ] + } + ], + "source": [ + "# connect to the running Flux instance\n", + "f = flux.Flux()\n", + "\n", + "# Create the Jobspec from a command to run a python script, and specify resources\n", + "jobspec = JobspecV1.from_command(\n", + " command=[\"sleep\", \"0\"], num_tasks=1, num_nodes=1, cores_per_task=1\n", + ")\n", + "\n", + "# Attributes like environment, cwd, attributes input/output, queue and ruration can be set directly.\n", + "jobspec.environment = {'TMPDIR': '/tmp/'}\n", + "jobid = flux.job.submit(f, jobspec)\n", + "\n", + "# When we submit, we get back the job identifier (JobID)\n", + "print(f'{jobid.f58 = }')" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We get a job ID, and the command will _block_ until a job ID is assigned. But, perhaps we want more information about the job, too." + ] + }, + { + "cell_type": "code", + "execution_count": 34, + "metadata": { + "tags": [] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{\n", + " \"t_run\": 1754855108.0022883,\n", + " \"t_cleanup\": 1754855108.0174737,\n", + " \"duration\": 0.0,\n", + " \"result\": \"COMPLETED\",\n", + " \"waitstatus\": 0,\n", + " \"id\": 95699806978048,\n", + " \"t_submit\": 1754855107.9783614,\n", + " \"runtime\": 0.015185356140136719,\n", + " \"returncode\": 0,\n", + " \"dependencies\": [],\n", + " \"annotations\": {},\n", + " \"exception\": {\n", + " \"occurred\": false\n", + " }\n", + "}\n" + ] + } + ], + "source": [ + "# Maybe we want a bit more information about this job\n", + "info = flux.job.result(f, jobid)\n", + "print(json.dumps(info.to_dict(), indent=4))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Once we create the job, when we submit it in Python we get back a job identifier or jobid. We can then interact with the Flux handle, a connection to Flux, to get information about that job.\n", + "\n", + "### `flux.job.get_job(handle, jobid)`\n", + "\n", + "
\n", + "Description: The `get_job` function makes it easy to get job information like state, status, and timings\n", + "
\n" + ] + }, + { + "cell_type": "code", + "execution_count": 36, + "metadata": { + "tags": [] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "🎉️ Hooray, we just submitted ƒkLsZEaNK!\n", + "🎉️ Hooray, we just submitted 😄🎪🍜💚🚙📓!\n", + "🎉️ Hooray, we just submitted teacher-alien-grace--lima-alarm-academy!\n", + "🎉️ Hooray, we just submitted 95699806978048!\n", + "🎉️ Hooray, we just submitted 95699806978048!\n", + "🎉️ Hooray, we just submitted 1!\n", + "🎉️ Hooray, we just submitted 0x5709d9000000!\n", + "🎉️ Hooray, we just submitted 0000.5709.d900.0000!\n", + "🎉️ Hooray, we just submitted ƒkLsZEaNK!\n", + "🎉️ Hooray, we just submitted fkLsZEaNK!\n", + "\n", + "{'t_depend': 1754855107.990325, 't_run': 1754855108.0022883, 't_cleanup': 1754855108.0174737, 't_inactive': 1754855108.018468, 'duration': 0.0, 'expiration': 9223372036.0, 'name': 'sleep', 'cwd': '', 'queue': '', 'project': '', 'bank': '', 'ntasks': 1, 'ncores': 1, 'nnodes': 1, 'priority': 16, 'ranks': '3', 'nodelist': 'ip-10-0-20-251', 'success': True, 'result': 'COMPLETED', 'waitstatus': 0, 'id': JobID(95699806978048), 't_submit': 1754855107.9783614, 't_remaining': 0.0, 'state': 'INACTIVE', 'username': 'ubuntu', 'userid': 1000, 'urgency': 16, 'runtime': 0.015185356140136719, 'status': 'COMPLETED', 'returncode': 0, 'dependencies': [], 'annotations': {}, 'exception': {'occurred': False, 'severity': '', 'type': '', 'note': ''}}\n" + ] + } + ], + "source": [ + "# Let's get a flux.job.JobID Python object for our job.\n", + "fid = JobID(jobid.f58)\n", + "\n", + "# Why? We can represent it in many ways!\n", + "print(f\"🎉️ Hooray, we just submitted {fid}!\")\n", + "print(f\"🎉️ Hooray, we just submitted {fid.emoji}!\")\n", + "print(f\"🎉️ Hooray, we just submitted {fid.words}!\")\n", + "print(f\"🎉️ Hooray, we just submitted {fid.real}!\")\n", + "print(f\"🎉️ Hooray, we just submitted {fid.dec}!\")\n", + "print(f\"🎉️ Hooray, we just submitted {fid.denominator}!\")\n", + "print(f\"🎉️ Hooray, we just submitted {fid.hex}!\")\n", + "print(f\"🎉️ Hooray, we just submitted {fid.dothex}!\")\n", + "print(f\"🎉️ Hooray, we just submitted {fid.f58}!\")\n", + "print(f\"🎉️ Hooray, we just submitted {fid.f58plain}!\\n\")\n", + "\n", + "# Here is how to get your info. The first argument is the flux handle, then the jobid\n", + "jobinfo = flux.job.get_job(f, fid)\n", + "print(jobinfo)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Look at what came from `.get_job()`. It didn't block, but the job isn't complete yet -- that `t_run` is 0, which makes no sense, because the run time should be around 5 seconds. Notice the status is also 'SCHED', meaning that the job is being scheduled, " + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "You can now run `flux jobs` to see the jobs that we submit from Python." + ] + }, + { + "cell_type": "code", + "execution_count": 37, + "metadata": { + "scrolled": true, + "tags": [] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + " JOBID USER NAME ST NTASKS NNODES TIME INFO\n", + "\u001b[01;32m ƒkLsZEaNK ubuntu sleep CD 1 1 0.015s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn6D4FZ ubuntu sleep CD 1 1 0.465s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn4j4y8 ubuntu sleep CD 1 1 0.466s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn3F5h4 ubuntu sleep CD 1 1 0.467s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn4j4yE ubuntu sleep CD 1 1 0.466s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn4j4yQ ubuntu sleep CD 1 1 0.466s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn3F5h8 ubuntu sleep CD 1 1 0.467s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn1m6R4 ubuntu sleep CD 1 1 0.467s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn4j4yL ubuntu sleep CD 1 1 0.466s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn6D4FR ubuntu sleep CD 1 1 0.466s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn6D4Fc ubuntu sleep CD 1 1 0.465s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn6D4Fh ubuntu sleep CD 1 1 0.461s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn3F5gr ubuntu sleep CD 1 1 0.467s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn6D4FV ubuntu sleep CD 1 1 0.465s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn3F5gn ubuntu sleep CD 1 1 0.463s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn3F5gz ubuntu sleep CD 1 1 0.462s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn6D4FX ubuntu sleep CD 1 1 0.461s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn4j4yA ubuntu sleep CD 1 1 0.461s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn1m6Qt ubuntu sleep CD 1 1 0.462s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn4j4y7 ubuntu sleep CD 1 1 0.463s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn3F5h3 ubuntu sleep CD 1 1 0.464s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn4j4yF ubuntu sleep CD 1 1 0.463s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn4j4yN ubuntu sleep CD 1 1 0.463s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn1m6R2 ubuntu sleep CD 1 1 0.464s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn3F5hA ubuntu sleep CD 1 1 0.461s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn3F5gk ubuntu sleep CD 1 1 0.467s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn1m6Qx ubuntu sleep CD 1 1 0.464s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn4j4yJ ubuntu sleep CD 1 1 0.464s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn4j4yS ubuntu sleep CD 1 1 0.462s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn3F5h7 ubuntu sleep CD 1 1 0.462s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn6D4FT ubuntu sleep CD 1 1 0.462s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn3F5gy ubuntu sleep CD 1 1 0.467s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn3F5gu ubuntu sleep CD 1 1 0.462s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn3F5gp ubuntu sleep CD 1 1 0.462s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn6D4Fd ubuntu sleep CD 1 1 0.461s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn6D4Ff ubuntu sleep CD 1 1 0.462s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn6D4Fg ubuntu sleep CD 1 1 0.463s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn1m6Qy ubuntu sleep CD 1 1 0.461s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn4j4yB ubuntu sleep CD 1 1 0.459s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn4j4yG ubuntu sleep CD 1 1 0.461s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn6D4FU ubuntu sleep CD 1 1 0.463s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn3F5gt ubuntu sleep CD 1 1 0.460s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn6D4FY ubuntu sleep CD 1 1 0.461s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn6D4Fb ubuntu sleep CD 1 1 0.464s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn1m6Qu ubuntu sleep CD 1 1 0.461s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn4j4y6 ubuntu sleep CD 1 1 0.462s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn4j4yT ubuntu sleep CD 1 1 0.465s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn3F5h6 ubuntu sleep CD 1 1 0.466s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn1m6Qq ubuntu sleep CD 1 1 0.458s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn3F5gx ubuntu sleep CD 1 1 0.466s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn4j4yK ubuntu sleep CD 1 1 0.460s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn3F5gq ubuntu sleep CD 1 1 0.467s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn3F5h2 ubuntu sleep CD 1 1 0.463s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn4j4yP ubuntu sleep CD 1 1 0.461s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn3F5gm ubuntu sleep CD 1 1 0.462s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn1m6R3 ubuntu sleep CD 1 1 0.461s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn3F5hB ubuntu sleep CD 1 1 0.459s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn3F5gv ubuntu sleep CD 1 1 0.455s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn4j4yC ubuntu sleep CD 1 1 0.453s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn3F5hC ubuntu sleep CD 1 1 0.458s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn1m6Qr ubuntu sleep CD 1 1 0.455s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn1m6Qz ubuntu sleep CD 1 1 0.455s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn1m6Qv ubuntu sleep CD 1 1 0.455s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn1m6Qh ubuntu sleep CD 1 1 0.442s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn1m6Qm ubuntu sleep CD 1 1 0.442s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn1m6Qe ubuntu sleep CD 1 1 0.442s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn1m6Qi ubuntu sleep CD 1 1 0.441s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn1m6Qn ubuntu sleep CD 1 1 0.441s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn1m6Qd ubuntu sleep CD 1 1 0.441s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn1m6Qk ubuntu sleep CD 1 1 0.425s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn1m6Qc ubuntu sleep CD 1 1 0.426s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn1m6Qp ubuntu sleep CD 1 1 0.424s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn1m6Qg ubuntu sleep CD 1 1 0.425s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn1m6Qa ubuntu sleep CD 1 1 0.410s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn1m6QW ubuntu sleep CD 1 1 0.410s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn1m6QS ubuntu sleep CD 1 1 0.407s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmzH78L ubuntu sleep CD 1 1 0.407s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmzH78g ubuntu sleep CD 1 1 0.408s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn1m6QQ ubuntu sleep CD 1 1 0.408s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn1m6QU ubuntu sleep CD 1 1 0.407s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn1m6QY ubuntu sleep CD 1 1 0.407s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmzH78K ubuntu sleep CD 1 1 0.409s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmzH78e ubuntu sleep CD 1 1 0.408s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmzH78c ubuntu sleep CD 1 1 0.409s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmxo7qz ubuntu sleep CD 1 1 0.410s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmzH78Y ubuntu sleep CD 1 1 0.409s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmzH78U ubuntu sleep CD 1 1 0.410s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmzH788 ubuntu sleep CD 1 1 0.410s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmzH78Q ubuntu sleep CD 1 1 0.409s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmzH784 ubuntu sleep CD 1 1 0.409s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn1m6QV ubuntu sleep CD 1 1 0.408s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmzH78X ubuntu sleep CD 1 1 0.407s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmzH78C ubuntu sleep CD 1 1 0.409s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmzH78J ubuntu sleep CD 1 1 0.408s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmzH78T ubuntu sleep CD 1 1 0.408s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmzH78G ubuntu sleep CD 1 1 0.409s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn1m6QR ubuntu sleep CD 1 1 0.407s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn1m6QZ ubuntu sleep CD 1 1 0.406s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmzH78f ubuntu sleep CD 1 1 0.407s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmzH78b ubuntu sleep CD 1 1 0.408s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmzH78P ubuntu sleep CD 1 1 0.408s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmxo7r4 ubuntu sleep CD 1 1 0.411s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmzH78a ubuntu sleep CD 1 1 0.407s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmxo7qv ubuntu sleep CD 1 1 0.409s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmxo7qr ubuntu sleep CD 1 1 0.412s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmzH78S ubuntu sleep CD 1 1 0.406s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmzH78W ubuntu sleep CD 1 1 0.403s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmzH78A ubuntu sleep CD 1 1 0.395s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmzH78B ubuntu sleep CD 1 1 0.394s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmzH78F ubuntu sleep CD 1 1 0.394s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmzH78N ubuntu sleep CD 1 1 0.393s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmzH78E ubuntu sleep CD 1 1 0.394s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmzH786 ubuntu sleep CD 1 1 0.394s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmxo7r3 ubuntu sleep CD 1 1 0.394s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmzH783 ubuntu sleep CD 1 1 0.395s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmzH787 ubuntu sleep CD 1 1 0.395s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmxo7r2 ubuntu sleep CD 1 1 0.395s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmxo7r6 ubuntu sleep CD 1 1 0.394s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmxo7qy ubuntu sleep CD 1 1 0.391s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmxo7qn ubuntu sleep CD 1 1 0.390s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmxo7qx ubuntu sleep CD 1 1 0.390s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmxo7qi ubuntu sleep CD 1 1 0.390s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmwK8ZY ubuntu sleep CD 1 1 0.389s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmwK8ZQ ubuntu sleep CD 1 1 0.390s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmwK8Zc ubuntu sleep CD 1 1 0.389s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmwK8ZP ubuntu sleep CD 1 1 0.389s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmwK8ZN ubuntu sleep CD 1 1 0.386s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmwK8ZM ubuntu sleep CD 1 1 0.384s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmuq9H9 ubuntu sleep CD 1 1 0.384s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmxo7qm ubuntu sleep CD 1 1 0.379s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmxo7qu ubuntu sleep CD 1 1 0.379s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmxo7qq ubuntu sleep CD 1 1 0.379s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmxo7qh ubuntu sleep CD 1 1 0.379s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmwK8Zb ubuntu sleep CD 1 1 0.379s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmuq9H6 ubuntu sleep CD 1 1 0.374s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmuq9H8 ubuntu sleep CD 1 1 0.379s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmuq9H7 ubuntu sleep CD 1 1 0.373s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmwK8ZX ubuntu sleep CD 1 1 0.372s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn6D4Fi ubuntu sleep CD 1 1 0.365s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmuq9H5 ubuntu sleep CD 1 1 0.372s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmuq9H4 ubuntu sleep CD 1 1 0.372s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn6D4FS ubuntu sleep CD 1 1 0.363s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn6D4Fe ubuntu sleep CD 1 1 0.362s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn6D4Fa ubuntu sleep CD 1 1 0.362s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn6D4FW ubuntu sleep CD 1 1 0.362s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn4j4yR ubuntu sleep CD 1 1 0.362s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn4j4yM ubuntu sleep CD 1 1 0.362s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn4j4yH ubuntu sleep CD 1 1 0.362s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn4j4yD ubuntu sleep CD 1 1 0.357s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn4j4y9 ubuntu sleep CD 1 1 0.347s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn4j4y5 ubuntu sleep CD 1 1 0.347s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn3F5h9 ubuntu sleep CD 1 1 0.347s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn3F5h5 ubuntu sleep CD 1 1 0.346s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn3F5h1 ubuntu sleep CD 1 1 0.345s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn3F5gw ubuntu sleep CD 1 1 0.344s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn3F5gs ubuntu sleep CD 1 1 0.344s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn3F5go ubuntu sleep CD 1 1 0.343s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn3F5gj ubuntu sleep CD 1 1 0.343s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn1m6R1 ubuntu sleep CD 1 1 0.339s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn1m6Qw ubuntu sleep CD 1 1 0.339s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn1m6Qs ubuntu sleep CD 1 1 0.338s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn1m6Qf ubuntu sleep CD 1 1 0.338s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn1m6Qo ubuntu sleep CD 1 1 0.338s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn1m6Qj ubuntu sleep CD 1 1 0.338s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmxo7qt ubuntu sleep CD 1 1 0.335s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmxo7qp ubuntu sleep CD 1 1 0.335s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmxo7qk ubuntu sleep CD 1 1 0.335s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn1m6QP ubuntu sleep CD 1 1 0.332s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn1m6QT ubuntu sleep CD 1 1 0.332s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn1m6Qb ubuntu sleep CD 1 1 0.331s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMn1m6QX ubuntu sleep CD 1 1 0.330s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmzH78d ubuntu sleep CD 1 1 0.330s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmzH78Z ubuntu sleep CD 1 1 0.329s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmzH78M ubuntu sleep CD 1 1 0.325s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmzH78V ubuntu sleep CD 1 1 0.325s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmzH78R ubuntu sleep CD 1 1 0.326s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmzH78H ubuntu sleep CD 1 1 0.326s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmzH78D ubuntu sleep CD 1 1 0.325s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmwK8Ze ubuntu sleep CD 1 1 0.322s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmzH789 ubuntu sleep CD 1 1 0.320s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmwK8Za ubuntu sleep CD 1 1 0.321s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmwK8ZW ubuntu sleep CD 1 1 0.320s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmuq9H3 ubuntu sleep CD 1 1 0.320s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmuq9H2 ubuntu sleep CD 1 1 0.320s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmuq9H1 ubuntu sleep CD 1 1 0.318s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmwK8ZR ubuntu sleep CD 1 1 0.313s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmzH785 ubuntu sleep CD 1 1 0.310s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmxo7r5 ubuntu sleep CD 1 1 0.311s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmxo7r1 ubuntu sleep CD 1 1 0.311s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmxo7qw ubuntu sleep CD 1 1 0.306s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmxo7qs ubuntu sleep CD 1 1 0.305s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmxo7qo ubuntu sleep CD 1 1 0.304s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmxo7qj ubuntu sleep CD 1 1 0.303s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmwK8Zd ubuntu sleep CD 1 1 0.304s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmwK8ZZ ubuntu sleep CD 1 1 0.303s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmwK8ZV ubuntu sleep CD 1 1 0.302s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmwK8ZU ubuntu sleep CD 1 1 0.301s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmwK8ZT ubuntu sleep CD 1 1 0.300s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmwK8ZS ubuntu sleep CD 1 1 0.300s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmtM9zg ubuntu sleep CD 1 1 0.298s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZMmtM9zf ubuntu sleep CD 1 1 0.326s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZDsboAUK ubuntu sleep CD 1 1 3.025s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZDsboAUJ ubuntu sleep CD 1 1 3.025s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZDsaKBBx ubuntu sleep CD 1 1 3.023s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZDsboAUG ubuntu sleep CD 1 1 3.023s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZDsaKBBu ubuntu sleep CD 1 1 3.024s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZDsboAUF ubuntu sleep CD 1 1 3.021s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZDsaKBBv ubuntu sleep CD 1 1 3.022s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZDsboAUH ubuntu sleep CD 1 1 3.020s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZDsaKBBw ubuntu sleep CD 1 1 3.020s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒZDsaKBBy ubuntu sleep CD 1 1 3.019s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒYzPRv5AX ubuntu sleep CD 4 1 5.018s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒYnSsqZsm ubuntu sleep CD 4 1 5.018s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒW6kYN9qh ubuntu sleep CD 1 1 0.022s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒUMuZwaqD ubuntu sleep CD 1 1 0.015s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒUBoc5SgP ubuntu sleep CD 1 1 0.015s ip-10-0-20-251\n", + "\u001b[0;0m\u001b[01;32m ƒSk9Be7Vh ubuntu sleep CD 1 1 0.425s ip-10-0-20-251\n", + "\u001b[0;0m" + ] + } + ], + "source": [ + "!flux jobs -a --name=\"sleep\"" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Under the hood, the `Jobspec` class is creating a YAML document that ultimately gets serialized as JSON and sent to Flux for ingestion, validation, queueing, scheduling, and eventually execution. We can dump the raw JSON jobspec that is submitted, where we can see the exact resources requested and the task set to be executed on those resources." + ] + }, + { + "cell_type": "code", + "execution_count": 38, + "metadata": { + "scrolled": true, + "tags": [] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{\n", + " \"resources\": [\n", + " {\n", + " \"type\": \"node\",\n", + " \"count\": 1,\n", + " \"with\": [\n", + " {\n", + " \"type\": \"slot\",\n", + " \"count\": 1,\n", + " \"with\": [\n", + " {\n", + " \"type\": \"core\",\n", + " \"count\": 1\n", + " }\n", + " ],\n", + " \"label\": \"task\"\n", + " }\n", + " ]\n", + " }\n", + " ],\n", + " \"tasks\": [\n", + " {\n", + " \"command\": [\n", + " \"sleep\",\n", + " \"0\"\n", + " ],\n", + " \"slot\": \"task\",\n", + " \"count\": {\n", + " \"per_slot\": 1\n", + " }\n", + " }\n", + " ],\n", + " \"attributes\": {\n", + " \"system\": {\n", + " \"duration\": 0,\n", + " \"environment\": {\n", + " \"TMPDIR\": \"/tmp/\"\n", + " }\n", + " }\n", + " },\n", + " \"version\": 1\n", + "}\n" + ] + } + ], + "source": [ + "print(jobspec.dumps(indent=4))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Playing with the Synchronous job submission API\n", + "\n", + "One slight hiccup to creating jobspecs in Python is that some attributes of a jobspec are set in the _initializer_ (`.from_command` or another method) while others are specified by methods to the jobspec itself. Below is an example, and here's a brief table for reference:\n", + "\n", + "| Operation | CLI | Python |\n", + "|------------------------------------------------|---------------------------------------------------------------------------|--------------------------------------------------------------------------------------------|\n", + "| Setting nodes, tasks, and cores | -N _n_, -n _n_, and -c _n_, respectively | in the initializer: num_nodes, num_tasks, num_cores |\n", + "| Setting node exclusivity | `-x` or enforced by policy | in the initializer: exclusive=True |\n", + "| Setting time limits | `-t MINUTES\\|FSD` | `jobspec.duration=[seconds\\|FSD]` |\n", + "| Set working directory | Automatically set to current directory or `--cwd=` | `jobspec.cwd=[string of path]` |\n", + "| Set environment | Default set to current environment, or `--env=[modifier]` | `jobspec.environment=[dict]` |\n", + "| Setting stdout and stderr | `--output [OUT] --error [ERR]` or your terminal by default for alloc/run | `jobspec.stdout = ` or `jobspec.stderr = ` |\n", + "| Setting system attributes | `-S KEY[=VAL]` | `jobspec.setattr(key, val)` |\n", + "| Set shell attributes | `-o KEY[=VAL]` | `jobspec.setattr_shell_option(key, val)` |\n", + "| Modifying configuration of a subinstance | `--conf` for batch/alloc | in the initializer, requires `.from_batch_command` or `.from_nest_command` |\n", + "\n", + "The example below is taken from the [El Cap documentation maintained by Ramesh Pankajakshan](https://hpc.llnl.gov/documentation/user-guides/using-el-capitan-systems/introduction-and-quickstart/flux)." + ] + }, + { + "cell_type": "code", + "execution_count": 39, + "metadata": { + "tags": [] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "1754855213.80323: finish {'status': 0}\n" + ] + } + ], + "source": [ + "# Create a Flux handle (discovers the FLUX_URI in the environment)\n", + "handle = flux.Flux()\n", + "jobspec = JobspecV1.from_command(\n", + " command=[\"sleep\", \"5\"], num_tasks=4, num_nodes=1,\n", + ")\n", + "jobspec.cwd = os.getcwd()\n", + "jobspec.exclusive=1\n", + "jobspec.duration=\"5m\"\n", + "jobspec.environment = {\"TMPDIR\": \"/tmp/\"}\n", + "jobspec.stdout=\"PYEXAMPLE.{{id}}.out\"\n", + "jobspec.stderr=\"PYEXAMPLE.{{id}}.err\"\n", + "id = flux.job.submit(handle, jobspec)\n", + "\n", + "info = flux.job.event_wait(handle, id, \"finish\")\n", + "print(info)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### `FluxExecutor`\n", + "\n", + "
\n", + "Description: The `FluxExector` class makes it easy to do bulk submission in Python\n", + "
\n", + "\n", + "We can use the FluxExecutor class to submit large numbers of jobs to Flux. This method resembles python's `concurrent.futures` interface." + ] + }, + { + "cell_type": "code", + "execution_count": 40, + "metadata": { + "tags": [] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "I'm all done\n" + ] + } + ], + "source": [ + "from flux.job import FluxExecutor\n", + "\n", + "with FluxExecutor() as executor:\n", + " jobspec = JobspecV1.from_command([\"sleep\", \"3\"])\n", + " jobspec.environment = {\"TMPDIR\": \"/tmp/\"}\n", + " futures = [executor.submit(jobspec) for _ in range(10)]\n", + " # wait for the jobid for each job, as a proxy for the job being submitted\n", + "\n", + "# all jobs submitted - print timings\n", + "print(\"I'm all done\")" + ] + }, + { + "cell_type": "code", + "execution_count": 41, + "metadata": { + "tags": [] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "bulksubmit_executor: submitted 200 jobs in 0.18s. 1118.31job/s\n", + "bulksubmit_executor: First job finished in about 0.381s\n", + "|██████████████████████████████████████████████████████████| 100.0% (310.2 job/s)\n", + "bulksubmit_executor: Ran 200 jobs in 0.8s. 249.6 job/s\n" + ] + } + ], + "source": [ + "# Submit the FluxExecutor based script.\n", + "!flux python bulksubmit_executor.py -n200 /bin/sleep 0" + ] + }, + { + "cell_type": "code", + "execution_count": 42, + "metadata": { + "tags": [] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "0\n", + "0\n", + "0\n", + "0\n", + "0\n" + ] + } + ], + "source": [ + "# Here is how to use concurrent futures\n", + "import concurrent.futures\n", + "\n", + "jobspec = flux.job.JobspecV1.from_command([\"/bin/true\"])\n", + "jobspec.environment = {\"TMPDIR\": \"/tmp/\"}\n", + "with flux.job.FluxExecutor() as executor:\n", + " futures = [executor.submit(jobspec) for _ in range(5)]\n", + " for f in concurrent.futures.as_completed(futures):\n", + " print(f.result())" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### `flux.event_watch` \n", + "\n", + "
\n", + "Description: The `flux.job.event_watch` function makes it easy to watch events for a job\n", + "
\n", + "\n", + "If you want to get the output of a job (or more generally, stream events) you can do that as follows. Let's submit a quick job, and then look at the output.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": { + "tags": [] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "ƒabUbzYD5\n", + "1754853825.08733: header {'version': 1, 'encoding': {'stdout': 'UTF-8', 'stderr': 'UTF-8'}, 'count': {'stdout': 1, 'stderr': 1}, 'options': {}}\n", + "1754853825.09276: data {'stream': 'stderr', 'rank': '0', 'eof': True}\n", + "1754853825.09278: data {'stream': 'stdout', 'rank': '0', 'data': 'I have two Dobermans, Rolex and timex. They are watch dogs.\\n'}\n", + "1754853825.09279: data {'stream': 'stdout', 'rank': '0', 'eof': True}\n" + ] + } + ], + "source": [ + "# Create the Jobspec from a command to run a python script, and specify resources\n", + "f = flux.Flux()\n", + "\n", + "joke = \"I have two Dobermans, Rolex and Timex. They are watch dogs.\"\n", + "jobspec = flux.job.JobspecV1.from_command(command=[\"echo\", joke], num_tasks=1, num_nodes=1, cores_per_task=1)\n", + "jobspec.environment = {\"TMPDIR\": \"/tmp/\"}\n", + "jobid = flux.job.submit(f, jobspec, waitable=True)\n", + "\n", + "# Wait until the job finishes\n", + "flux.job.wait(f, jobid)\n", + "print(jobid)\n", + "\n", + "# Wait on an event to complete and then print its associated data\n", + "for line in flux.job.event_watch(f, jobid, \"guest.output\"):\n", + " print(line)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### `flux.job.JobOutputWatch`\n", + "\n", + "
\n", + "Description: Synchronously or asynchronously watch for job output with asynchronous job submission\n", + "
\n", + "\n", + "#### Watch for job output with a Sychronous job submission" + ] + }, + { + "cell_type": "code", + "execution_count": 44, + "metadata": { + "tags": [] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "['stdout', 'The inventor of the throat lozenges passed away. There was no coffin at his funeral.']\n" + ] + } + ], + "source": [ + "# Create the Jobspec from a command to run a python script, and specify resources\n", + "f = flux.Flux()\n", + "\n", + "joke = \"The inventor of the throat lozenges passed away. There was no coffin at his funeral.\"\n", + "jobspec = JobspecV1.from_command(command=[\"echo\", joke], num_tasks=4, num_nodes=1)\n", + "jobspec.environment = {\"TMPDIR\": \"/tmp/\"}\n", + "jobid = flux.job.submit(f, jobspec)\n", + "\n", + "t = flux.job.output.JobOutputWatchLines(f, jobid).getline()\n", + "print(t)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### Watch for job output with asynchronous job submission" + ] + }, + { + "cell_type": "code", + "execution_count": 45, + "metadata": { + "tags": [] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "ƒnrM71A1V: stdout\n", + "ƒnrM71A1V: Wingardium Leviosa! ✨\n", + "ƒnrM71A1W: stdout\n", + "ƒnrM71A1W: Wingardium Leviosa! ✨\n", + "ƒnrM71A1X: stdout\n", + "ƒnrM71A1X: Wingardium Leviosa! ✨\n" + ] + } + ], + "source": [ + "import concurrent.futures\n", + "import flux.job\n", + "\n", + "## Define something we want to happen when the futures are fulfilled\n", + "def print_output(fut):\n", + " t = flux.job.output.JobOutputWatchLines(f, fut.jobid()).getline()\n", + " for line in t:\n", + " print(f'{fut.jobid()}: {line}')\n", + "\n", + "## Submit all of the futures using the executor.\n", + "jobspec = flux.job.JobspecV1.from_command([\"echo\", \"Wingardium Leviosa! ✨\"])\n", + "jobspec.environment = {\"TMPDIR\": \"/tmp/\"}\n", + "\n", + "with flux.job.FluxExecutor() as executor:\n", + " futures = [executor.submit(jobspec) for _ in range(3)]\n", + " for future in futures:\n", + " future.add_done_callback(print_output)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### `flux.job.job_list`\n", + "\n", + "
\n", + "Description: Get an entire listing of jobs via remote procedure call (rpc)\n", + "
\n", + "\n", + "\n", + "Finally, it can be really helpful to get an entire listing of jobs. You can do that as follows. Note that the `job_list` is creating a remote procedure call (rpc) and we call `get` to retrieve the output." + ] + }, + { + "cell_type": "code", + "execution_count": 46, + "metadata": { + "scrolled": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'jobs': [{'id': 101237848539138,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855438.071756,\n", + " 't_depend': 1754855438.0827703,\n", + " 't_run': 1754855438.0959415,\n", + " 't_cleanup': 1754855438.1109397,\n", + " 't_inactive': 1754855438.1127431,\n", + " 'state': 64,\n", + " 'name': 'echo',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 101237848539136,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855438.0714943,\n", + " 't_depend': 1754855438.0826428,\n", + " 't_run': 1754855438.0956306,\n", + " 't_cleanup': 1754855438.1106298,\n", + " 't_inactive': 1754855438.1125581,\n", + " 'state': 64,\n", + " 'name': 'echo',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 101237848539137,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855438.0716355,\n", + " 't_depend': 1754855438.082703,\n", + " 't_run': 1754855438.0958874,\n", + " 't_cleanup': 1754855438.1103518,\n", + " 't_inactive': 1754855438.1120129,\n", + " 'state': 64,\n", + " 'name': 'echo',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 101163156373504,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855433.6192713,\n", + " 't_depend': 1754855433.6312466,\n", + " 't_run': 1754855433.6432583,\n", + " 't_cleanup': 1754855433.6607547,\n", + " 't_inactive': 1754855433.6617224,\n", + " 'state': 64,\n", + " 'name': 'echo',\n", + " 'ntasks': 4,\n", + " 'ncores': 4,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 99408930668544,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855329.059233,\n", + " 't_depend': 1754855329.0704498,\n", + " 't_run': 1754855329.0825646,\n", + " 't_cleanup': 1754855329.1000926,\n", + " 't_inactive': 1754855329.1010785,\n", + " 'state': 64,\n", + " 'name': 'echo',\n", + " 'ntasks': 4,\n", + " 'ncores': 4,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98143509151748,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855253.6345878,\n", + " 't_depend': 1754855253.6461256,\n", + " 't_run': 1754855253.660246,\n", + " 't_cleanup': 1754855253.6793222,\n", + " 't_inactive': 1754855253.6816137,\n", + " 'state': 64,\n", + " 'name': 'true',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98143509151747,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855253.6345646,\n", + " 't_depend': 1754855253.6460786,\n", + " 't_run': 1754855253.6601963,\n", + " 't_cleanup': 1754855253.6790485,\n", + " 't_inactive': 1754855253.681434,\n", + " 'state': 64,\n", + " 'name': 'true',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98143509151746,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855253.6345265,\n", + " 't_depend': 1754855253.6460218,\n", + " 't_run': 1754855253.660003,\n", + " 't_cleanup': 1754855253.6789882,\n", + " 't_inactive': 1754855253.6813636,\n", + " 'state': 64,\n", + " 'name': 'true',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98143509151744,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855253.6344764,\n", + " 't_depend': 1754855253.6459258,\n", + " 't_run': 1754855253.65934,\n", + " 't_cleanup': 1754855253.6786776,\n", + " 't_inactive': 1754855253.6808107,\n", + " 'state': 64,\n", + " 'name': 'true',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98143509151745,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855253.634511,\n", + " 't_depend': 1754855253.6459749,\n", + " 't_run': 1754855253.659956,\n", + " 't_cleanup': 1754855253.6783955,\n", + " 't_inactive': 1754855253.6803267,\n", + " 'state': 64,\n", + " 'name': 'true',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129651171341,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8090084,\n", + " 't_depend': 1754855252.8513622,\n", + " 't_run': 1754855252.927338,\n", + " 't_cleanup': 1754855253.3222594,\n", + " 't_inactive': 1754855253.3544333,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129667948564,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.810059,\n", + " 't_depend': 1754855252.852415,\n", + " 't_run': 1754855252.9282265,\n", + " 't_cleanup': 1754855253.322006,\n", + " 't_inactive': 1754855253.3543642,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129667948562,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8099704,\n", + " 't_depend': 1754855252.8523123,\n", + " 't_run': 1754855252.9281487,\n", + " 't_cleanup': 1754855253.3218975,\n", + " 't_inactive': 1754855253.3542945,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129667948559,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8098493,\n", + " 't_depend': 1754855252.8521597,\n", + " 't_run': 1754855252.9280121,\n", + " 't_cleanup': 1754855253.320927,\n", + " 't_inactive': 1754855253.3542242,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129667948563,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8100061,\n", + " 't_depend': 1754855252.8523674,\n", + " 't_run': 1754855252.9281883,\n", + " 't_cleanup': 1754855253.3216183,\n", + " 't_inactive': 1754855253.3541458,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129667948560,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8098881,\n", + " 't_depend': 1754855252.8522198,\n", + " 't_run': 1754855252.9280505,\n", + " 't_cleanup': 1754855253.3212395,\n", + " 't_inactive': 1754855253.3540676,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129667948554,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8095927,\n", + " 't_depend': 1754855252.8519268,\n", + " 't_run': 1754855252.9278166,\n", + " 't_cleanup': 1754855253.3208404,\n", + " 't_inactive': 1754855253.3539984,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129667948558,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8097653,\n", + " 't_depend': 1754855252.852115,\n", + " 't_run': 1754855252.927973,\n", + " 't_cleanup': 1754855253.3207455,\n", + " 't_inactive': 1754855253.35393,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129651171330,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8084073,\n", + " 't_depend': 1754855252.8508174,\n", + " 't_run': 1754855252.9268658,\n", + " 't_cleanup': 1754855253.3207645,\n", + " 't_inactive': 1754855253.3538585,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129651171342,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8090484,\n", + " 't_depend': 1754855252.8514109,\n", + " 't_run': 1754855252.9273791,\n", + " 't_cleanup': 1754855253.3199394,\n", + " 't_inactive': 1754855253.3537881,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129667948556,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8096888,\n", + " 't_depend': 1754855252.8520167,\n", + " 't_run': 1754855252.9278948,\n", + " 't_cleanup': 1754855253.3208215,\n", + " 't_inactive': 1754855253.35372,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129651171336,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8087206,\n", + " 't_depend': 1754855252.8511162,\n", + " 't_run': 1754855252.9271317,\n", + " 't_cleanup': 1754855253.3207831,\n", + " 't_inactive': 1754855253.3536408,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129667948557,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8097198,\n", + " 't_depend': 1754855252.8520606,\n", + " 't_run': 1754855252.9279342,\n", + " 't_cleanup': 1754855253.3208022,\n", + " 't_inactive': 1754855253.353571,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129667948553,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8095179,\n", + " 't_depend': 1754855252.8518827,\n", + " 't_run': 1754855252.927778,\n", + " 't_cleanup': 1754855253.3204534,\n", + " 't_inactive': 1754855253.3535023,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129667948555,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8096404,\n", + " 't_depend': 1754855252.8519716,\n", + " 't_run': 1754855252.9278562,\n", + " 't_cleanup': 1754855253.319474,\n", + " 't_inactive': 1754855253.35343,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129667948551,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8094366,\n", + " 't_depend': 1754855252.8517907,\n", + " 't_run': 1754855252.9276981,\n", + " 't_cleanup': 1754855253.319239,\n", + " 't_inactive': 1754855253.3533547,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129651171333,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8085403,\n", + " 't_depend': 1754855252.8509588,\n", + " 't_run': 1754855252.9269922,\n", + " 't_cleanup': 1754855253.3197312,\n", + " 't_inactive': 1754855253.353283,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129667948552,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8094866,\n", + " 't_depend': 1754855252.851835,\n", + " 't_run': 1754855252.9277377,\n", + " 't_cleanup': 1754855253.3198478,\n", + " 't_inactive': 1754855253.3532095,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129651171332,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8084738,\n", + " 't_depend': 1754855252.8509119,\n", + " 't_run': 1754855252.9269507,\n", + " 't_cleanup': 1754855253.3198292,\n", + " 't_inactive': 1754855253.3531342,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129667948550,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8094018,\n", + " 't_depend': 1754855252.8517435,\n", + " 't_run': 1754855252.9276588,\n", + " 't_cleanup': 1754855253.3197129,\n", + " 't_inactive': 1754855253.3530467,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129667948546,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8092358,\n", + " 't_depend': 1754855252.8515499,\n", + " 't_run': 1754855252.9275012,\n", + " 't_cleanup': 1754855253.318416,\n", + " 't_inactive': 1754855253.35297,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129667948547,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.809272,\n", + " 't_depend': 1754855252.851594,\n", + " 't_run': 1754855252.927541,\n", + " 't_cleanup': 1754855253.317373,\n", + " 't_inactive': 1754855253.3529015,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129667948548,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8093126,\n", + " 't_depend': 1754855252.8516397,\n", + " 't_run': 1754855252.92758,\n", + " 't_cleanup': 1754855253.3163702,\n", + " 't_inactive': 1754855253.3528337,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129667948544,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8091006,\n", + " 't_depend': 1754855252.8514562,\n", + " 't_run': 1754855252.9274194,\n", + " 't_cleanup': 1754855253.3160744,\n", + " 't_inactive': 1754855253.3527672,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129667948561,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8099291,\n", + " 't_depend': 1754855252.8522666,\n", + " 't_run': 1754855252.928108,\n", + " 't_cleanup': 1754855253.321709,\n", + " 't_inactive': 1754855253.3526988,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129651171337,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8087633,\n", + " 't_depend': 1754855252.8511677,\n", + " 't_run': 1754855252.9271743,\n", + " 't_cleanup': 1754855253.3161633,\n", + " 't_inactive': 1754855253.3526273,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129651171338,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8088846,\n", + " 't_depend': 1754855252.8512182,\n", + " 't_run': 1754855252.9272156,\n", + " 't_cleanup': 1754855253.315454,\n", + " 't_inactive': 1754855253.35256,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129667948545,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8091528,\n", + " 't_depend': 1754855252.8515007,\n", + " 't_run': 1754855252.9274614,\n", + " 't_cleanup': 1754855253.3155994,\n", + " 't_inactive': 1754855253.3524933,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129667948549,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8093534,\n", + " 't_depend': 1754855252.8516963,\n", + " 't_run': 1754855252.9276195,\n", + " 't_cleanup': 1754855253.3155348,\n", + " 't_inactive': 1754855253.3524265,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129651171340,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8089716,\n", + " 't_depend': 1754855252.851316,\n", + " 't_run': 1754855252.9272974,\n", + " 't_cleanup': 1754855253.315657,\n", + " 't_inactive': 1754855253.3523579,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129651171335,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8086753,\n", + " 't_depend': 1754855252.8510544,\n", + " 't_run': 1754855252.92709,\n", + " 't_cleanup': 1754855253.316255,\n", + " 't_inactive': 1754855253.3522894,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129651171339,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8089006,\n", + " 't_depend': 1754855252.8512695,\n", + " 't_run': 1754855252.9272559,\n", + " 't_cleanup': 1754855253.3135066,\n", + " 't_inactive': 1754855253.3522203,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129634394115,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8076751,\n", + " 't_depend': 1754855252.8505247,\n", + " 't_run': 1754855252.9266055,\n", + " 't_cleanup': 1754855253.3137717,\n", + " 't_inactive': 1754855253.3521519,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129651171331,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.808419,\n", + " 't_depend': 1754855252.8508644,\n", + " 't_run': 1754855252.9269078,\n", + " 't_cleanup': 1754855253.3138292,\n", + " 't_inactive': 1754855253.3520796,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129634394116,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.807806,\n", + " 't_depend': 1754855252.850574,\n", + " 't_run': 1754855252.926649,\n", + " 't_cleanup': 1754855253.313933,\n", + " 't_inactive': 1754855253.3520029,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129651171329,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8083937,\n", + " 't_depend': 1754855252.8507695,\n", + " 't_run': 1754855252.926821,\n", + " 't_cleanup': 1754855253.3144064,\n", + " 't_inactive': 1754855253.3519359,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129634394117,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8078816,\n", + " 't_depend': 1754855252.8506207,\n", + " 't_run': 1754855252.9266922,\n", + " 't_cleanup': 1754855253.3118775,\n", + " 't_inactive': 1754855253.3518677,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129634394118,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8083317,\n", + " 't_depend': 1754855252.8506749,\n", + " 't_run': 1754855252.9267352,\n", + " 't_cleanup': 1754855253.3122923,\n", + " 't_inactive': 1754855253.3517997,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129651171334,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8085845,\n", + " 't_depend': 1754855252.851008,\n", + " 't_run': 1754855252.927034,\n", + " 't_cleanup': 1754855253.3117657,\n", + " 't_inactive': 1754855253.3517303,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129651171328,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8083746,\n", + " 't_depend': 1754855252.8507218,\n", + " 't_run': 1754855252.9267778,\n", + " 't_cleanup': 1754855253.3112533,\n", + " 't_inactive': 1754855253.3516617,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129634394114,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8075647,\n", + " 't_depend': 1754855252.8504753,\n", + " 't_run': 1754855252.9265609,\n", + " 't_cleanup': 1754855253.3111897,\n", + " 't_inactive': 1754855253.3515904,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129634394113,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8074203,\n", + " 't_depend': 1754855252.8504188,\n", + " 't_run': 1754855252.9265103,\n", + " 't_cleanup': 1754855253.3103232,\n", + " 't_inactive': 1754855253.3515086,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129634394112,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.807334,\n", + " 't_depend': 1754855252.8503683,\n", + " 't_run': 1754855252.926322,\n", + " 't_cleanup': 1754855253.2955942,\n", + " 't_inactive': 1754855253.3450634,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129701502982,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8114223,\n", + " 't_depend': 1754855252.8536422,\n", + " 't_run': 1754855252.9292622,\n", + " 't_cleanup': 1754855253.267521,\n", + " 't_inactive': 1754855253.320702,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129718280201,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.812958,\n", + " 't_depend': 1754855252.8546503,\n", + " 't_run': 1754855252.930121,\n", + " 't_cleanup': 1754855253.2676344,\n", + " 't_inactive': 1754855253.3206353,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129718280200,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8128479,\n", + " 't_depend': 1754855252.8545997,\n", + " 't_run': 1754855252.9300802,\n", + " 't_cleanup': 1754855253.2673953,\n", + " 't_inactive': 1754855253.320568,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129701502992,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8119476,\n", + " 't_depend': 1754855252.8541303,\n", + " 't_run': 1754855252.9296732,\n", + " 't_cleanup': 1754855253.2674663,\n", + " 't_inactive': 1754855253.3204997,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129718280199,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8127763,\n", + " 't_depend': 1754855252.8545547,\n", + " 't_run': 1754855252.9300318,\n", + " 't_cleanup': 1754855253.2672484,\n", + " 't_inactive': 1754855253.3204105,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129718280198,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.812694,\n", + " 't_depend': 1754855252.8545103,\n", + " 't_run': 1754855252.9299908,\n", + " 't_cleanup': 1754855253.2738047,\n", + " 't_inactive': 1754855253.3203433,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129718280196,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8125057,\n", + " 't_depend': 1754855252.854416,\n", + " 't_run': 1754855252.929911,\n", + " 't_cleanup': 1754855253.2729723,\n", + " 't_inactive': 1754855253.3202734,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129718280197,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8125832,\n", + " 't_depend': 1754855252.8544629,\n", + " 't_run': 1754855252.9299514,\n", + " 't_cleanup': 1754855253.2732325,\n", + " 't_inactive': 1754855253.3202066,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129718280193,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8122263,\n", + " 't_depend': 1754855252.8542714,\n", + " 't_run': 1754855252.9297924,\n", + " 't_cleanup': 1754855253.272016,\n", + " 't_inactive': 1754855253.3201358,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129701502983,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8114681,\n", + " 't_depend': 1754855252.8536975,\n", + " 't_run': 1754855252.9293044,\n", + " 't_cleanup': 1754855253.273431,\n", + " 't_inactive': 1754855253.3200524,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129718280192,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8121212,\n", + " 't_depend': 1754855252.8542228,\n", + " 't_run': 1754855252.9297523,\n", + " 't_cleanup': 1754855253.2716424,\n", + " 't_inactive': 1754855253.3199837,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129718280195,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8123956,\n", + " 't_depend': 1754855252.8543684,\n", + " 't_run': 1754855252.929871,\n", + " 't_cleanup': 1754855253.2726312,\n", + " 't_inactive': 1754855253.319896,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129701502993,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8120284,\n", + " 't_depend': 1754855252.8541799,\n", + " 't_run': 1754855252.9297132,\n", + " 't_cleanup': 1754855253.2718217,\n", + " 't_inactive': 1754855253.3197837,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129718280194,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8122978,\n", + " 't_depend': 1754855252.8543167,\n", + " 't_run': 1754855252.929831,\n", + " 't_cleanup': 1754855253.2722397,\n", + " 't_inactive': 1754855253.31967,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129684725778,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8110454,\n", + " 't_depend': 1754855252.8533247,\n", + " 't_run': 1754855252.928976,\n", + " 't_cleanup': 1754855253.2711017,\n", + " 't_inactive': 1754855253.3195963,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129701502990,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8117938,\n", + " 't_depend': 1754855252.854031,\n", + " 't_run': 1754855252.9295874,\n", + " 't_cleanup': 1754855253.269862,\n", + " 't_inactive': 1754855253.319521,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129701502991,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8118634,\n", + " 't_depend': 1754855252.8540828,\n", + " 't_run': 1754855252.9296265,\n", + " 't_cleanup': 1754855253.2709696,\n", + " 't_inactive': 1754855253.3194284,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129701502988,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.811718,\n", + " 't_depend': 1754855252.8539379,\n", + " 't_run': 1754855252.9295068,\n", + " 't_cleanup': 1754855253.2702947,\n", + " 't_inactive': 1754855253.3193588,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129701502987,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8116379,\n", + " 't_depend': 1754855252.85389,\n", + " 't_run': 1754855252.9294672,\n", + " 't_cleanup': 1754855253.2705858,\n", + " 't_inactive': 1754855253.3192873,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129701502989,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8117595,\n", + " 't_depend': 1754855252.8539848,\n", + " 't_run': 1754855252.9295473,\n", + " 't_cleanup': 1754855253.2692573,\n", + " 't_inactive': 1754855253.3191905,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129701502984,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8115096,\n", + " 't_depend': 1754855252.8537416,\n", + " 't_run': 1754855252.9293451,\n", + " 't_cleanup': 1754855253.2696328,\n", + " 't_inactive': 1754855253.3191211,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129701502985,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.811553,\n", + " 't_depend': 1754855252.853788,\n", + " 't_run': 1754855252.9293869,\n", + " 't_cleanup': 1754855253.2675786,\n", + " 't_inactive': 1754855253.319039,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129684725775,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8108895,\n", + " 't_depend': 1754855252.8531895,\n", + " 't_run': 1754855252.9288538,\n", + " 't_cleanup': 1754855253.264321,\n", + " 't_inactive': 1754855253.3189697,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129701502986,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.811598,\n", + " 't_depend': 1754855252.8538356,\n", + " 't_run': 1754855252.929427,\n", + " 't_cleanup': 1754855253.2677245,\n", + " 't_inactive': 1754855253.3188992,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129684725773,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8107946,\n", + " 't_depend': 1754855252.8530962,\n", + " 't_run': 1754855252.9287715,\n", + " 't_cleanup': 1754855253.2694075,\n", + " 't_inactive': 1754855253.318826,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129701502980,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8113117,\n", + " 't_depend': 1754855252.8535511,\n", + " 't_run': 1754855252.9291809,\n", + " 't_cleanup': 1754855253.2646337,\n", + " 't_inactive': 1754855253.3187544,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129684725765,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8103704,\n", + " 't_depend': 1754855252.8527193,\n", + " 't_run': 1754855252.928458,\n", + " 't_cleanup': 1754855253.262809,\n", + " 't_inactive': 1754855253.3186848,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129701502981,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8113499,\n", + " 't_depend': 1754855252.8535967,\n", + " 't_run': 1754855252.9292202,\n", + " 't_cleanup': 1754855253.2645571,\n", + " 't_inactive': 1754855253.318612,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129701502979,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.811265,\n", + " 't_depend': 1754855252.8535068,\n", + " 't_run': 1754855252.929143,\n", + " 't_cleanup': 1754855253.2647676,\n", + " 't_inactive': 1754855253.3185413,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129684725776,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8109324,\n", + " 't_depend': 1754855252.8532338,\n", + " 't_run': 1754855252.9288957,\n", + " 't_cleanup': 1754855253.2644851,\n", + " 't_inactive': 1754855253.3184679,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129684725772,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8107164,\n", + " 't_depend': 1754855252.8530424,\n", + " 't_run': 1754855252.9287317,\n", + " 't_cleanup': 1754855253.264702,\n", + " 't_inactive': 1754855253.318367,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129684725761,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8101869,\n", + " 't_depend': 1754855252.8525248,\n", + " 't_run': 1754855252.928304,\n", + " 't_cleanup': 1754855253.2626383,\n", + " 't_inactive': 1754855253.3181567,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129701502978,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8112185,\n", + " 't_depend': 1754855252.85346,\n", + " 't_run': 1754855252.9291027,\n", + " 't_cleanup': 1754855253.2626905,\n", + " 't_inactive': 1754855253.3180673,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129701502977,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8111389,\n", + " 't_depend': 1754855252.8534157,\n", + " 't_run': 1754855252.9290545,\n", + " 't_cleanup': 1754855253.2622645,\n", + " 't_inactive': 1754855253.3179872,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129701502976,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8111045,\n", + " 't_depend': 1754855252.8533704,\n", + " 't_run': 1754855252.9290156,\n", + " 't_cleanup': 1754855253.2620025,\n", + " 't_inactive': 1754855253.3179038,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129684725777,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8110025,\n", + " 't_depend': 1754855252.8532798,\n", + " 't_run': 1754855252.928937,\n", + " 't_cleanup': 1754855253.2615845,\n", + " 't_inactive': 1754855253.3178196,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129684725774,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8108444,\n", + " 't_depend': 1754855252.8531435,\n", + " 't_run': 1754855252.9288127,\n", + " 't_cleanup': 1754855253.2614386,\n", + " 't_inactive': 1754855253.3177388,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129684725766,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8104172,\n", + " 't_depend': 1754855252.85277,\n", + " 't_run': 1754855252.928498,\n", + " 't_cleanup': 1754855253.261225,\n", + " 't_inactive': 1754855253.3176475,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129684725770,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8106282,\n", + " 't_depend': 1754855252.8529527,\n", + " 't_run': 1754855252.9286537,\n", + " 't_cleanup': 1754855253.2613065,\n", + " 't_inactive': 1754855253.3175638,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129684725769,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8105881,\n", + " 't_depend': 1754855252.8529067,\n", + " 't_run': 1754855252.9286153,\n", + " 't_cleanup': 1754855253.2597504,\n", + " 't_inactive': 1754855253.3174665,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129684725771,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.810679,\n", + " 't_depend': 1754855252.852998,\n", + " 't_run': 1754855252.9286916,\n", + " 't_cleanup': 1754855253.2331526,\n", + " 't_inactive': 1754855253.3089502,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129684725768,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8105464,\n", + " 't_depend': 1754855252.8528626,\n", + " 't_run': 1754855252.9285762,\n", + " 't_cleanup': 1754855253.232665,\n", + " 't_inactive': 1754855253.308883,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129684725764,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8103294,\n", + " 't_depend': 1754855252.8526704,\n", + " 't_run': 1754855252.9284189,\n", + " 't_cleanup': 1754855253.2328906,\n", + " 't_inactive': 1754855253.3088129,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129684725767,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8105106,\n", + " 't_depend': 1754855252.8528166,\n", + " 't_run': 1754855252.9285371,\n", + " 't_cleanup': 1754855253.233053,\n", + " 't_inactive': 1754855253.3087413,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129684725760,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8101113,\n", + " 't_depend': 1754855252.8524618,\n", + " 't_run': 1754855252.9282658,\n", + " 't_cleanup': 1754855253.232765,\n", + " 't_inactive': 1754855253.3086662,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129684725763,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8102858,\n", + " 't_depend': 1754855252.8526268,\n", + " 't_run': 1754855252.9283812,\n", + " 't_cleanup': 1754855253.2308354,\n", + " 't_inactive': 1754855253.3082693,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98129684725762,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.8102376,\n", + " 't_depend': 1754855252.8525748,\n", + " 't_run': 1754855252.9283428,\n", + " 't_cleanup': 1754855253.2322786,\n", + " 't_inactive': 1754855253.3081706,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128141221894,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7186248,\n", + " 't_depend': 1754855252.7351725,\n", + " 't_run': 1754855252.8029745,\n", + " 't_cleanup': 1754855253.1400247,\n", + " 't_inactive': 1754855253.2551045,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128141221892,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7185326,\n", + " 't_depend': 1754855252.7350554,\n", + " 't_run': 1754855252.8028982,\n", + " 't_cleanup': 1754855253.140449,\n", + " 't_inactive': 1754855253.2550082,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128141221896,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7187183,\n", + " 't_depend': 1754855252.73527,\n", + " 't_run': 1754855252.8030505,\n", + " 't_cleanup': 1754855253.1399472,\n", + " 't_inactive': 1754855253.2549117,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128141221895,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7186718,\n", + " 't_depend': 1754855252.7352211,\n", + " 't_run': 1754855252.8030124,\n", + " 't_cleanup': 1754855253.1391377,\n", + " 't_inactive': 1754855253.2548304,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128141221891,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7184238,\n", + " 't_depend': 1754855252.7350082,\n", + " 't_run': 1754855252.8028593,\n", + " 't_cleanup': 1754855253.138547,\n", + " 't_inactive': 1754855253.2547412,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128141221893,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7185793,\n", + " 't_depend': 1754855252.7351184,\n", + " 't_run': 1754855252.802936,\n", + " 't_cleanup': 1754855253.140565,\n", + " 't_inactive': 1754855253.2546554,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128141221888,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7181618,\n", + " 't_depend': 1754855252.7348657,\n", + " 't_run': 1754855252.802745,\n", + " 't_cleanup': 1754855253.1403623,\n", + " 't_inactive': 1754855253.2545733,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128208330755,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7224038,\n", + " 't_depend': 1754855252.737609,\n", + " 't_run': 1754855252.8049293,\n", + " 't_cleanup': 1754855253.1389444,\n", + " 't_inactive': 1754855253.254485,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128208330759,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.722779,\n", + " 't_depend': 1754855252.7378273,\n", + " 't_run': 1754855252.805095,\n", + " 't_cleanup': 1754855253.1395884,\n", + " 't_inactive': 1754855253.2542405,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128141221890,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7183535,\n", + " 't_depend': 1754855252.7349615,\n", + " 't_run': 1754855252.8028212,\n", + " 't_cleanup': 1754855253.140276,\n", + " 't_inactive': 1754855253.2541666,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128191553546,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7217748,\n", + " 't_depend': 1754855252.7372549,\n", + " 't_run': 1754855252.8046315,\n", + " 't_cleanup': 1754855253.1388626,\n", + " 't_inactive': 1754855253.254061,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128208330756,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7224982,\n", + " 't_depend': 1754855252.7376678,\n", + " 't_run': 1754855252.8049676,\n", + " 't_cleanup': 1754855253.139225,\n", + " 't_inactive': 1754855253.2539802,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128191553543,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7215369,\n", + " 't_depend': 1754855252.7371075,\n", + " 't_run': 1754855252.8045154,\n", + " 't_cleanup': 1754855253.1398537,\n", + " 't_inactive': 1754855253.2538874,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128141221889,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.718258,\n", + " 't_depend': 1754855252.7349145,\n", + " 't_run': 1754855252.802783,\n", + " 't_cleanup': 1754855253.140109,\n", + " 't_inactive': 1754855253.2537584,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128124444680,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7180068,\n", + " 't_depend': 1754855252.734762,\n", + " 't_run': 1754855252.8026676,\n", + " 't_cleanup': 1754855253.1397712,\n", + " 't_inactive': 1754855253.2536678,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128124444677,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7177725,\n", + " 't_depend': 1754855252.7346127,\n", + " 't_run': 1754855252.8025532,\n", + " 't_cleanup': 1754855253.132899,\n", + " 't_inactive': 1754855253.2535694,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128124444678,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7178695,\n", + " 't_depend': 1754855252.7346618,\n", + " 't_run': 1754855252.8025918,\n", + " 't_cleanup': 1754855253.1331403,\n", + " 't_inactive': 1754855253.2534852,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128124444681,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7180886,\n", + " 't_depend': 1754855252.7348166,\n", + " 't_run': 1754855252.8027067,\n", + " 't_cleanup': 1754855253.1325483,\n", + " 't_inactive': 1754855253.2534108,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128208330758,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7226884,\n", + " 't_depend': 1754855252.7377772,\n", + " 't_run': 1754855252.8050454,\n", + " 't_cleanup': 1754855253.1323788,\n", + " 't_inactive': 1754855253.2533052,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128208330757,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7225895,\n", + " 't_depend': 1754855252.7377276,\n", + " 't_run': 1754855252.8050077,\n", + " 't_cleanup': 1754855253.1320505,\n", + " 't_inactive': 1754855253.2532263,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128124444679,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7179384,\n", + " 't_depend': 1754855252.7347124,\n", + " 't_run': 1754855252.8026295,\n", + " 't_cleanup': 1754855253.1332343,\n", + " 't_inactive': 1754855253.2531378,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128124444675,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7175322,\n", + " 't_depend': 1754855252.734515,\n", + " 't_run': 1754855252.8024762,\n", + " 't_cleanup': 1754855253.1321893,\n", + " 't_inactive': 1754855253.2530503,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128124444673,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7173204,\n", + " 't_depend': 1754855252.734415,\n", + " 't_run': 1754855252.8023996,\n", + " 't_cleanup': 1754855253.131827,\n", + " 't_inactive': 1754855253.252961,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128124444676,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7176073,\n", + " 't_depend': 1754855252.734564,\n", + " 't_run': 1754855252.8025148,\n", + " 't_cleanup': 1754855253.1401927,\n", + " 't_inactive': 1754855253.2528782,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128208330754,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7223136,\n", + " 't_depend': 1754855252.7375596,\n", + " 't_run': 1754855252.8048618,\n", + " 't_cleanup': 1754855253.1317308,\n", + " 't_inactive': 1754855253.2527847,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128191553547,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7218645,\n", + " 't_depend': 1754855252.737306,\n", + " 't_run': 1754855252.8046696,\n", + " 't_cleanup': 1754855253.1327996,\n", + " 't_inactive': 1754855253.2527092,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128208330753,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7222235,\n", + " 't_depend': 1754855252.7375104,\n", + " 't_run': 1754855252.8048232,\n", + " 't_cleanup': 1754855253.1310241,\n", + " 't_inactive': 1754855253.2525954,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128208330752,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7221358,\n", + " 't_depend': 1754855252.7374606,\n", + " 't_run': 1754855252.804784,\n", + " 't_cleanup': 1754855253.1305685,\n", + " 't_inactive': 1754855253.252501,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128191553549,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.72204,\n", + " 't_depend': 1754855252.7374103,\n", + " 't_run': 1754855252.8047464,\n", + " 't_cleanup': 1754855253.1273532,\n", + " 't_inactive': 1754855253.2524185,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128191553538,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.721438,\n", + " 't_depend': 1754855252.736857,\n", + " 't_run': 1754855252.8043237,\n", + " 't_cleanup': 1754855253.1274273,\n", + " 't_inactive': 1754855253.2523007,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128191553548,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.721951,\n", + " 't_depend': 1754855252.7373595,\n", + " 't_run': 1754855252.8047082,\n", + " 't_cleanup': 1754855253.114064,\n", + " 't_inactive': 1754855253.21855,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128191553545,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7216797,\n", + " 't_depend': 1754855252.7372055,\n", + " 't_run': 1754855252.8045924,\n", + " 't_cleanup': 1754855253.1138558,\n", + " 't_inactive': 1754855253.2184603,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128191553544,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7215874,\n", + " 't_depend': 1754855252.7371564,\n", + " 't_run': 1754855252.8045537,\n", + " 't_cleanup': 1754855253.113995,\n", + " 't_inactive': 1754855253.218364,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128191553541,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7215042,\n", + " 't_depend': 1754855252.737003,\n", + " 't_run': 1754855252.8044386,\n", + " 't_cleanup': 1754855253.1139224,\n", + " 't_inactive': 1754855253.2182255,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128174776331,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7208796,\n", + " 't_depend': 1754855252.7365665,\n", + " 't_run': 1754855252.8040783,\n", + " 't_cleanup': 1754855253.1133814,\n", + " 't_inactive': 1754855253.218125,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128191553540,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7214904,\n", + " 't_depend': 1754855252.7369545,\n", + " 't_run': 1754855252.8044007,\n", + " 't_cleanup': 1754855253.115282,\n", + " 't_inactive': 1754855253.2180188,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128191553542,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7215226,\n", + " 't_depend': 1754855252.7370524,\n", + " 't_run': 1754855252.8044772,\n", + " 't_cleanup': 1754855253.1153388,\n", + " 't_inactive': 1754855253.2179296,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128191553539,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7214692,\n", + " 't_depend': 1754855252.7369068,\n", + " 't_run': 1754855252.804362,\n", + " 't_cleanup': 1754855253.1152136,\n", + " 't_inactive': 1754855253.2177918,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128191553537,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7213068,\n", + " 't_depend': 1754855252.736809,\n", + " 't_run': 1754855252.8042848,\n", + " 't_cleanup': 1754855253.1151428,\n", + " 't_inactive': 1754855253.2176938,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128191553536,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7211769,\n", + " 't_depend': 1754855252.7367609,\n", + " 't_run': 1754855252.804245,\n", + " 't_cleanup': 1754855253.1135738,\n", + " 't_inactive': 1754855253.2175853,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128174776333,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7209725,\n", + " 't_depend': 1754855252.7366648,\n", + " 't_run': 1754855252.804164,\n", + " 't_cleanup': 1754855253.1095963,\n", + " 't_inactive': 1754855253.217439,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128174776334,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7210398,\n", + " 't_depend': 1754855252.7367127,\n", + " 't_run': 1754855252.8042037,\n", + " 't_cleanup': 1754855253.1091998,\n", + " 't_inactive': 1754855253.2173357,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128124444674,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7174513,\n", + " 't_depend': 1754855252.7344666,\n", + " 't_run': 1754855252.802438,\n", + " 't_cleanup': 1754855253.1086552,\n", + " 't_inactive': 1754855253.2172384,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128174776332,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.720926,\n", + " 't_depend': 1754855252.7366161,\n", + " 't_run': 1754855252.8041217,\n", + " 't_cleanup': 1754855253.1071613,\n", + " 't_inactive': 1754855253.217146,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128174776330,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.720833,\n", + " 't_depend': 1754855252.7365189,\n", + " 't_run': 1754855252.8040276,\n", + " 't_cleanup': 1754855253.1066523,\n", + " 't_inactive': 1754855253.2169998,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128174776326,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7206497,\n", + " 't_depend': 1754855252.736325,\n", + " 't_run': 1754855252.8038683,\n", + " 't_cleanup': 1754855253.105882,\n", + " 't_inactive': 1754855253.2169244,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128124444672,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7171876,\n", + " 't_depend': 1754855252.7343662,\n", + " 't_run': 1754855252.8023615,\n", + " 't_cleanup': 1754855253.1036189,\n", + " 't_inactive': 1754855253.2168267,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128107667462,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7164054,\n", + " 't_depend': 1754855252.734119,\n", + " 't_run': 1754855252.8021693,\n", + " 't_cleanup': 1754855253.1051667,\n", + " 't_inactive': 1754855253.2167504,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128174776329,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7207875,\n", + " 't_depend': 1754855252.7364712,\n", + " 't_run': 1754855252.8039863,\n", + " 't_cleanup': 1754855253.1047132,\n", + " 't_inactive': 1754855253.2166405,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128107667466,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7169597,\n", + " 't_depend': 1754855252.7343152,\n", + " 't_run': 1754855252.8023233,\n", + " 't_cleanup': 1754855253.1041129,\n", + " 't_inactive': 1754855253.2165294,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128174776327,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.720696,\n", + " 't_depend': 1754855252.7363737,\n", + " 't_run': 1754855252.8039074,\n", + " 't_cleanup': 1754855253.1050925,\n", + " 't_inactive': 1754855253.2163882,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128107667465,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7167733,\n", + " 't_depend': 1754855252.734266,\n", + " 't_run': 1754855252.8022847,\n", + " 't_cleanup': 1754855253.104813,\n", + " 't_inactive': 1754855253.2162359,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128174776328,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7207417,\n", + " 't_depend': 1754855252.7364228,\n", + " 't_run': 1754855252.8039465,\n", + " 't_cleanup': 1754855253.104909,\n", + " 't_inactive': 1754855253.2161405,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128174776324,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.720468,\n", + " 't_depend': 1754855252.7362268,\n", + " 't_run': 1754855252.8037913,\n", + " 't_cleanup': 1754855253.1052494,\n", + " 't_inactive': 1754855253.2160277,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128174776325,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7206037,\n", + " 't_depend': 1754855252.7362761,\n", + " 't_run': 1754855252.80383,\n", + " 't_cleanup': 1754855253.1053572,\n", + " 't_inactive': 1754855253.2159448,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128107667464,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.716614,\n", + " 't_depend': 1754855252.7342172,\n", + " 't_run': 1754855252.8022463,\n", + " 't_cleanup': 1754855253.1049898,\n", + " 't_inactive': 1754855253.2158122,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128174776323,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7203574,\n", + " 't_depend': 1754855252.7361717,\n", + " 't_run': 1754855252.803752,\n", + " 't_cleanup': 1754855253.1017919,\n", + " 't_inactive': 1754855253.2157116,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128107667463,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7164757,\n", + " 't_depend': 1754855252.7341685,\n", + " 't_run': 1754855252.8022082,\n", + " 't_cleanup': 1754855253.1042974,\n", + " 't_inactive': 1754855253.2156239,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128157999112,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7200437,\n", + " 't_depend': 1754855252.7359626,\n", + " 't_run': 1754855252.8035982,\n", + " 't_cleanup': 1754855253.101898,\n", + " 't_inactive': 1754855253.2155433,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128174776322,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.720288,\n", + " 't_depend': 1754855252.7361202,\n", + " 't_run': 1754855252.8037138,\n", + " 't_cleanup': 1754855253.1015637,\n", + " 't_inactive': 1754855253.2154148,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128174776321,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7202184,\n", + " 't_depend': 1754855252.73606,\n", + " 't_run': 1754855252.803676,\n", + " 't_cleanup': 1754855253.1007988,\n", + " 't_inactive': 1754855253.215324,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128157999110,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.719877,\n", + " 't_depend': 1754855252.7358608,\n", + " 't_run': 1754855252.80352,\n", + " 't_cleanup': 1754855253.1007185,\n", + " 't_inactive': 1754855253.2152421,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128174776320,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7201488,\n", + " 't_depend': 1754855252.7360113,\n", + " 't_run': 1754855252.803637,\n", + " 't_cleanup': 1754855253.100971,\n", + " 't_inactive': 1754855253.215156,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128107667458,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7162075,\n", + " 't_depend': 1754855252.7339146,\n", + " 't_run': 1754855252.8019998,\n", + " 't_cleanup': 1754855253.100028,\n", + " 't_inactive': 1754855253.215012,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128107667459,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7162604,\n", + " 't_depend': 1754855252.7339625,\n", + " 't_run': 1754855252.80204,\n", + " 't_cleanup': 1754855253.0994728,\n", + " 't_inactive': 1754855253.214922,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128157999106,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7193704,\n", + " 't_depend': 1754855252.735664,\n", + " 't_run': 1754855252.8033664,\n", + " 't_cleanup': 1754855253.100135,\n", + " 't_inactive': 1754855253.2148395,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128157999108,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7196424,\n", + " 't_depend': 1754855252.7357616,\n", + " 't_run': 1754855252.8034425,\n", + " 't_cleanup': 1754855253.0997667,\n", + " 't_inactive': 1754855253.2147107,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128157999104,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7191095,\n", + " 't_depend': 1754855252.7355657,\n", + " 't_run': 1754855252.8032892,\n", + " 't_cleanup': 1754855253.0995705,\n", + " 't_inactive': 1754855253.2146282,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128107667457,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7161605,\n", + " 't_depend': 1754855252.7338662,\n", + " 't_run': 1754855252.8019624,\n", + " 't_cleanup': 1754855253.0990095,\n", + " 't_inactive': 1754855253.2145348,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128157999105,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7192385,\n", + " 't_depend': 1754855252.7356148,\n", + " 't_run': 1754855252.8033283,\n", + " 't_cleanup': 1754855253.0998569,\n", + " 't_inactive': 1754855253.2144444,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128090890243,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7160006,\n", + " 't_depend': 1754855252.7337205,\n", + " 't_run': 1754855252.8018458,\n", + " 't_cleanup': 1754855253.100889,\n", + " 't_inactive': 1754855253.2143192,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128107667461,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7163558,\n", + " 't_depend': 1754855252.73406,\n", + " 't_run': 1754855252.8021312,\n", + " 't_cleanup': 1754855253.1011577,\n", + " 't_inactive': 1754855253.2142465,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128107667460,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7163098,\n", + " 't_depend': 1754855252.7340112,\n", + " 't_run': 1754855252.8020895,\n", + " 't_cleanup': 1754855253.0991693,\n", + " 't_inactive': 1754855253.2141697,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128074113026,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7143152,\n", + " 't_depend': 1754855252.7330935,\n", + " 't_run': 1754855252.801359,\n", + " 't_cleanup': 1754855253.100288,\n", + " 't_inactive': 1754855253.214007,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128107667456,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7161102,\n", + " 't_depend': 1754855252.7338183,\n", + " 't_run': 1754855252.8019238,\n", + " 't_cleanup': 1754855253.0959845,\n", + " 't_inactive': 1754855253.2139232,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128141221901,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7189715,\n", + " 't_depend': 1754855252.7355168,\n", + " 't_run': 1754855252.8032515,\n", + " 't_cleanup': 1754855253.0959053,\n", + " 't_inactive': 1754855253.2138495,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128157999111,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7199736,\n", + " 't_depend': 1754855252.7359118,\n", + " 't_run': 1754855252.803559,\n", + " 't_cleanup': 1754855253.0956955,\n", + " 't_inactive': 1754855253.2137647,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128090890242,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.715955,\n", + " 't_depend': 1754855252.7336686,\n", + " 't_run': 1754855252.8018074,\n", + " 't_cleanup': 1754855253.0996604,\n", + " 't_inactive': 1754855253.2136314,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128141221897,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7187636,\n", + " 't_depend': 1754855252.7353182,\n", + " 't_run': 1754855252.8030992,\n", + " 't_cleanup': 1754855253.095604,\n", + " 't_inactive': 1754855253.2135398,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128090890241,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.715244,\n", + " 't_depend': 1754855252.7336142,\n", + " 't_run': 1754855252.801768,\n", + " 't_cleanup': 1754855253.1005964,\n", + " 't_inactive': 1754855253.2134643,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128074113028,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7145004,\n", + " 't_depend': 1754855252.733199,\n", + " 't_run': 1754855252.8014424,\n", + " 't_cleanup': 1754855253.0961955,\n", + " 't_inactive': 1754855253.2133276,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128157999109,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7197466,\n", + " 't_depend': 1754855252.7358098,\n", + " 't_run': 1754855252.8034806,\n", + " 't_cleanup': 1754855253.0951455,\n", + " 't_inactive': 1754855253.2132368,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128090890244,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7160482,\n", + " 't_depend': 1754855252.7337697,\n", + " 't_run': 1754855252.801885,\n", + " 't_cleanup': 1754855253.0955005,\n", + " 't_inactive': 1754855253.21316,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128157999107,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7195082,\n", + " 't_depend': 1754855252.735713,\n", + " 't_run': 1754855252.8034046,\n", + " 't_cleanup': 1754855253.094736,\n", + " 't_inactive': 1754855253.2130463,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128090890240,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7151964,\n", + " 't_depend': 1754855252.7335658,\n", + " 't_run': 1754855252.8017285,\n", + " 't_cleanup': 1754855253.0954225,\n", + " 't_inactive': 1754855253.2129245,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128074113034,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7150228,\n", + " 't_depend': 1754855252.7335172,\n", + " 't_run': 1754855252.8016884,\n", + " 't_cleanup': 1754855253.0945864,\n", + " 't_inactive': 1754855253.2128372,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128074113032,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7148678,\n", + " 't_depend': 1754855252.7334197,\n", + " 't_run': 1754855252.8016024,\n", + " 't_cleanup': 1754855253.0937245,\n", + " 't_inactive': 1754855253.2127573,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128074113033,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7149491,\n", + " 't_depend': 1754855252.7334688,\n", + " 't_run': 1754855252.8016412,\n", + " 't_cleanup': 1754855253.0936055,\n", + " 't_inactive': 1754855253.2126627,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128074113031,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7147653,\n", + " 't_depend': 1754855252.73337,\n", + " 't_run': 1754855252.8015618,\n", + " 't_cleanup': 1754855253.091453,\n", + " 't_inactive': 1754855253.2125833,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128141221900,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.718903,\n", + " 't_depend': 1754855252.7354667,\n", + " 't_run': 1754855252.8032131,\n", + " 't_cleanup': 1754855253.080201,\n", + " 't_inactive': 1754855253.2125096,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128074113029,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.714611,\n", + " 't_depend': 1754855252.7332544,\n", + " 't_run': 1754855252.8014824,\n", + " 't_cleanup': 1754855253.0797517,\n", + " 't_inactive': 1754855253.212436,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128074113030,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.714687,\n", + " 't_depend': 1754855252.7333193,\n", + " 't_run': 1754855252.801523,\n", + " 't_cleanup': 1754855253.079613,\n", + " 't_inactive': 1754855253.2123594,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128141221899,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7188563,\n", + " 't_depend': 1754855252.7354162,\n", + " 't_run': 1754855252.8031754,\n", + " 't_cleanup': 1754855253.0751503,\n", + " 't_inactive': 1754855253.2122836,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128074113027,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.71439,\n", + " 't_depend': 1754855252.733145,\n", + " 't_run': 1754855252.801401,\n", + " 't_cleanup': 1754855253.0727305,\n", + " 't_inactive': 1754855253.2122083,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128141221898,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.71881,\n", + " 't_depend': 1754855252.7353673,\n", + " 't_run': 1754855252.8031378,\n", + " 't_cleanup': 1754855253.0726614,\n", + " 't_inactive': 1754855253.2121289,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128074113025,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7142315,\n", + " 't_depend': 1754855252.7330363,\n", + " 't_run': 1754855252.8013172,\n", + " 't_cleanup': 1754855253.0722356,\n", + " 't_inactive': 1754855253.212032,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128074113024,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.714117,\n", + " 't_depend': 1754855252.7329843,\n", + " 't_run': 1754855252.801276,\n", + " 't_cleanup': 1754855253.0720768,\n", + " 't_inactive': 1754855253.2119167,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128057335810,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7138066,\n", + " 't_depend': 1754855252.7328281,\n", + " 't_run': 1754855252.8011186,\n", + " 't_cleanup': 1754855253.0612702,\n", + " 't_inactive': 1754855253.1830552,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128057335811,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7138913,\n", + " 't_depend': 1754855252.7328787,\n", + " 't_run': 1754855252.8011625,\n", + " 't_cleanup': 1754855253.0605497,\n", + " 't_inactive': 1754855253.1829634,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128057335812,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7140043,\n", + " 't_depend': 1754855252.732929,\n", + " 't_run': 1754855252.8012338,\n", + " 't_cleanup': 1754855253.030491,\n", + " 't_inactive': 1754855253.1559489,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128225107972,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7235146,\n", + " 't_depend': 1754855252.7382288,\n", + " 't_run': 1754855252.805404,\n", + " 't_cleanup': 1754855252.9643753,\n", + " 't_inactive': 1754855253.1333895,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128225107968,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7231567,\n", + " 't_depend': 1754855252.738022,\n", + " 't_run': 1754855252.8052511,\n", + " 't_cleanup': 1754855252.9642434,\n", + " 't_inactive': 1754855253.1332996,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128225107969,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.723239,\n", + " 't_depend': 1754855252.7380805,\n", + " 't_run': 1754855252.8052886,\n", + " 't_cleanup': 1754855252.9640532,\n", + " 't_inactive': 1754855253.1331885,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128225107971,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7234251,\n", + " 't_depend': 1754855252.7381794,\n", + " 't_run': 1754855252.805366,\n", + " 't_cleanup': 1754855252.9636073,\n", + " 't_inactive': 1754855253.1330922,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128208330762,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7230468,\n", + " 't_depend': 1754855252.7379732,\n", + " 't_run': 1754855252.8052125,\n", + " 't_cleanup': 1754855252.9639962,\n", + " 't_inactive': 1754855253.1330059,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128208330760,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.722869,\n", + " 't_depend': 1754855252.7378762,\n", + " 't_run': 1754855252.8051355,\n", + " 't_cleanup': 1754855252.9641426,\n", + " 't_inactive': 1754855253.1328506,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128208330761,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.722959,\n", + " 't_depend': 1754855252.7379248,\n", + " 't_run': 1754855252.805174,\n", + " 't_cleanup': 1754855252.9642909,\n", + " 't_inactive': 1754855253.13275,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128225107970,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.723336,\n", + " 't_depend': 1754855252.7381306,\n", + " 't_run': 1754855252.805327,\n", + " 't_cleanup': 1754855252.9630506,\n", + " 't_inactive': 1754855253.1326675,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128057335808,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.7136045,\n", + " 't_depend': 1754855252.732721,\n", + " 't_run': 1754855252.770426,\n", + " 't_cleanup': 1754855252.9398298,\n", + " 't_inactive': 1754855253.1324303,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98128057335809,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855252.713725,\n", + " 't_depend': 1754855252.732776,\n", + " 't_run': 1754855252.8010523,\n", + " 't_cleanup': 1754855252.939659,\n", + " 't_inactive': 1754855253.132297,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98072558305289,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855249.4060845,\n", + " 't_depend': 1754855249.4187722,\n", + " 't_run': 1754855249.4355564,\n", + " 't_cleanup': 1754855252.4579248,\n", + " 't_inactive': 1754855252.4609509,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98072558305287,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855249.4058933,\n", + " 't_depend': 1754855249.4186697,\n", + " 't_run': 1754855249.4354606,\n", + " 't_cleanup': 1754855252.4578815,\n", + " 't_inactive': 1754855252.4608767,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98072558305285,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855249.405704,\n", + " 't_depend': 1754855249.4185705,\n", + " 't_run': 1754855249.4349399,\n", + " 't_cleanup': 1754855252.4578416,\n", + " 't_inactive': 1754855252.4607882,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98072558305284,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855249.4055917,\n", + " 't_depend': 1754855249.418525,\n", + " 't_run': 1754855249.4348974,\n", + " 't_cleanup': 1754855252.4577734,\n", + " 't_inactive': 1754855252.4606445,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98072558305288,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855249.4059825,\n", + " 't_depend': 1754855249.4187233,\n", + " 't_run': 1754855249.4355133,\n", + " 't_cleanup': 1754855252.4577289,\n", + " 't_inactive': 1754855252.4605713,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98072558305282,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855249.4053779,\n", + " 't_depend': 1754855249.418422,\n", + " 't_run': 1754855249.4348035,\n", + " 't_cleanup': 1754855252.4576113,\n", + " 't_inactive': 1754855252.4604936,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98072558305286,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855249.4058,\n", + " 't_depend': 1754855249.4186203,\n", + " 't_run': 1754855249.434983,\n", + " 't_cleanup': 1754855252.4575338,\n", + " 't_inactive': 1754855252.4598033,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98072558305283,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855249.4054854,\n", + " 't_depend': 1754855249.4184742,\n", + " 't_run': 1754855249.4348538,\n", + " 't_cleanup': 1754855252.4565592,\n", + " 't_inactive': 1754855252.4586184,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98072558305280,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855249.405114,\n", + " 't_depend': 1754855249.4183168,\n", + " 't_run': 1754855249.4329443,\n", + " 't_cleanup': 1754855252.4560106,\n", + " 't_inactive': 1754855252.4580784,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 98072558305281,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855249.4052591,\n", + " 't_depend': 1754855249.4183702,\n", + " 't_run': 1754855249.4338248,\n", + " 't_cleanup': 1754855252.4559445,\n", + " 't_inactive': 1754855252.45745,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 97390665138176,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855208.7620118,\n", + " 't_depend': 1754855208.7736819,\n", + " 't_run': 1754855208.7856562,\n", + " 't_cleanup': 1754855213.8032315,\n", + " 't_inactive': 1754855213.8041298,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'cwd': '/home/ubuntu/tutorial/ch2',\n", + " 'ntasks': 4,\n", + " 'ncores': 4,\n", + " 'duration': 300.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 1754855508.0,\n", + " 'waitstatus': 0},\n", + " {'id': 95699806978048,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754855107.9783614,\n", + " 't_depend': 1754855107.990325,\n", + " 't_run': 1754855108.0022883,\n", + " 't_cleanup': 1754855108.0174737,\n", + " 't_inactive': 1754855108.018468,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 76665434669056,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853973.4414752,\n", + " 't_depend': 1754853973.4541082,\n", + " 't_run': 1754853973.4667373,\n", + " 't_cleanup': 1754853973.4862418,\n", + " 't_inactive': 1754853973.4875677,\n", + " 'state': 64,\n", + " 'name': 'echo',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 76665434669058,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853973.4417293,\n", + " 't_depend': 1754853973.4542122,\n", + " 't_run': 1754853973.4669917,\n", + " 't_cleanup': 1754853973.4861856,\n", + " 't_inactive': 1754853973.4873977,\n", + " 'state': 64,\n", + " 'name': 'echo',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 76665434669057,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853973.4415967,\n", + " 't_depend': 1754853973.4541576,\n", + " 't_run': 1754853973.4669406,\n", + " 't_cleanup': 1754853973.4829597,\n", + " 't_inactive': 1754853973.4838068,\n", + " 'state': 64,\n", + " 'name': 'echo',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 76182200516608,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853944.638146,\n", + " 't_depend': 1754853944.6503308,\n", + " 't_run': 1754853944.6646285,\n", + " 't_cleanup': 1754853944.682941,\n", + " 't_inactive': 1754853944.685201,\n", + " 'state': 64,\n", + " 'name': 'echo',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 76182200516609,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853944.6382601,\n", + " 't_depend': 1754853944.6503785,\n", + " 't_run': 1754853944.6646698,\n", + " 't_cleanup': 1754853944.6828341,\n", + " 't_inactive': 1754853944.6850755,\n", + " 'state': 64,\n", + " 'name': 'echo',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 76182183739394,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853944.6380236,\n", + " 't_depend': 1754853944.6502824,\n", + " 't_run': 1754853944.664582,\n", + " 't_cleanup': 1754853944.682731,\n", + " 't_inactive': 1754853944.6849923,\n", + " 'state': 64,\n", + " 'name': 'echo',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 76182183739393,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853944.6379013,\n", + " 't_depend': 1754853944.6502318,\n", + " 't_run': 1754853944.663905,\n", + " 't_cleanup': 1754853944.6826649,\n", + " 't_inactive': 1754853944.6844978,\n", + " 'state': 64,\n", + " 'name': 'echo',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 76182183739392,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853944.6377776,\n", + " 't_depend': 1754853944.6501806,\n", + " 't_run': 1754853944.66327,\n", + " 't_cleanup': 1754853944.68239,\n", + " 't_inactive': 1754853944.6838923,\n", + " 'state': 64,\n", + " 'name': 'echo',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 75471635087360,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853902.285606,\n", + " 't_depend': 1754853902.2971485,\n", + " 't_run': 1754853902.3089676,\n", + " 't_cleanup': 1754853902.3277683,\n", + " 't_inactive': 1754853902.328647,\n", + " 'state': 64,\n", + " 'name': 'echo',\n", + " 'ntasks': 4,\n", + " 'ncores': 4,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 74604571787264,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853850.6042378,\n", + " 't_depend': 1754853850.615589,\n", + " 't_run': 1754853850.627505,\n", + " 't_cleanup': 1754853850.6461422,\n", + " 't_inactive': 1754853850.6470063,\n", + " 'state': 64,\n", + " 'name': 'echo',\n", + " 'ntasks': 4,\n", + " 'ncores': 4,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 74175930695680,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853825.055359,\n", + " 't_depend': 1754853825.067586,\n", + " 't_run': 1754853825.0794768,\n", + " 't_cleanup': 1754853825.0947456,\n", + " 't_inactive': 1754853825.095677,\n", + " 'state': 64,\n", + " 'name': 'echo',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 72864136953856,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853746.866107,\n", + " 't_depend': 1754853746.878482,\n", + " 't_run': 1754853746.8925583,\n", + " 't_cleanup': 1754853746.9111614,\n", + " 't_inactive': 1754853746.913246,\n", + " 'state': 64,\n", + " 'name': 'true',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 72864120176643,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853746.8659925,\n", + " 't_depend': 1754853746.878429,\n", + " 't_run': 1754853746.8921726,\n", + " 't_cleanup': 1754853746.9111123,\n", + " 't_inactive': 1754853746.9131558,\n", + " 'state': 64,\n", + " 'name': 'true',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 72864120176640,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853746.8656335,\n", + " 't_depend': 1754853746.8782654,\n", + " 't_run': 1754853746.8915832,\n", + " 't_cleanup': 1754853746.9107945,\n", + " 't_inactive': 1754853746.9128923,\n", + " 'state': 64,\n", + " 'name': 'true',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 72864120176641,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853746.8657641,\n", + " 't_depend': 1754853746.8783247,\n", + " 't_run': 1754853746.892066,\n", + " 't_cleanup': 1754853746.9106827,\n", + " 't_inactive': 1754853746.9123867,\n", + " 'state': 64,\n", + " 'name': 'true',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 72864120176642,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853746.865883,\n", + " 't_depend': 1754853746.8783774,\n", + " 't_run': 1754853746.8921294,\n", + " 't_cleanup': 1754853746.9105082,\n", + " 't_inactive': 1754853746.911934,\n", + " 'state': 64,\n", + " 'name': 'true',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 72620548554755,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853732.347508,\n", + " 't_depend': 1754853732.3601756,\n", + " 't_run': 1754853732.3742747,\n", + " 't_cleanup': 1754853732.392648,\n", + " 't_inactive': 1754853732.3946176,\n", + " 'state': 64,\n", + " 'name': 'true',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 72620548554753,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853732.3473024,\n", + " 't_depend': 1754853732.3600526,\n", + " 't_run': 1754853732.3741627,\n", + " 't_cleanup': 1754853732.392509,\n", + " 't_inactive': 1754853732.3945296,\n", + " 'state': 64,\n", + " 'name': 'true',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 72620548554754,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853732.347405,\n", + " 't_depend': 1754853732.3601215,\n", + " 't_run': 1754853732.374221,\n", + " 't_cleanup': 1754853732.3924034,\n", + " 't_inactive': 1754853732.3944438,\n", + " 'state': 64,\n", + " 'name': 'true',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 72620548554752,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853732.3471866,\n", + " 't_depend': 1754853732.3599985,\n", + " 't_run': 1754853732.3735409,\n", + " 't_cleanup': 1754853732.3922856,\n", + " 't_inactive': 1754853732.3943555,\n", + " 'state': 64,\n", + " 'name': 'true',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 72620531777536,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853732.3470883,\n", + " 't_depend': 1754853732.3599417,\n", + " 't_run': 1754853732.372953,\n", + " 't_cleanup': 1754853732.3922281,\n", + " 't_inactive': 1754853732.3938308,\n", + " 'state': 64,\n", + " 'name': 'true',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71669582397441,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853675.6652343,\n", + " 't_depend': 1754853675.691446,\n", + " 't_run': 1754853675.7065737,\n", + " 't_cleanup': 1754853675.724817,\n", + " 't_inactive': 1754853675.726773,\n", + " 'state': 64,\n", + " 'name': 'true',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71669565620224,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853675.6647668,\n", + " 't_depend': 1754853675.6912425,\n", + " 't_run': 1754853675.7055557,\n", + " 't_cleanup': 1754853675.7246819,\n", + " 't_inactive': 1754853675.7266953,\n", + " 'state': 64,\n", + " 'name': 'true',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71669582397440,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853675.6651232,\n", + " 't_depend': 1754853675.691396,\n", + " 't_run': 1754853675.7061923,\n", + " 't_cleanup': 1754853675.7244496,\n", + " 't_inactive': 1754853675.7264988,\n", + " 'state': 64,\n", + " 'name': 'true',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71669565620226,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853675.6650088,\n", + " 't_depend': 1754853675.6913455,\n", + " 't_run': 1754853675.7061489,\n", + " 't_cleanup': 1754853675.7243674,\n", + " 't_inactive': 1754853675.726426,\n", + " 'state': 64,\n", + " 'name': 'true',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71669565620225,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853675.6648924,\n", + " 't_depend': 1754853675.6912947,\n", + " 't_run': 1754853675.7060993,\n", + " 't_cleanup': 1754853675.724243,\n", + " 't_inactive': 1754853675.7258468,\n", + " 'state': 64,\n", + " 'name': 'true',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446462201864,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3665187,\n", + " 't_depend': 1754853662.3959835,\n", + " 't_run': 1754853662.504606,\n", + " 't_cleanup': 1754853662.9692764,\n", + " 't_inactive': 1754853663.0223095,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446445424643,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3651845,\n", + " 't_depend': 1754853662.3945527,\n", + " 't_run': 1754853662.5035317,\n", + " 't_cleanup': 1754853662.9696383,\n", + " 't_inactive': 1754853663.0184355,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446428647443,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3647943,\n", + " 't_depend': 1754853662.3939342,\n", + " 't_run': 1754853662.503048,\n", + " 't_cleanup': 1754853662.9697158,\n", + " 't_inactive': 1754853663.0183709,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446445424649,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3653765,\n", + " 't_depend': 1754853662.394857,\n", + " 't_run': 1754853662.5037603,\n", + " 't_cleanup': 1754853662.9696584,\n", + " 't_inactive': 1754853663.0183034,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446445424659,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3659031,\n", + " 't_depend': 1754853662.3953846,\n", + " 't_run': 1754853662.5041494,\n", + " 't_cleanup': 1754853662.9697843,\n", + " 't_inactive': 1754853663.0182378,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446428647447,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3649302,\n", + " 't_depend': 1754853662.3941412,\n", + " 't_run': 1754853662.5032132,\n", + " 't_cleanup': 1754853662.9699998,\n", + " 't_inactive': 1754853663.01817,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446411870247,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3640814,\n", + " 't_depend': 1754853662.3929043,\n", + " 't_run': 1754853662.5022955,\n", + " 't_cleanup': 1754853662.9697576,\n", + " 't_inactive': 1754853663.0181,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446445424655,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.365703,\n", + " 't_depend': 1754853662.3951685,\n", + " 't_run': 1754853662.5039864,\n", + " 't_cleanup': 1754853662.9699056,\n", + " 't_inactive': 1754853663.018024,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446462201856,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3661072,\n", + " 't_depend': 1754853662.3955865,\n", + " 't_run': 1754853662.5042994,\n", + " 't_cleanup': 1754853662.9699538,\n", + " 't_inactive': 1754853663.0179534,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446462201867,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3666735,\n", + " 't_depend': 1754853662.3961399,\n", + " 't_run': 1754853662.5047524,\n", + " 't_cleanup': 1754853662.9696972,\n", + " 't_inactive': 1754853663.0178866,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446462201872,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.366933,\n", + " 't_depend': 1754853662.3963919,\n", + " 't_run': 1754853662.5049698,\n", + " 't_cleanup': 1754853662.9655952,\n", + " 't_inactive': 1754853663.0178156,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446428647431,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3643699,\n", + " 't_depend': 1754853662.3933256,\n", + " 't_run': 1754853662.502596,\n", + " 't_cleanup': 1754853662.9700596,\n", + " 't_inactive': 1754853663.0177464,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446462201860,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3663058,\n", + " 't_depend': 1754853662.395788,\n", + " 't_run': 1754853662.5044537,\n", + " 't_cleanup': 1754853662.969847,\n", + " 't_inactive': 1754853663.0176682,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446428647427,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.364236,\n", + " 't_depend': 1754853662.393126,\n", + " 't_run': 1754853662.5024452,\n", + " 't_cleanup': 1754853662.9657433,\n", + " 't_inactive': 1754853663.0176008,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446428647439,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3646536,\n", + " 't_depend': 1754853662.3937325,\n", + " 't_run': 1754853662.5028973,\n", + " 't_cleanup': 1754853662.9647405,\n", + " 't_inactive': 1754853663.0175288,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446462201862,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.366411,\n", + " 't_depend': 1754853662.395884,\n", + " 't_run': 1754853662.5045314,\n", + " 't_cleanup': 1754853662.9653502,\n", + " 't_inactive': 1754853663.0174599,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446445424645,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3652432,\n", + " 't_depend': 1754853662.394655,\n", + " 't_run': 1754853662.5036085,\n", + " 't_cleanup': 1754853662.9646251,\n", + " 't_inactive': 1754853663.0173912,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446411870237,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3638527,\n", + " 't_depend': 1754853662.3924043,\n", + " 't_run': 1754853662.5019057,\n", + " 't_cleanup': 1754853662.96431,\n", + " 't_inactive': 1754853663.0173218,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446445424642,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3651493,\n", + " 't_depend': 1754853662.3944979,\n", + " 't_run': 1754853662.5034919,\n", + " 't_cleanup': 1754853662.9663322,\n", + " 't_inactive': 1754853663.0172532,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446428647442,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3647432,\n", + " 't_depend': 1754853662.3938844,\n", + " 't_run': 1754853662.5030103,\n", + " 't_cleanup': 1754853662.9665127,\n", + " 't_inactive': 1754853663.0171814,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446445424650,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3654623,\n", + " 't_depend': 1754853662.394908,\n", + " 't_run': 1754853662.503798,\n", + " 't_cleanup': 1754853662.9663706,\n", + " 't_inactive': 1754853663.0171127,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446445424657,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.365806,\n", + " 't_depend': 1754853662.3952708,\n", + " 't_run': 1754853662.5040617,\n", + " 't_cleanup': 1754853662.9667106,\n", + " 't_inactive': 1754853663.017035,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446411870245,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3640254,\n", + " 't_depend': 1754853662.3928046,\n", + " 't_run': 1754853662.5022202,\n", + " 't_cleanup': 1754853662.9663072,\n", + " 't_inactive': 1754853663.0169666,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446428647449,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3649833,\n", + " 't_depend': 1754853662.3942442,\n", + " 't_run': 1754853662.5032868,\n", + " 't_cleanup': 1754853662.9644425,\n", + " 't_inactive': 1754853663.0168955,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446428647425,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3641644,\n", + " 't_depend': 1754853662.3930151,\n", + " 't_run': 1754853662.50237,\n", + " 't_cleanup': 1754853662.9696188,\n", + " 't_inactive': 1754853663.0168266,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446411870241,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3639474,\n", + " 't_depend': 1754853662.392604,\n", + " 't_run': 1754853662.5020578,\n", + " 't_cleanup': 1754853662.9656694,\n", + " 't_inactive': 1754853663.016756,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446445424653,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.365591,\n", + " 't_depend': 1754853662.3950593,\n", + " 't_run': 1754853662.5039115,\n", + " 't_cleanup': 1754853662.9681315,\n", + " 't_inactive': 1754853663.0166824,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446445424661,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3660035,\n", + " 't_depend': 1754853662.3954856,\n", + " 't_run': 1754853662.504225,\n", + " 't_cleanup': 1754853662.9665785,\n", + " 't_inactive': 1754853663.0166101,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446428647446,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3648853,\n", + " 't_depend': 1754853662.394091,\n", + " 't_run': 1754853662.5031757,\n", + " 't_cleanup': 1754853662.9654262,\n", + " 't_inactive': 1754853663.0165384,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446462201858,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3662024,\n", + " 't_depend': 1754853662.3956866,\n", + " 't_run': 1754853662.5043738,\n", + " 't_cleanup': 1754853662.9664476,\n", + " 't_inactive': 1754853663.0164678,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446428647438,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3646386,\n", + " 't_depend': 1754853662.3936825,\n", + " 't_run': 1754853662.5028586,\n", + " 't_cleanup': 1754853662.969739,\n", + " 't_inactive': 1754853663.0163932,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446428647434,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3644898,\n", + " 't_depend': 1754853662.3934753,\n", + " 't_run': 1754853662.5027092,\n", + " 't_cleanup': 1754853662.9644995,\n", + " 't_inactive': 1754853663.0163112,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446428647429,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3642921,\n", + " 't_depend': 1754853662.393226,\n", + " 't_run': 1754853662.502521,\n", + " 't_cleanup': 1754853662.964055,\n", + " 't_inactive': 1754853663.0162404,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446462201868,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3667226,\n", + " 't_depend': 1754853662.3961892,\n", + " 't_run': 1754853662.5047977,\n", + " 't_cleanup': 1754853662.9658988,\n", + " 't_inactive': 1754853663.0161645,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446462201870,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3668292,\n", + " 't_depend': 1754853662.396291,\n", + " 't_run': 1754853662.5048816,\n", + " 't_cleanup': 1754853662.9666479,\n", + " 't_inactive': 1754853663.0160894,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446462201871,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3668745,\n", + " 't_depend': 1754853662.3963416,\n", + " 't_run': 1754853662.5049276,\n", + " 't_cleanup': 1754853662.9684012,\n", + " 't_inactive': 1754853663.0160017,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446411870242,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.36397,\n", + " 't_depend': 1754853662.392655,\n", + " 't_run': 1754853662.5021057,\n", + " 't_cleanup': 1754853662.9632995,\n", + " 't_inactive': 1754853663.0159297,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446445424646,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3652859,\n", + " 't_depend': 1754853662.3947053,\n", + " 't_run': 1754853662.5036469,\n", + " 't_cleanup': 1754853662.9628696,\n", + " 't_inactive': 1754853663.015863,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446445424651,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3654869,\n", + " 't_depend': 1754853662.394956,\n", + " 't_run': 1754853662.5038357,\n", + " 't_cleanup': 1754853662.964549,\n", + " 't_inactive': 1754853663.0157962,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446462201859,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3662817,\n", + " 't_depend': 1754853662.3957381,\n", + " 't_run': 1754853662.5044103,\n", + " 't_cleanup': 1754853662.9678128,\n", + " 't_inactive': 1754853663.0156028,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446428647433,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3644323,\n", + " 't_depend': 1754853662.3934247,\n", + " 't_run': 1754853662.5026715,\n", + " 't_cleanup': 1754853662.9630375,\n", + " 't_inactive': 1754853663.015538,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446462201863,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3664777,\n", + " 't_depend': 1754853662.395933,\n", + " 't_run': 1754853662.5045695,\n", + " 't_cleanup': 1754853662.965996,\n", + " 't_inactive': 1754853663.0154724,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446462201866,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3666177,\n", + " 't_depend': 1754853662.3960898,\n", + " 't_run': 1754853662.5047061,\n", + " 't_cleanup': 1754853662.9684942,\n", + " 't_inactive': 1754853663.015405,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446411870238,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3638742,\n", + " 't_depend': 1754853662.3924532,\n", + " 't_run': 1754853662.5019438,\n", + " 't_cleanup': 1754853662.962784,\n", + " 't_inactive': 1754853663.0153353,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446445424641,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3651247,\n", + " 't_depend': 1754853662.3944466,\n", + " 't_run': 1754853662.5034497,\n", + " 't_cleanup': 1754853662.9655123,\n", + " 't_inactive': 1754853663.0151372,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446445424662,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3660846,\n", + " 't_depend': 1754853662.3955352,\n", + " 't_run': 1754853662.5042627,\n", + " 't_cleanup': 1754853662.9690032,\n", + " 't_inactive': 1754853663.015063,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446428647445,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3648386,\n", + " 't_depend': 1754853662.394035,\n", + " 't_run': 1754853662.5031378,\n", + " 't_cleanup': 1754853662.969107,\n", + " 't_inactive': 1754853663.0149918,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446411870234,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3637831,\n", + " 't_depend': 1754853662.3922522,\n", + " 't_run': 1754853662.5017917,\n", + " 't_cleanup': 1754853662.9601343,\n", + " 't_inactive': 1754853663.0146759,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446428647437,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3646078,\n", + " 't_depend': 1754853662.3936267,\n", + " 't_run': 1754853662.5028217,\n", + " 't_cleanup': 1754853662.968901,\n", + " 't_inactive': 1754853663.0146093,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446445424654,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3656871,\n", + " 't_depend': 1754853662.3951178,\n", + " 't_run': 1754853662.503949,\n", + " 't_cleanup': 1754853662.9642572,\n", + " 't_inactive': 1754853663.014544,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446428647430,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3643315,\n", + " 't_depend': 1754853662.393276,\n", + " 't_run': 1754853662.5025582,\n", + " 't_cleanup': 1754853662.9695199,\n", + " 't_inactive': 1754853663.0144782,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446428647441,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3647282,\n", + " 't_depend': 1754853662.393832,\n", + " 't_run': 1754853662.5029724,\n", + " 't_cleanup': 1754853662.9658031,\n", + " 't_inactive': 1754853663.0144114,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446445424658,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3658793,\n", + " 't_depend': 1754853662.395333,\n", + " 't_run': 1754853662.504111,\n", + " 't_cleanup': 1754853662.9652536,\n", + " 't_inactive': 1754853663.0143461,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446428647426,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.364183,\n", + " 't_depend': 1754853662.3930657,\n", + " 't_run': 1754853662.502408,\n", + " 't_cleanup': 1754853662.9641924,\n", + " 't_inactive': 1754853663.0142763,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446411870246,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.364059,\n", + " 't_depend': 1754853662.392854,\n", + " 't_run': 1754853662.502257,\n", + " 't_cleanup': 1754853662.963644,\n", + " 't_inactive': 1754853663.0142117,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446428647450,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3650303,\n", + " 't_depend': 1754853662.394295,\n", + " 't_run': 1754853662.5033238,\n", + " 't_cleanup': 1754853662.9624784,\n", + " 't_inactive': 1754853663.0141432,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446428647435,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3645122,\n", + " 't_depend': 1754853662.3935263,\n", + " 't_run': 1754853662.5027466,\n", + " 't_cleanup': 1754853662.9574983,\n", + " 't_inactive': 1754853663.0140765,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446445424647,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3653126,\n", + " 't_depend': 1754853662.3947554,\n", + " 't_run': 1754853662.503685,\n", + " 't_cleanup': 1754853662.956851,\n", + " 't_inactive': 1754853663.014001,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446428647451,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3650448,\n", + " 't_depend': 1754853662.3943455,\n", + " 't_run': 1754853662.5033731,\n", + " 't_cleanup': 1754853662.961299,\n", + " 't_inactive': 1754853663.0139358,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446411870235,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3638005,\n", + " 't_depend': 1754853662.3923008,\n", + " 't_run': 1754853662.5018294,\n", + " 't_cleanup': 1754853662.9563591,\n", + " 't_inactive': 1754853663.013868,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446411870243,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.363986,\n", + " 't_depend': 1754853662.3927052,\n", + " 't_run': 1754853662.5021439,\n", + " 't_cleanup': 1754853662.9575937,\n", + " 't_inactive': 1754853663.0137904,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446411870239,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3638873,\n", + " 't_depend': 1754853662.3925037,\n", + " 't_run': 1754853662.5019827,\n", + " 't_cleanup': 1754853662.9570239,\n", + " 't_inactive': 1754853663.0137174,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446411870226,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.363573,\n", + " 't_depend': 1754853662.3918285,\n", + " 't_run': 1754853662.501477,\n", + " 't_cleanup': 1754853662.943869,\n", + " 't_inactive': 1754853663.0059347,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446411870230,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.363673,\n", + " 't_depend': 1754853662.3920324,\n", + " 't_run': 1754853662.5016305,\n", + " 't_cleanup': 1754853662.9439378,\n", + " 't_inactive': 1754853663.0058653,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446411870223,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3634863,\n", + " 't_depend': 1754853662.3916762,\n", + " 't_run': 1754853662.5013623,\n", + " 't_cleanup': 1754853662.9432974,\n", + " 't_inactive': 1754853663.0057955,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446411870227,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.363589,\n", + " 't_depend': 1754853662.391879,\n", + " 't_run': 1754853662.5015154,\n", + " 't_cleanup': 1754853662.942944,\n", + " 't_inactive': 1754853663.005725,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446411870231,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3636987,\n", + " 't_depend': 1754853662.392088,\n", + " 't_run': 1754853662.501669,\n", + " 't_cleanup': 1754853662.9431067,\n", + " 't_inactive': 1754853663.005644,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446411870222,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3634655,\n", + " 't_depend': 1754853662.3916245,\n", + " 't_run': 1754853662.501324,\n", + " 't_cleanup': 1754853662.9425864,\n", + " 't_inactive': 1754853663.005567,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446411870229,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.363652,\n", + " 't_depend': 1754853662.3919806,\n", + " 't_run': 1754853662.5015924,\n", + " 't_cleanup': 1754853662.9268684,\n", + " 't_inactive': 1754853662.987147,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446411870221,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3634233,\n", + " 't_depend': 1754853662.3915741,\n", + " 't_run': 1754853662.5012856,\n", + " 't_cleanup': 1754853662.9268038,\n", + " 't_inactive': 1754853662.987077,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446411870233,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3637636,\n", + " 't_depend': 1754853662.392196,\n", + " 't_run': 1754853662.501754,\n", + " 't_cleanup': 1754853662.9262059,\n", + " 't_inactive': 1754853662.986999,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446411870225,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3635352,\n", + " 't_depend': 1754853662.3917778,\n", + " 't_run': 1754853662.5014386,\n", + " 't_cleanup': 1754853662.9267583,\n", + " 't_inactive': 1754853662.9869297,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446411870219,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3633692,\n", + " 't_depend': 1754853662.3914745,\n", + " 't_run': 1754853662.5012088,\n", + " 't_cleanup': 1754853662.9108365,\n", + " 't_inactive': 1754853662.9868457,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446411870215,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3632655,\n", + " 't_depend': 1754853662.3912752,\n", + " 't_run': 1754853662.5010455,\n", + " 't_cleanup': 1754853662.9108977,\n", + " 't_inactive': 1754853662.96957,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446411870211,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3631692,\n", + " 't_depend': 1754853662.3910668,\n", + " 't_run': 1754853662.5008926,\n", + " 't_cleanup': 1754853662.9081855,\n", + " 't_inactive': 1754853662.9694731,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446395093009,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3625333,\n", + " 't_depend': 1754853662.3898253,\n", + " 't_run': 1754853662.499929,\n", + " 't_cleanup': 1754853662.9070513,\n", + " 't_inactive': 1754853662.9693985,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446395093029,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3630617,\n", + " 't_depend': 1754853662.3908644,\n", + " 't_run': 1754853662.5007386,\n", + " 't_cleanup': 1754853662.9091818,\n", + " 't_inactive': 1754853662.9693282,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446411870209,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3631136,\n", + " 't_depend': 1754853662.3909676,\n", + " 't_run': 1754853662.5008168,\n", + " 't_cleanup': 1754853662.909093,\n", + " 't_inactive': 1754853662.9692304,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446411870213,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3632104,\n", + " 't_depend': 1754853662.3911757,\n", + " 't_run': 1754853662.500969,\n", + " 't_cleanup': 1754853662.90754,\n", + " 't_inactive': 1754853662.9691544,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446411870217,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.363307,\n", + " 't_depend': 1754853662.3913755,\n", + " 't_run': 1754853662.501132,\n", + " 't_cleanup': 1754853662.9084668,\n", + " 't_inactive': 1754853662.9690514,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446395093008,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3625085,\n", + " 't_depend': 1754853662.3897758,\n", + " 't_run': 1754853662.4998875,\n", + " 't_cleanup': 1754853662.9089446,\n", + " 't_inactive': 1754853662.9689558,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446395093027,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3630087,\n", + " 't_depend': 1754853662.3907623,\n", + " 't_run': 1754853662.5006597,\n", + " 't_cleanup': 1754853662.9087434,\n", + " 't_inactive': 1754853662.9688525,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446395093025,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.362973,\n", + " 't_depend': 1754853662.390652,\n", + " 't_run': 1754853662.5005813,\n", + " 't_cleanup': 1754853662.9092798,\n", + " 't_inactive': 1754853662.9687655,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446378315793,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3618717,\n", + " 't_depend': 1754853662.3885856,\n", + " 't_run': 1754853662.498956,\n", + " 't_cleanup': 1754853662.9088116,\n", + " 't_inactive': 1754853662.9686942,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446395093021,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3628526,\n", + " 't_depend': 1754853662.3904467,\n", + " 't_run': 1754853662.5004253,\n", + " 't_cleanup': 1754853662.9094663,\n", + " 't_inactive': 1754853662.9686182,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446395093017,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3627694,\n", + " 't_depend': 1754853662.3902488,\n", + " 't_run': 1754853662.5002706,\n", + " 't_cleanup': 1754853662.9099963,\n", + " 't_inactive': 1754853662.968546,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446395092997,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3622327,\n", + " 't_depend': 1754853662.389202,\n", + " 't_run': 1754853662.4994385,\n", + " 't_cleanup': 1754853662.909356,\n", + " 't_inactive': 1754853662.9684465,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446395093013,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3626623,\n", + " 't_depend': 1754853662.3900359,\n", + " 't_run': 1754853662.5001066,\n", + " 't_cleanup': 1754853662.9095876,\n", + " 't_inactive': 1754853662.9683537,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446395092993,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3621252,\n", + " 't_depend': 1754853662.3889945,\n", + " 't_run': 1754853662.4992805,\n", + " 't_cleanup': 1754853662.9086847,\n", + " 't_inactive': 1754853662.9682777,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446411870214,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3632321,\n", + " 't_depend': 1754853662.391225,\n", + " 't_run': 1754853662.5010073,\n", + " 't_cleanup': 1754853662.9091501,\n", + " 't_inactive': 1754853662.968195,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446395093020,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.362823,\n", + " 't_depend': 1754853662.3903975,\n", + " 't_run': 1754853662.500387,\n", + " 't_cleanup': 1754853662.9073465,\n", + " 't_inactive': 1754853662.968077,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446395093001,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.362343,\n", + " 't_depend': 1754853662.389408,\n", + " 't_run': 1754853662.4995942,\n", + " 't_cleanup': 1754853662.9090393,\n", + " 't_inactive': 1754853662.967982,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446395093007,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3624728,\n", + " 't_depend': 1754853662.3897245,\n", + " 't_run': 1754853662.499845,\n", + " 't_cleanup': 1754853662.9078493,\n", + " 't_inactive': 1754853662.9679081,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446395093016,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.362715,\n", + " 't_depend': 1754853662.3901997,\n", + " 't_run': 1754853662.5002303,\n", + " 't_cleanup': 1754853662.908039,\n", + " 't_inactive': 1754853662.96777,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446395093005,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.362436,\n", + " 't_depend': 1754853662.3896163,\n", + " 't_run': 1754853662.4997606,\n", + " 't_cleanup': 1754853662.908405,\n", + " 't_inactive': 1754853662.9677038,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446411870210,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3631485,\n", + " 't_depend': 1754853662.3910174,\n", + " 't_run': 1754853662.5008554,\n", + " 't_cleanup': 1754853662.908341,\n", + " 't_inactive': 1754853662.967637,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446411870218,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3633509,\n", + " 't_depend': 1754853662.3914251,\n", + " 't_run': 1754853662.5011709,\n", + " 't_cleanup': 1754853662.906841,\n", + " 't_inactive': 1754853662.9675713,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446395093028,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3630416,\n", + " 't_depend': 1754853662.3908134,\n", + " 't_run': 1754853662.5006988,\n", + " 't_cleanup': 1754853662.9074574,\n", + " 't_inactive': 1754853662.9675016,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446395093024,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.362931,\n", + " 't_depend': 1754853662.3906014,\n", + " 't_run': 1754853662.500543,\n", + " 't_cleanup': 1754853662.9085693,\n", + " 't_inactive': 1754853662.9674315,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446395093012,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3626194,\n", + " 't_depend': 1754853662.389985,\n", + " 't_run': 1754853662.5000527,\n", + " 't_cleanup': 1754853662.9076087,\n", + " 't_inactive': 1754853662.967362,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446378315797,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3619804,\n", + " 't_depend': 1754853662.3887906,\n", + " 't_run': 1754853662.4991202,\n", + " 't_cleanup': 1754853662.9099169,\n", + " 't_inactive': 1754853662.9672906,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446395093023,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3628905,\n", + " 't_depend': 1754853662.3905497,\n", + " 't_run': 1754853662.5005035,\n", + " 't_cleanup': 1754853662.9072225,\n", + " 't_inactive': 1754853662.967222,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446378315789,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.361787,\n", + " 't_depend': 1754853662.3883762,\n", + " 't_run': 1754853662.498806,\n", + " 't_cleanup': 1754853662.9081163,\n", + " 't_inactive': 1754853662.9671488,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446378315785,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3615983,\n", + " 't_depend': 1754853662.3881683,\n", + " 't_run': 1754853662.498656,\n", + " 't_cleanup': 1754853662.9102745,\n", + " 't_inactive': 1754853662.9670582,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446395093015,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3627005,\n", + " 't_depend': 1754853662.3901463,\n", + " 't_run': 1754853662.5001893,\n", + " 't_cleanup': 1754853662.906491,\n", + " 't_inactive': 1754853662.9669867,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446395093019,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3628082,\n", + " 't_depend': 1754853662.3903484,\n", + " 't_run': 1754853662.500348,\n", + " 't_cleanup': 1754853662.9036934,\n", + " 't_inactive': 1754853662.9669118,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446395092999,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3622754,\n", + " 't_depend': 1754853662.3893058,\n", + " 't_run': 1754853662.4995162,\n", + " 't_cleanup': 1754853662.8943133,\n", + " 't_inactive': 1754853662.9668114,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446395093000,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3623111,\n", + " 't_depend': 1754853662.3893561,\n", + " 't_run': 1754853662.4995556,\n", + " 't_cleanup': 1754853662.8936048,\n", + " 't_inactive': 1754853662.9587376,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446395093004,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.362406,\n", + " 't_depend': 1754853662.3895652,\n", + " 't_run': 1754853662.4997187,\n", + " 't_cleanup': 1754853662.8932605,\n", + " 't_inactive': 1754853662.9586658,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446395093011,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3625998,\n", + " 't_depend': 1754853662.3899295,\n", + " 't_run': 1754853662.5000122,\n", + " 't_cleanup': 1754853662.893457,\n", + " 't_inactive': 1754853662.958589,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446395093003,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.362382,\n", + " 't_depend': 1754853662.3895125,\n", + " 't_run': 1754853662.4996767,\n", + " 't_cleanup': 1754853662.8939917,\n", + " 't_inactive': 1754853662.9585152,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446395092995,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.362163,\n", + " 't_depend': 1754853662.3891003,\n", + " 't_run': 1754853662.4993591,\n", + " 't_cleanup': 1754853662.8933792,\n", + " 't_inactive': 1754853662.95844,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446378315796,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3619685,\n", + " 't_depend': 1754853662.3887408,\n", + " 't_run': 1754853662.499081,\n", + " 't_cleanup': 1754853662.8935208,\n", + " 't_inactive': 1754853662.9583633,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446395092992,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3620996,\n", + " 't_depend': 1754853662.3889441,\n", + " 't_run': 1754853662.4992406,\n", + " 't_cleanup': 1754853662.8944159,\n", + " 't_inactive': 1754853662.9582834,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446395092996,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3622062,\n", + " 't_depend': 1754853662.3891513,\n", + " 't_run': 1754853662.4993994,\n", + " 't_cleanup': 1754853662.8940418,\n", + " 't_inactive': 1754853662.9582024,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446378315795,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3619351,\n", + " 't_depend': 1754853662.38869,\n", + " 't_run': 1754853662.4990306,\n", + " 't_cleanup': 1754853662.8942537,\n", + " 't_inactive': 1754853662.9581249,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446378315799,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3620522,\n", + " 't_depend': 1754853662.3888931,\n", + " 't_run': 1754853662.4991975,\n", + " 't_cleanup': 1754853662.8936539,\n", + " 't_inactive': 1754853662.9580302,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446378315792,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3618534,\n", + " 't_depend': 1754853662.3885338,\n", + " 't_run': 1754853662.4989185,\n", + " 't_cleanup': 1754853662.890204,\n", + " 't_inactive': 1754853662.957937,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446378315781,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3613951,\n", + " 't_depend': 1754853662.387956,\n", + " 't_run': 1754853662.4985054,\n", + " 't_cleanup': 1754853662.8886442,\n", + " 't_inactive': 1754853662.957864,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446378315791,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.361835,\n", + " 't_depend': 1754853662.388483,\n", + " 't_run': 1754853662.4988813,\n", + " 't_cleanup': 1754853662.8893383,\n", + " 't_inactive': 1754853662.9577928,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446378315777,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3611822,\n", + " 't_depend': 1754853662.38775,\n", + " 't_run': 1754853662.4983528,\n", + " 't_cleanup': 1754853662.8881269,\n", + " 't_inactive': 1754853662.9577198,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446361538571,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.360758,\n", + " 't_depend': 1754853662.387342,\n", + " 't_run': 1754853662.4977615,\n", + " 't_cleanup': 1754853662.8871157,\n", + " 't_inactive': 1754853662.957639,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446361538563,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.360315,\n", + " 't_depend': 1754853662.3869076,\n", + " 't_run': 1754853662.4974515,\n", + " 't_cleanup': 1754853662.887062,\n", + " 't_inactive': 1754853662.9575467,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446361538575,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3609703,\n", + " 't_depend': 1754853662.3875463,\n", + " 't_run': 1754853662.4979103,\n", + " 't_cleanup': 1754853662.8865304,\n", + " 't_inactive': 1754853662.9574513,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446361538562,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3602562,\n", + " 't_depend': 1754853662.3868566,\n", + " 't_run': 1754853662.4974139,\n", + " 't_cleanup': 1754853662.8863401,\n", + " 't_inactive': 1754853662.9573774,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446361538561,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3602164,\n", + " 't_depend': 1754853662.3868048,\n", + " 't_run': 1754853662.4973762,\n", + " 't_cleanup': 1754853662.8829648,\n", + " 't_inactive': 1754853662.9573066,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446361538560,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.36011,\n", + " 't_depend': 1754853662.3867552,\n", + " 't_run': 1754853662.4973378,\n", + " 't_cleanup': 1754853662.881754,\n", + " 't_inactive': 1754853662.957233,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446344761352,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3600023,\n", + " 't_depend': 1754853662.3867047,\n", + " 't_run': 1754853662.4973001,\n", + " 't_cleanup': 1754853662.881552,\n", + " 't_inactive': 1754853662.957158,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446378315780,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3613107,\n", + " 't_depend': 1754853662.3879035,\n", + " 't_run': 1754853662.4984667,\n", + " 't_cleanup': 1754853662.8774822,\n", + " 't_inactive': 1754853662.9570856,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446378315788,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3617072,\n", + " 't_depend': 1754853662.388324,\n", + " 't_run': 1754853662.498769,\n", + " 't_cleanup': 1754853662.8773715,\n", + " 't_inactive': 1754853662.956978,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446378315784,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3615546,\n", + " 't_depend': 1754853662.3881168,\n", + " 't_run': 1754853662.4986188,\n", + " 't_cleanup': 1754853662.8774235,\n", + " 't_inactive': 1754853662.956901,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446378315776,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.361115,\n", + " 't_depend': 1754853662.387698,\n", + " 't_run': 1754853662.498309,\n", + " 't_cleanup': 1754853662.877322,\n", + " 't_inactive': 1754853662.9568048,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446361538574,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.360918,\n", + " 't_depend': 1754853662.387496,\n", + " 't_run': 1754853662.497873,\n", + " 't_cleanup': 1754853662.8766553,\n", + " 't_inactive': 1754853662.9567213,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446344761349,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3596811,\n", + " 't_depend': 1754853662.386551,\n", + " 't_run': 1754853662.4971845,\n", + " 't_cleanup': 1754853662.8712692,\n", + " 't_inactive': 1754853662.9566412,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446344761351,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3598871,\n", + " 't_depend': 1754853662.3866534,\n", + " 't_run': 1754853662.4972615,\n", + " 't_cleanup': 1754853662.876526,\n", + " 't_inactive': 1754853662.956561,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446344761350,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.359781,\n", + " 't_depend': 1754853662.3866026,\n", + " 't_run': 1754853662.4972234,\n", + " 't_cleanup': 1754853662.870572,\n", + " 't_inactive': 1754853662.9564822,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446361538570,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3607216,\n", + " 't_depend': 1754853662.3872912,\n", + " 't_run': 1754853662.4977226,\n", + " 't_cleanup': 1754853662.8696613,\n", + " 't_inactive': 1754853662.9564097,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446462201873,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3670437,\n", + " 't_depend': 1754853662.3964431,\n", + " 't_run': 1754853662.5050128,\n", + " 't_cleanup': 1754853662.86957,\n", + " 't_inactive': 1754853662.9563153,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446344761348,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.359578,\n", + " 't_depend': 1754853662.3864987,\n", + " 't_run': 1754853662.4971457,\n", + " 't_cleanup': 1754853662.8695238,\n", + " 't_inactive': 1754853662.9562268,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446344761347,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3594735,\n", + " 't_depend': 1754853662.3864465,\n", + " 't_run': 1754853662.4971066,\n", + " 't_cleanup': 1754853662.869448,\n", + " 't_inactive': 1754853662.9561355,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446462201857,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3661783,\n", + " 't_depend': 1754853662.395636,\n", + " 't_run': 1754853662.5043368,\n", + " 't_cleanup': 1754853662.8673625,\n", + " 't_inactive': 1754853662.9560483,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446462201869,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3667772,\n", + " 't_depend': 1754853662.39624,\n", + " 't_run': 1754853662.5048418,\n", + " 't_cleanup': 1754853662.867208,\n", + " 't_inactive': 1754853662.936552,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446462201865,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3665786,\n", + " 't_depend': 1754853662.3960338,\n", + " 't_run': 1754853662.5046647,\n", + " 't_cleanup': 1754853662.8666623,\n", + " 't_inactive': 1754853662.9364858,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446462201861,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3663745,\n", + " 't_depend': 1754853662.3958356,\n", + " 't_run': 1754853662.504494,\n", + " 't_cleanup': 1754853662.8661466,\n", + " 't_inactive': 1754853662.9364114,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446445424660,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3659792,\n", + " 't_depend': 1754853662.3954353,\n", + " 't_run': 1754853662.504187,\n", + " 't_cleanup': 1754853662.866036,\n", + " 't_inactive': 1754853662.936341,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446445424656,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3657823,\n", + " 't_depend': 1754853662.3952198,\n", + " 't_run': 1754853662.5040243,\n", + " 't_cleanup': 1754853662.8657942,\n", + " 't_inactive': 1754853662.9362721,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446445424652,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3655694,\n", + " 't_depend': 1754853662.39501,\n", + " 't_run': 1754853662.5038736,\n", + " 't_cleanup': 1754853662.8656316,\n", + " 't_inactive': 1754853662.9362016,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446445424648,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3653526,\n", + " 't_depend': 1754853662.394806,\n", + " 't_run': 1754853662.503722,\n", + " 't_cleanup': 1754853662.8606212,\n", + " 't_inactive': 1754853662.9361174,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446445424644,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3652184,\n", + " 't_depend': 1754853662.3946052,\n", + " 't_run': 1754853662.5035708,\n", + " 't_cleanup': 1754853662.8510134,\n", + " 't_inactive': 1754853662.921053,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446445424640,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3650866,\n", + " 't_depend': 1754853662.3943958,\n", + " 't_run': 1754853662.5034106,\n", + " 't_cleanup': 1754853662.850615,\n", + " 't_inactive': 1754853662.920981,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446428647448,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3649442,\n", + " 't_depend': 1754853662.3941922,\n", + " 't_run': 1754853662.5032504,\n", + " 't_cleanup': 1754853662.8503432,\n", + " 't_inactive': 1754853662.9209065,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446428647444,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3648243,\n", + " 't_depend': 1754853662.393984,\n", + " 't_run': 1754853662.5030997,\n", + " 't_cleanup': 1754853662.8489723,\n", + " 't_inactive': 1754853662.9208379,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446428647440,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.364693,\n", + " 't_depend': 1754853662.3937826,\n", + " 't_run': 1754853662.5029352,\n", + " 't_cleanup': 1754853662.8477852,\n", + " 't_inactive': 1754853662.9207633,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446428647436,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3645358,\n", + " 't_depend': 1754853662.3935766,\n", + " 't_run': 1754853662.502784,\n", + " 't_cleanup': 1754853662.846631,\n", + " 't_inactive': 1754853662.9206948,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446428647432,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3643863,\n", + " 't_depend': 1754853662.393375,\n", + " 't_run': 1754853662.5026338,\n", + " 't_cleanup': 1754853662.846266,\n", + " 't_inactive': 1754853662.9206154,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446428647428,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3642771,\n", + " 't_depend': 1754853662.3931758,\n", + " 't_run': 1754853662.5024831,\n", + " 't_cleanup': 1754853662.845565,\n", + " 't_inactive': 1754853662.9205465,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446428647424,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3641362,\n", + " 't_depend': 1754853662.3929558,\n", + " 't_run': 1754853662.5023332,\n", + " 't_cleanup': 1754853662.8450496,\n", + " 't_inactive': 1754853662.9204688,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446411870244,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3639982,\n", + " 't_depend': 1754853662.3927555,\n", + " 't_run': 1754853662.502182,\n", + " 't_cleanup': 1754853662.8412366,\n", + " 't_inactive': 1754853662.9203959,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446411870240,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3639076,\n", + " 't_depend': 1754853662.392554,\n", + " 't_run': 1754853662.5020204,\n", + " 't_cleanup': 1754853662.8408055,\n", + " 't_inactive': 1754853662.9203112,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446411870236,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3638225,\n", + " 't_depend': 1754853662.3923526,\n", + " 't_run': 1754853662.5018673,\n", + " 't_cleanup': 1754853662.8399656,\n", + " 't_inactive': 1754853662.920241,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446411870224,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3635018,\n", + " 't_depend': 1754853662.3917265,\n", + " 't_run': 1754853662.5014005,\n", + " 't_cleanup': 1754853662.839872,\n", + " 't_inactive': 1754853662.9201622,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446411870232,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.363725,\n", + " 't_depend': 1754853662.3921504,\n", + " 't_run': 1754853662.501716,\n", + " 't_cleanup': 1754853662.8394487,\n", + " 't_inactive': 1754853662.9200892,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446411870228,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3636048,\n", + " 't_depend': 1754853662.3919294,\n", + " 't_run': 1754853662.5015543,\n", + " 't_cleanup': 1754853662.8393002,\n", + " 't_inactive': 1754853662.9200034,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446378315787,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3616624,\n", + " 't_depend': 1754853662.3882725,\n", + " 't_run': 1754853662.4987311,\n", + " 't_cleanup': 1754853662.833584,\n", + " 't_inactive': 1754853662.919925,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446378315783,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3614995,\n", + " 't_depend': 1754853662.3880582,\n", + " 't_run': 1754853662.498581,\n", + " 't_cleanup': 1754853662.833354,\n", + " 't_inactive': 1754853662.9198308,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446378315779,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3612878,\n", + " 't_depend': 1754853662.3878524,\n", + " 't_run': 1754853662.4984293,\n", + " 't_cleanup': 1754853662.8330781,\n", + " 't_inactive': 1754853662.9197438,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446411870208,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.363101,\n", + " 't_depend': 1754853662.3909168,\n", + " 't_run': 1754853662.5007775,\n", + " 't_cleanup': 1754853662.8332593,\n", + " 't_inactive': 1754853662.9196749,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446411870212,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3631928,\n", + " 't_depend': 1754853662.391125,\n", + " 't_run': 1754853662.5009315,\n", + " 't_cleanup': 1754853662.8329806,\n", + " 't_inactive': 1754853662.9196017,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446411870220,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3633873,\n", + " 't_depend': 1754853662.3915243,\n", + " 't_run': 1754853662.5012476,\n", + " 't_cleanup': 1754853662.832172,\n", + " 't_inactive': 1754853662.9195251,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446411870216,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3632822,\n", + " 't_depend': 1754853662.3913245,\n", + " 't_run': 1754853662.5010939,\n", + " 't_cleanup': 1754853662.831501,\n", + " 't_inactive': 1754853662.919453,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446395093026,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3629913,\n", + " 't_depend': 1754853662.3907027,\n", + " 't_run': 1754853662.5006204,\n", + " 't_cleanup': 1754853662.8303008,\n", + " 't_inactive': 1754853662.9193726,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446395093022,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3628728,\n", + " 't_depend': 1754853662.3904972,\n", + " 't_run': 1754853662.5004644,\n", + " 't_cleanup': 1754853662.8298755,\n", + " 't_inactive': 1754853662.9192948,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446395093010,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3625717,\n", + " 't_depend': 1754853662.3898766,\n", + " 't_run': 1754853662.4999707,\n", + " 't_cleanup': 1754853662.824953,\n", + " 't_inactive': 1754853662.9191844,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446395093018,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3627915,\n", + " 't_depend': 1754853662.3902996,\n", + " 't_run': 1754853662.5003097,\n", + " 't_cleanup': 1754853662.8256893,\n", + " 't_inactive': 1754853662.9008257,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446395093014,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3626788,\n", + " 't_depend': 1754853662.3900938,\n", + " 't_run': 1754853662.5001485,\n", + " 't_cleanup': 1754853662.8257794,\n", + " 't_inactive': 1754853662.9007578,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446395093006,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.362458,\n", + " 't_depend': 1754853662.389667,\n", + " 't_run': 1754853662.4998035,\n", + " 't_cleanup': 1754853662.825504,\n", + " 't_inactive': 1754853662.9006884,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446395093002,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3623636,\n", + " 't_depend': 1754853662.389461,\n", + " 't_run': 1754853662.499635,\n", + " 't_cleanup': 1754853662.824899,\n", + " 't_inactive': 1754853662.9006207,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446361538577,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3610778,\n", + " 't_depend': 1754853662.3876472,\n", + " 't_run': 1754853662.4979851,\n", + " 't_cleanup': 1754853662.8203604,\n", + " 't_inactive': 1754853662.9005477,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446395092998,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3622525,\n", + " 't_depend': 1754853662.3892536,\n", + " 't_run': 1754853662.4994774,\n", + " 't_cleanup': 1754853662.8199735,\n", + " 't_inactive': 1754853662.900478,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446361538573,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3608649,\n", + " 't_depend': 1754853662.3874447,\n", + " 't_run': 1754853662.497836,\n", + " 't_cleanup': 1754853662.8186843,\n", + " 't_inactive': 1754853662.9004042,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446361538569,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.360644,\n", + " 't_depend': 1754853662.3872306,\n", + " 't_run': 1754853662.4976847,\n", + " 't_cleanup': 1754853662.8173192,\n", + " 't_inactive': 1754853662.9003322,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446344761346,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3593583,\n", + " 't_depend': 1754853662.3863919,\n", + " 't_run': 1754853662.497057,\n", + " 't_cleanup': 1754853662.8170545,\n", + " 't_inactive': 1754853662.900258,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446344761345,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3592393,\n", + " 't_depend': 1754853662.3863387,\n", + " 't_run': 1754853662.4970171,\n", + " 't_cleanup': 1754853662.8167284,\n", + " 't_inactive': 1754853662.9001825,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446344761344,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3591125,\n", + " 't_depend': 1754853662.386284,\n", + " 't_run': 1754853662.4969637,\n", + " 't_cleanup': 1754853662.8147757,\n", + " 't_inactive': 1754853662.9000947,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446361538564,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3604078,\n", + " 't_depend': 1754853662.3869588,\n", + " 't_run': 1754853662.497489,\n", + " 't_cleanup': 1754853662.8101833,\n", + " 't_inactive': 1754853662.900004,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446395092994,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.362137,\n", + " 't_depend': 1754853662.3890448,\n", + " 't_run': 1754853662.4993203,\n", + " 't_cleanup': 1754853662.8096673,\n", + " 't_inactive': 1754853662.8999305,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446378315798,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3620045,\n", + " 't_depend': 1754853662.3888416,\n", + " 't_run': 1754853662.4991598,\n", + " 't_cleanup': 1754853662.8103406,\n", + " 't_inactive': 1754853662.8998547,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446378315794,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.361909,\n", + " 't_depend': 1754853662.3886373,\n", + " 't_run': 1754853662.498993,\n", + " 't_cleanup': 1754853662.8095639,\n", + " 't_inactive': 1754853662.8997793,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446378315790,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3618085,\n", + " 't_depend': 1754853662.3884318,\n", + " 't_run': 1754853662.4988434,\n", + " 't_cleanup': 1754853662.8047864,\n", + " 't_inactive': 1754853662.8997056,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446378315786,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3616383,\n", + " 't_depend': 1754853662.3882203,\n", + " 't_run': 1754853662.498694,\n", + " 't_cleanup': 1754853662.8041599,\n", + " 't_inactive': 1754853662.8996294,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446378315782,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.36142,\n", + " 't_depend': 1754853662.3880074,\n", + " 't_run': 1754853662.4985433,\n", + " 't_cleanup': 1754853662.8022892,\n", + " 't_inactive': 1754853662.899553,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446378315778,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3612223,\n", + " 't_depend': 1754853662.387801,\n", + " 't_run': 1754853662.498392,\n", + " 't_cleanup': 1754853662.8013206,\n", + " 't_inactive': 1754853662.8994737,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446361538576,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3610144,\n", + " 't_depend': 1754853662.3875964,\n", + " 't_run': 1754853662.497948,\n", + " 't_cleanup': 1754853662.8018272,\n", + " 't_inactive': 1754853662.8993838,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446361538572,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.360819,\n", + " 't_depend': 1754853662.3873925,\n", + " 't_run': 1754853662.4977984,\n", + " 't_cleanup': 1754853662.8006694,\n", + " 't_inactive': 1754853662.89929,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446361538568,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.360619,\n", + " 't_depend': 1754853662.3871756,\n", + " 't_run': 1754853662.49764,\n", + " 't_cleanup': 1754853662.8000731,\n", + " 't_inactive': 1754853662.8991568,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446361538567,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3605442,\n", + " 't_depend': 1754853662.3871224,\n", + " 't_run': 1754853662.4976027,\n", + " 't_cleanup': 1754853662.7983077,\n", + " 't_inactive': 1754853662.8807077,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446361538566,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3605177,\n", + " 't_depend': 1754853662.3870604,\n", + " 't_run': 1754853662.497565,\n", + " 't_cleanup': 1754853662.7977715,\n", + " 't_inactive': 1754853662.880629,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446361538565,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.3604321,\n", + " 't_depend': 1754853662.3870094,\n", + " 't_run': 1754853662.497527,\n", + " 't_cleanup': 1754853662.7970324,\n", + " 't_inactive': 1754853662.8805478,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446327984129,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.358981,\n", + " 't_depend': 1754853662.3862288,\n", + " 't_run': 1754853662.4883041,\n", + " 't_cleanup': 1754853662.7864063,\n", + " 't_inactive': 1754853662.8804607,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71446327984128,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853662.358846,\n", + " 't_depend': 1754853662.3861692,\n", + " 't_run': 1754853662.4600134,\n", + " 't_cleanup': 1754853662.786043,\n", + " 't_inactive': 1754853662.880339,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71145529278468,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853644.4295368,\n", + " 't_depend': 1754853644.440716,\n", + " 't_run': 1754853644.457095,\n", + " 't_cleanup': 1754853647.4821184,\n", + " 't_inactive': 1754853647.4832523,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71145529278467,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853644.429431,\n", + " 't_depend': 1754853644.440665,\n", + " 't_run': 1754853644.457046,\n", + " 't_cleanup': 1754853647.4816954,\n", + " 't_inactive': 1754853647.4826124,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71145512501251,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853644.4288616,\n", + " 't_depend': 1754853644.4404027,\n", + " 't_run': 1754853644.45633,\n", + " 't_cleanup': 1754853647.4797156,\n", + " 't_inactive': 1754853647.4817853,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71145529278465,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853644.429222,\n", + " 't_depend': 1754853644.4405563,\n", + " 't_run': 1754853644.4569604,\n", + " 't_cleanup': 1754853647.4795685,\n", + " 't_inactive': 1754853647.4815254,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71145512501248,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853644.42847,\n", + " 't_depend': 1754853644.440243,\n", + " 't_run': 1754853644.4546218,\n", + " 't_cleanup': 1754853647.4789846,\n", + " 't_inactive': 1754853647.4810257,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71145529278464,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853644.4290965,\n", + " 't_depend': 1754853644.440505,\n", + " 't_run': 1754853644.4569154,\n", + " 't_cleanup': 1754853647.4776478,\n", + " 't_inactive': 1754853647.4803913,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71145512501249,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853644.4286125,\n", + " 't_depend': 1754853644.4402974,\n", + " 't_run': 1754853644.4552867,\n", + " 't_cleanup': 1754853647.4771519,\n", + " 't_inactive': 1754853647.4797895,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71145529278466,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853644.429328,\n", + " 't_depend': 1754853644.4406097,\n", + " 't_run': 1754853644.4570055,\n", + " 't_cleanup': 1754853647.4770553,\n", + " 't_inactive': 1754853647.479257,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71145512501250,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853644.4287438,\n", + " 't_depend': 1754853644.440351,\n", + " 't_run': 1754853644.4562747,\n", + " 't_cleanup': 1754853647.4763496,\n", + " 't_inactive': 1754853647.4785619,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 71145512501252,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853644.4289756,\n", + " 't_depend': 1754853644.4404533,\n", + " 't_run': 1754853644.4563742,\n", + " 't_cleanup': 1754853647.4756453,\n", + " 't_inactive': 1754853647.4767482,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 70632146468864,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853613.829938,\n", + " 't_depend': 1754853613.8407938,\n", + " 't_run': 1754853613.8526556,\n", + " 't_cleanup': 1754853618.870364,\n", + " 't_inactive': 1754853618.8712044,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'cwd': '/home/ubuntu/tutorial/ch2',\n", + " 'ntasks': 4,\n", + " 'ncores': 4,\n", + " 'duration': 300.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 1754853913.0,\n", + " 'waitstatus': 0},\n", + " {'id': 70177584578560,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853586.735776,\n", + " 't_depend': 1754853586.7468407,\n", + " 't_run': 1754853586.7587156,\n", + " 't_cleanup': 1754853591.7765565,\n", + " 't_inactive': 1754853591.7774115,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'cwd': '/home/ubuntu/tutorial/ch2',\n", + " 'ntasks': 4,\n", + " 'ncores': 4,\n", + " 'duration': 300.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 1754853886.0,\n", + " 'waitstatus': 0},\n", + " {'id': 64250462601216,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853233.4514174,\n", + " 't_depend': 1754853233.4622488,\n", + " 't_run': 1754853233.474043,\n", + " 't_cleanup': 1754853233.4960055,\n", + " 't_inactive': 1754853233.496893,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '0',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 60411449704448,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754853004.6284952,\n", + " 't_depend': 1754853004.6392798,\n", + " 't_run': 1754853004.650989,\n", + " 't_cleanup': 1754853004.6660106,\n", + " 't_inactive': 1754853004.666885,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '1',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 60026848804864,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754852981.7048187,\n", + " 't_depend': 1754852981.715752,\n", + " 't_run': 1754852981.7275193,\n", + " 't_cleanup': 1754852981.7425592,\n", + " 't_inactive': 1754852981.74341,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '2',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0},\n", + " {'id': 56841929228288,\n", + " 'userid': 1000,\n", + " 'urgency': 16,\n", + " 'priority': 16,\n", + " 't_submit': 1754852791.868542,\n", + " 't_depend': 1754852791.8793094,\n", + " 't_run': 1754852791.8915725,\n", + " 't_cleanup': 1754852792.3162365,\n", + " 't_inactive': 1754852792.3171062,\n", + " 'state': 64,\n", + " 'name': 'sleep',\n", + " 'ntasks': 1,\n", + " 'ncores': 1,\n", + " 'duration': 0.0,\n", + " 'nnodes': 1,\n", + " 'ranks': '3',\n", + " 'nodelist': 'ip-10-0-20-251',\n", + " 'success': True,\n", + " 'exception_occurred': False,\n", + " 'result': 1,\n", + " 'expiration': 9223372036.0,\n", + " 'waitstatus': 0}]}" + ] + }, + "execution_count": 46, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "flux.job.job_list(f).get()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Stream Events\n", + "\n", + "
\n", + "Description: Use the `JournalConsumer` to stream events in real time\n", + "
\n", + "\n", + "For this example, we recommend you open up two side-by-side terminals. In the first, create a `JournalConsumer`. Since we set the timeout to -1, we will get all events that have been seen by the instance and then it will wait for new events.\n", + "\n", + "```bash\n", + "handle = flux.Flux()\n", + "consumer = flux.job.JournalConsumer(handle).start()\n", + "while True:\n", + " event = consumer.poll(timeout=-1)\n", + " print(event)\n", + "```\n", + "In the second terminal, try submitting a job. Anything will do!\n", + "\n", + "```bash\n", + "flux run sleep 5\n", + "flux run echo \"A slice of apple pie is 2:50 in Jamaica and 3:50 in the Bahamas. These are the pie rates of the Caribbean.\"\n", + "```\n", + "This is a powerful capability, because it allows for getting events for jobs as they happen, in real-time. We have built an entire state machine library to run complex workflows with this functionality." + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.12.3" + } + }, + "nbformat": 4, + "nbformat_minor": 4 +} diff --git a/2026/HPCIC-AWS/tutorial/ch2/bulksubmit_executor.py b/2026/HPCIC-AWS/tutorial/ch2/bulksubmit_executor.py new file mode 100755 index 00000000..527b7752 --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/ch2/bulksubmit_executor.py @@ -0,0 +1,68 @@ +#!/usr/bin/env python3 + +import time +import sys +import argparse +import concurrent.futures as cf + +from flux.job import FluxExecutor, JobspecV1 + + +def log(label, s): + print(label + ": " + s) + + +def progress(fraction, length=72, suffix=""): + fill = int(round(length * fraction)) + bar = "\u2588" * fill + "-" * (length - fill) + s = f"\r|{bar}| {100 * fraction:.1f}% {suffix}" + sys.stdout.write(s) + if fraction == 1.0: + sys.stdout.write("\n") + + +def main(): + parser = argparse.ArgumentParser( + description="Submit a command repeatedly using FluxExecutor" + ) + parser.add_argument( + "-n", + "--njobs", + type=int, + metavar="N", + help="Set the total number of jobs to run", + default=100, + ) + parser.add_argument("command", nargs=argparse.REMAINDER) + args = parser.parse_args() + if not args.command: + args.command = ["true"] + t0 = time.perf_counter() + label = "bulksubmit_executor" + with FluxExecutor() as executor: + compute_jobspec = JobspecV1.from_command(args.command) + compute_jobspec.environment = {"TMPDIR": "/tmp/"} + futures = [executor.submit(compute_jobspec) for _ in range(args.njobs)] + # wait for the jobid for each job, as a proxy for the job being submitted + for fut in futures: + fut.jobid() + # all jobs submitted - print timings + dt = time.perf_counter() - t0 + jps = args.njobs / dt + log(label, f"submitted {args.njobs} jobs in {dt:.2f}s. {jps:.2f}job/s") + # wait for jobs to complete + for i, _ in enumerate(cf.as_completed(futures)): + if i == 0: + log( + label, + f"First job finished in about {time.perf_counter() - t0:.3f}s", + ) + jps = (i + 1) / (time.perf_counter() - t0) + progress((i + 1) / args.njobs, length=58, suffix=f"({jps:.1f} job/s)") + # print time summary + dt = time.perf_counter() - t0 + log(label, f"Ran {args.njobs} jobs in {dt:.1f}s. {args.njobs / dt:.1f} job/s") + + +if __name__ == "__main__": + main() diff --git a/2026/HPCIC-AWS/tutorial/ch2/compute.py b/2026/HPCIC-AWS/tutorial/ch2/compute.py new file mode 100755 index 00000000..1f860f2a --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/ch2/compute.py @@ -0,0 +1,17 @@ +#!/usr/bin/env python3 + +import argparse +import time + +parser = argparse.ArgumentParser(description="compute for seconds") +parser.add_argument( + "integer", + metavar="S", + type=int, + help="an integer for the number of seconds to compute", +) + +args = parser.parse_args() + +print("Will compute for " + str(args.integer) + " seconds.") +time.sleep(args.integer) diff --git a/2026/HPCIC-AWS/tutorial/ch3/03_flux_tutorial.ipynb b/2026/HPCIC-AWS/tutorial/ch3/03_flux_tutorial.ipynb new file mode 100644 index 00000000..c9f49c9a --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/ch3/03_flux_tutorial.ipynb @@ -0,0 +1,589 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "
\n", + "
\n", + "
\n", + "\n", + "# Chapter 3: Process, Monitoring, Utilities, and More, Oh My!\n", + "\n", + "Now that we have learned about basic flux commands, and hierarchical scheduling and its benefits, let's dive deeper into the structure of the individual Flux instances that comprise a hierarchy and talk about some additional \"plumbing\" that helps Flux to run. In this module, we cover:\n", + "\n", + "1. More advanced flux commands for querying data\n", + "2. The structure of Flux instances\n", + "3. Examples `flux kvs` that powers a lot of higher level commands\n", + "4. Advanced job specification interaction with flux job\n", + "
\n", + "\n", + "If you aren't interested in Flux plumbing, feel free to skip to [Chapter 4](../ch4/04_flux_framework_usernetes.ipynb) to run Flux alongside user-space Kubernetes!" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "tags": [] + }, + "source": [ + "# Process, Monitoring, and Job Utilities ⚙️\n", + "## flux exec 👊️\n", + "\n", + "
\n", + "Description: Executing commands across ranks\n", + "
\n", + "\n", + "Have you ever wanted a quick way to execute a command to all of your nodes in a flux instance? It might be to create a directory, or otherwise interact with a file. This can be hugely useful in environments where you don't have a shared filesystem, for example. This is a job for flux exec! Here is a toy example to execute the command to every rank (`-r all`) to print.\n", + "\n", + "```bash\n", + "flux exec -r all echo \"Hello from a flux rank!\"\n", + "```\n", + "\n", + "You can also use `-x` to exclude ranks. For example, we often do custom actions on the main or \"leader\" rank, and just want to issue commands to the workers.\n", + "\n", + "```bash\n", + "flux exec -r all -x 0 echo \"Hello from everyone except the lead (0) rank!\"\n", + "```\n", + "\n", + "Here is a similar example, but asking to execute only on rank 2, and to have it print the rank.\n", + "\n", + "```bash\n", + "flux exec -r 2 flux getattr rank \n", + "```\n", + "\n", + "And of course, we could do the same to print for all ranks! This is a derivative of the first example we showed you.\n", + "\n", + "```bash\n", + "flux exec flux getattr rank\n", + "```\n", + "\n", + "You can imagine that `flux exec` is hugely useful in the context of batch jobs and machine learning leader/worker designs that need different commands to start or interact with component nodes." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## flux jobs\n", + "\n", + "
\n", + "Description: Querying all jobs on a cluster\n", + "
\n", + "\n", + "Flux provides a way to query information about all of the jobs on a cluster, which was discussed in Chapter 1. We sort of skipped over the powerful ways through which you can customize this command. As a reminder, the basic usage to see current and \"all\" jobs:\n", + "\n", + "```bash\n", + "# Show me currently active jobs\n", + "flux jobs\n", + "\n", + "# Show me ALL jobs\n", + "flux jobs -a\n", + "```\n", + "\n", + "You can remove the header and ask for a format string to get exact information. Here is how to get a list of status:\n", + "\n", + "```bash\n", + "flux jobs -a --no-header --format=\"{status}\"\n", + "\n", + "# Get one status\n", + "flux jobs --no-header --format=\"{status}\" $(flux job last)\n", + "```\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## flux job info\n", + "\n", + "
\n", + "Description: Querying information about a Flux job\n", + "
\n", + "\n", + "Flux provides a way to get information from a JOBID, like jobspec, eventlog, and R (resource set).\n", + "\n", + "```bash\n", + "echo \"Querying the jobspec\"\n", + "flux job info $(flux job last) jobspec | jq .resources\n", + "echo \"Querying the resource set\"\n", + "flux job info $(flux job last) R | jq .execution.nodelist\n", + "Querying the jobspec\n", + "[\n", + " {\n", + " \"type\": \"node\",\n", + " \"count\": 1,\n", + " \"with\": [\n", + " {\n", + " \"type\": \"slot\",\n", + " \"count\": 64,\n", + " \"with\": [\n", + " {\n", + " \"type\": \"core\",\n", + " \"count\": 1\n", + " }\n", + " ],\n", + " \"label\": \"task\"\n", + " }\n", + " ]\n", + " }\n", + "]\n", + "Querying the resource set\n", + "[\n", + " \"ip-10-0-25-10\"\n", + "]\n", + "```\n", + "\n", + "We can query the job event log:\n", + "\n", + "```bash\n", + "flux job eventlog -H $(flux job last)\n", + "[Aug09 01:10] submit userid=1000 urgency=16 flags=0 version=1\n", + "[ +0.011880] validate\n", + "[ +0.022558] depend\n", + "[ +0.022598] priority priority=16\n", + "[ +0.024352] alloc\n", + "[ +0.026081] start\n", + "[ +0.042162] memo uri=\"ssh://ip-10-0-25-10/tmp/flux-JE2tV6/local-0\"\n", + "[ +1.491008] finish status=0\n", + "[ +1.491826] release ranks=\"all\" final=true\n", + "[ +1.491856] free\n", + "[ +1.491869] clean\n", + "```\n", + "Or the exec event log:\n", + "\n", + "```bash\n", + "[Aug09 01:10] init\n", + "[ +0.009753] shell.init service=\"1000-shell-f31zWKSBmy\" leader-rank=0 size=1\n", + "[ +0.011289] shell.start taskmap={\"version\":1,\"map\":[[0,1,1,1]]}\n", + "[ +0.000687] starting\n", + "[ +1.464679] shell.task-exit localid=0 rank=0 state=\"Exited\" pid=72151 wait_status=0 signaled=0 exitcode=0\n", + "[ +1.466305] complete status=0\n", + "[ +1.466328] done\n", + "```" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## flux uptime\n", + "\n", + "
\n", + "Description: Showing how long a flux instance has been running\n", + "
\n", + "\n", + "Flux provides an `uptime` utility to display properties of the Flux instance such as state of the current instance, how long it has been running, its size and if scheduling is disabled or stopped. The output shows how long the instance has been up, the instance owner, the instance depth (depth in the Flux hierarchy), and the size of the instance (number of brokers).\n", + "\n", + "```bash\n", + "flux uptime\n", + " 01:52:57 run 5h, owner ubuntu, depth 0, size 1\n", + "```" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## flux top \n", + "\n", + "
\n", + "Description: Showing a table of real-time Flux processes\n", + "
\n", + "\n", + "Flux provides a feature-full version of `top` for nested Flux instances and jobs. Try it out! \n", + "\n", + "```bash\n", + "flux top\n", + "```\n", + "\n", + "If you don't have any jobs running, run a few `flux submit sleep inf` to see output. \n", + "\n", + "## flux proxy\n", + "\n", + "
\n", + "Description: Interacting with a job hierarchy\n", + "
\n", + "\n", + "Flux proxy is used to route messages to and from a Flux instance. We can use `flux proxy` to connect to a running Flux instance and then submit more nested jobs inside it. This can work with flux uris that start with `ssh://` or even local files! Let's do a dummy example to reconnect to our own uri:\n", + "\n", + "```bash\n", + "flux proxy $FLUX_URI flux resource list\n", + "```\n", + "\n", + "In the case that was another (remote) Flux instance or a batch job, we can easily execute commands." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## flux queue\n", + "\n", + "
\n", + "Description: Interacting with and inspecting Flux queues\n", + "
\n", + "\n", + "Flux has a command for controlling the queue within the `job-manager`: `flux queue`. This includes disabling job submission, re-enabling it, waiting for the queue to become idle or empty, and checking the queue status:\n", + "\n", + "```bash\n", + "flux queue enable\n", + "flux queue -h\n", + "```" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## flux getattr\n", + "\n", + "
\n", + "Description: Getting attributes about your system and environment\n", + "
\n", + "\n", + "Each Flux instance has a set of attributes that are set at startup that affect the operation of Flux, such as `rank`, `size`, and `local-uri` (the Unix socket usable for communicating with Flux). Many of these attributes can be modified at runtime, such as `log-stderr-level` (1 logs only critical messages to stderr while 7 logs everything, including debug messages). Here is an example set that you might be interested in looking at:\n", + "\n", + "```bash\n", + "# What is the rank of the node I'm sitting on?\n", + "flux getattr rank\n", + "\n", + "# What is the size of the Flux instance?\n", + "flux getattr size\n", + "\n", + "# What is the local uri?\n", + "flux getattr local-uri\n", + "\n", + "# Set an attribute for the log level\n", + "flux setattr log-stderr-level 3\n", + "\n", + "# List current attributes\n", + "flux lsattr -v\n", + "```" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## flux dmesg\n", + "\n", + "
\n", + "Description: Viewing Flux system messages\n", + "
\n", + "\n", + "\n", + "If you need some additional help debugging your Flux setup, you might be interested in `flux dmesg`, which is akin to the [Linux dmesg](https://man7.org/linux/man-pages/man1/dmesg.1.html) but delivers messages for Flux." + ] + }, + { + "cell_type": "code", + "execution_count": 58, + "metadata": { + "scrolled": true, + "tags": [] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\u001b[1m\u001b[32m[May21 19:51]\u001b[0m \u001b[33mbroker.debug[0]\u001b[0m: \u001b[34minsmod connector-local\u001b[0m\n", + "\u001b[32m[ +0.005402]\u001b[0m \u001b[33mbroker.info[0]\u001b[0m: start: none->join 12.9368ms\u001b[0m\n", + "\u001b[32m[ +0.008454]\u001b[0m \u001b[33mbroker.info[0]\u001b[0m: parent-none: join->init 2.93317ms\u001b[0m\n", + "\u001b[32m[ +0.027397]\u001b[0m \u001b[33mconnector-local.debug[0]\u001b[0m: \u001b[34mallow-guest-user=false\u001b[0m\n", + "\u001b[32m[ +0.027451]\u001b[0m \u001b[33mconnector-local.debug[0]\u001b[0m: \u001b[34mallow-root-owner=false\u001b[0m\n", + "\u001b[32m[ +0.041013]\u001b[0m \u001b[33mbroker.debug[0]\u001b[0m: \u001b[34maccepting connection from a100cf62dc4f (rank 3) status full\u001b[0m\n", + "\u001b[32m[ +0.041447]\u001b[0m \u001b[33mbroker.debug[0]\u001b[0m: \u001b[34maccepting connection from a100cf62dc4f (rank 1) status full\u001b[0m\n", + "\u001b[32m[ +0.041517]\u001b[0m \u001b[33mbroker.debug[0]\u001b[0m: \u001b[34maccepting connection from a100cf62dc4f (rank 2) status full\u001b[0m\n", + "\u001b[32m[ +0.133438]\u001b[0m \u001b[33mbroker.debug[0]\u001b[0m: \u001b[34minsmod content\u001b[0m\n", + "\u001b[32m[ +0.170387]\u001b[0m \u001b[33mbroker.debug[0]\u001b[0m: \u001b[34minsmod barrier\u001b[0m\n", + "\u001b[32m[ +0.284711]\u001b[0m \u001b[33mbroker.debug[0]\u001b[0m: \u001b[34minsmod content-sqlite\u001b[0m\n", + "\u001b[32m[ +0.312658]\u001b[0m \u001b[33mcontent-sqlite.debug[0]\u001b[0m: \u001b[34m/tmp/flux-nygpcr/content.sqlite (0 objects) journal_mode=OFF synchronous=OFF\u001b[0m\n", + "\u001b[32m[ +0.313154]\u001b[0m \u001b[33mcontent.debug[0]\u001b[0m: \u001b[34mcontent backing store: enabled content-sqlite\u001b[0m\n", + "\u001b[32m[ +0.350060]\u001b[0m \u001b[33mbroker.debug[0]\u001b[0m: \u001b[34minsmod kvs\u001b[0m\n", + "\u001b[32m[ +0.391564]\u001b[0m \u001b[33mbroker.debug[0]\u001b[0m: \u001b[34minsmod kvs-watch\u001b[0m\n", + "\u001b[32m[ +0.474022]\u001b[0m \u001b[33mbroker.debug[0]\u001b[0m: \u001b[34minsmod resource\u001b[0m\n", + "\u001b[32m[ +0.562086]\u001b[0m \u001b[33mbroker.debug[0]\u001b[0m: \u001b[34minsmod cron\u001b[0m\n", + "\u001b[32m[ +0.562626]\u001b[0m \u001b[33mcron.info[0]\u001b[0m: synchronizing cron tasks to event heartbeat.pulse\u001b[0m\n", + "\u001b[32m[ +0.599858]\u001b[0m \u001b[33mbroker.debug[0]\u001b[0m: \u001b[34minsmod job-manager\u001b[0m\n", + "\u001b[32m[ +0.604520]\u001b[0m \u001b[33mjob-manager.debug[0]\u001b[0m: \u001b[34mjobtap plugin .history registered method job-manager.history.get\u001b[0m\n", + "\u001b[32m[ +0.605755]\u001b[0m \u001b[33mjob-manager.debug[0]\u001b[0m: \u001b[34mjobtap plugin .post-event registered method job-manager.post-event.post\u001b[0m\n", + "\u001b[32m[ +0.607280]\u001b[0m \u001b[33mjob-manager.debug[0]\u001b[0m: \u001b[34mhousekeeping is not configured\u001b[0m\n", + "\u001b[32m[ +0.607973]\u001b[0m \u001b[33mjob-manager.info[0]\u001b[0m: restart: 0 jobs\u001b[0m\n", + "\u001b[32m[ +0.608288]\u001b[0m \u001b[33mjob-manager.info[0]\u001b[0m: restart: 0 running jobs\u001b[0m\n", + "\u001b[32m[ +0.608510]\u001b[0m \u001b[33mjob-manager.info[0]\u001b[0m: restart: checkpoint.job-manager not found\u001b[0m\n", + "\u001b[32m[ +0.608669]\u001b[0m \u001b[33mjob-manager.debug[0]\u001b[0m: \u001b[34mrestart: max_jobid=ƒ1\u001b[0m\n", + "\u001b[32m[ +0.645613]\u001b[0m \u001b[33mbroker.debug[0]\u001b[0m: \u001b[34minsmod job-info\u001b[0m\n", + "\u001b[32m[ +0.683151]\u001b[0m \u001b[33mbroker.debug[0]\u001b[0m: \u001b[34minsmod job-list\u001b[0m\n", + "\u001b[32m[ +0.750683]\u001b[0m \u001b[33mbroker.debug[0]\u001b[0m: \u001b[34minsmod job-ingest\u001b[0m\n", + "\u001b[32m[ +0.751926]\u001b[0m \u001b[33mjob-ingest.debug[0]\u001b[0m: \u001b[34mconfiguring validator with plugins=(null), args=(null) (enabled)\u001b[0m\n", + "\u001b[32m[ +0.756057]\u001b[0m \u001b[33mjob-ingest.debug[0]\u001b[0m: \u001b[34mfluid ts=1ms\u001b[0m\n", + "\u001b[32m[ +0.792923]\u001b[0m \u001b[33mbroker.debug[0]\u001b[0m: \u001b[34minsmod job-exec\u001b[0m\n", + "\u001b[32m[ +0.835145]\u001b[0m \u001b[33mbroker.debug[0]\u001b[0m: \u001b[34minsmod heartbeat\u001b[0m\n", + "\u001b[32m[ +0.843296]\u001b[0m \u001b[33mbroker.info[0]\u001b[0m: rc1.0: running /etc/flux/rc1.d/01-sched-fluxion\u001b[0m\n", + "\u001b[32m[ +0.937478]\u001b[0m \u001b[33mbroker.debug[0]\u001b[0m: \u001b[34minsmod sched-fluxion-resource\u001b[0m\n", + "\u001b[32m[ +0.937900]\u001b[0m \u001b[33msched-fluxion-resource.info[0]\u001b[0m: version 0.44.0-7-g34f30920\u001b[0m\n", + "\u001b[32m[ +0.939829]\u001b[0m \u001b[33msched-fluxion-resource.debug[0]\u001b[0m: \u001b[34mmod_main: resource module starting\u001b[0m\n", + "\u001b[32m[ +0.980168]\u001b[0m \u001b[33mbroker.debug[0]\u001b[0m: \u001b[34minsmod sched-fluxion-qmanager\u001b[0m\n", + "\u001b[32m[ +0.980394]\u001b[0m \u001b[33msched-fluxion-qmanager.info[0]\u001b[0m: version 0.44.0-7-g34f30920\u001b[0m\n", + "\u001b[32m[ +0.981404]\u001b[0m \u001b[33msched-fluxion-qmanager.debug[0]\u001b[0m: \u001b[34mservice_register\u001b[0m\n", + "\u001b[32m[ +0.982103]\u001b[0m \u001b[33msched-fluxion-qmanager.debug[0]\u001b[0m: \u001b[34menforced policy (queue=default): fcfs\u001b[0m\n", + "\u001b[32m[ +0.982430]\u001b[0m \u001b[33msched-fluxion-qmanager.debug[0]\u001b[0m: \u001b[34meffective queue params (queue=default): default\u001b[0m\n", + "\u001b[32m[ +0.982498]\u001b[0m \u001b[33msched-fluxion-qmanager.debug[0]\u001b[0m: \u001b[34meffective policy params (queue=default): default\u001b[0m\n", + "\u001b[32m[ +0.984002]\u001b[0m \u001b[33mbroker.info[0]\u001b[0m: rc1.0: running /etc/flux/rc1.d/02-cron\u001b[0m\n", + "\u001b[32m[ +1.378086]\u001b[0m \u001b[33mbroker.info[0]\u001b[0m: rc1.0: /etc/flux/rc1 Exited (rc=0) 1.4s\u001b[0m\n", + "\u001b[32m[ +1.378472]\u001b[0m \u001b[33mbroker.info[0]\u001b[0m: rc1-success: init->quorum 1.36994s\u001b[0m\n", + "\u001b[32m[ +1.481261]\u001b[0m \u001b[33mbroker.info[0]\u001b[0m: online: a100cf62dc4f (ranks 0)\u001b[0m\n", + "\u001b[32m[ +1.767881]\u001b[0m \u001b[33msched-fluxion-resource.warning[0]\u001b[0m: \u001b[1mcreate_reader: allowlist unsupported\u001b[0m\n", + "\u001b[32m[ +1.773598]\u001b[0m \u001b[33msched-fluxion-resource.debug[0]\u001b[0m: \u001b[34mresource graph datastore loaded with rv1exec reader\u001b[0m\n", + "\u001b[32m[ +1.773883]\u001b[0m \u001b[33msched-fluxion-resource.info[0]\u001b[0m: populate_resource_db: loaded resources from core's resource.acquire\u001b[0m\n", + "\u001b[32m[ +1.776881]\u001b[0m \u001b[33msched-fluxion-resource.debug[0]\u001b[0m: \u001b[34mresource status changed (rankset=[all] status=DOWN)\u001b[0m\n", + "\u001b[32m[ +1.776903]\u001b[0m \u001b[33msched-fluxion-resource.debug[0]\u001b[0m: \u001b[34mresource status changed (rankset=[0] status=UP)\u001b[0m\n", + "\u001b[32m[ +1.777069]\u001b[0m \u001b[33msched-fluxion-resource.debug[0]\u001b[0m: \u001b[34mmod_main: resource graph database loaded\u001b[0m\n", + "\u001b[32m[ +1.777577]\u001b[0m \u001b[33msched-fluxion-qmanager.debug[0]\u001b[0m: \u001b[34mhandshaking with sched-fluxion-resource completed\u001b[0m\n", + "\u001b[32m[ +1.777848]\u001b[0m \u001b[33mjob-manager.debug[0]\u001b[0m: \u001b[34mscheduler: hello +partial-ok\u001b[0m\n", + "\u001b[32m[ +1.778479]\u001b[0m \u001b[33mjob-manager.debug[0]\u001b[0m: \u001b[34mscheduler: ready unlimited\u001b[0m\n", + "\u001b[32m[ +1.778572]\u001b[0m \u001b[33msched-fluxion-qmanager.debug[0]\u001b[0m: \u001b[34mhandshaking with job-manager completed\u001b[0m\n", + "\u001b[32m[ +2.272310]\u001b[0m \u001b[33mbroker.info[0]\u001b[0m: online: a100cf62dc4f,a100cf62dc4f,a100cf62dc4f,a100cf62dc4f (ranks 0-3)\u001b[0m\n", + "\u001b[32m[ +2.272758]\u001b[0m \u001b[33mbroker.info[0]\u001b[0m: quorum-full: quorum->run 0.894045s\u001b[0m\n", + "\u001b[32m[ +2.272901]\u001b[0m \u001b[33msched-fluxion-resource.debug[0]\u001b[0m: \u001b[34mresource status changed (rankset=[1-3] status=UP)\u001b[0m\n", + "\u001b[1m\u001b[32m[May21 22:31]\u001b[0m \u001b[33mkvs.debug[0]\u001b[0m: \u001b[34maggregated 2 transactions (2 ops)\u001b[0m\n", + "\u001b[32m[ +0.000771]\u001b[0m \u001b[33mkvs.debug[0]\u001b[0m: \u001b[34maggregated 2 transactions (2 ops)\u001b[0m\n", + "\u001b[32m[ +0.003297]\u001b[0m \u001b[33mkvs.debug[0]\u001b[0m: \u001b[34maggregated 2 transactions (2 ops)\u001b[0m\n", + "\u001b[32m[ +23.375291]\u001b[0m \u001b[33mkvs.debug[0]\u001b[0m: \u001b[34maggregated 3 transactions (3 ops)\u001b[0m\n", + "\u001b[32m[ +23.377798]\u001b[0m \u001b[33mkvs.debug[0]\u001b[0m: \u001b[34maggregated 2 transactions (2 ops)\u001b[0m\n", + "\u001b[32m[ +28.256627]\u001b[0m \u001b[33mkvs.debug[0]\u001b[0m: \u001b[34maggregated 3 transactions (3 ops)\u001b[0m\n", + "\u001b[32m[ +28.259229]\u001b[0m \u001b[33mkvs.debug[0]\u001b[0m: \u001b[34maggregated 2 transactions (2 ops)\u001b[0m\n", + "\u001b[32m[ +28.397358]\u001b[0m \u001b[33mkvs.debug[0]\u001b[0m: \u001b[34maggregated 4 transactions (4 ops)\u001b[0m\n", + "\u001b[1m\u001b[32m[May21 22:32]\u001b[0m \u001b[33mkvs.debug[0]\u001b[0m: \u001b[34maggregated 3 transactions (3 ops)\u001b[0m\n", + "\u001b[32m[ +0.004236]\u001b[0m \u001b[33mkvs.debug[0]\u001b[0m: \u001b[34maggregated 3 transactions (3 ops)\u001b[0m\n", + "\u001b[32m[ +28.174461]\u001b[0m \u001b[33mkvs.debug[0]\u001b[0m: \u001b[34maggregated 3 transactions (3 ops)\u001b[0m\n", + "\u001b[32m[ +28.176897]\u001b[0m \u001b[33mkvs.debug[0]\u001b[0m: \u001b[34maggregated 2 transactions (2 ops)\u001b[0m\n", + "\u001b[32m[ +37.720830]\u001b[0m \u001b[33mkvs.debug[0]\u001b[0m: \u001b[34maggregated 3 transactions (3 ops)\u001b[0m\n", + "\u001b[32m[ +37.723531]\u001b[0m \u001b[33mkvs.debug[0]\u001b[0m: \u001b[34maggregated 3 transactions (3 ops)\u001b[0m\n", + "\u001b[32m[ +37.853998]\u001b[0m \u001b[33mkvs.debug[0]\u001b[0m: \u001b[34maggregated 2 transactions (2 ops)\u001b[0m\n", + "\u001b[1m\u001b[32m[May21 22:33]\u001b[0m \u001b[33mkvs.debug[0]\u001b[0m: \u001b[34maggregated 3 transactions (3 ops)\u001b[0m\n", + "\u001b[32m[ +0.003143]\u001b[0m \u001b[33mkvs.debug[0]\u001b[0m: \u001b[34maggregated 3 transactions (3 ops)\u001b[0m\n", + "\u001b[32m[ +41.781505]\u001b[0m \u001b[33mkvs.debug[0]\u001b[0m: \u001b[34maggregated 3 transactions (3 ops)\u001b[0m\n", + "\u001b[32m[ +41.784053]\u001b[0m \u001b[33mkvs.debug[0]\u001b[0m: \u001b[34maggregated 2 transactions (2 ops)\u001b[0m\n", + "\u001b[1m\u001b[32m[May21 22:34]\u001b[0m \u001b[33mkvs.debug[0]\u001b[0m: \u001b[34maggregated 2 transactions (2 ops)\u001b[0m\n", + "\u001b[32m[ +0.000832]\u001b[0m \u001b[33mkvs.debug[0]\u001b[0m: \u001b[34maggregated 2 transactions (2 ops)\u001b[0m\n", + "\u001b[32m[ +0.003005]\u001b[0m \u001b[33mkvs.debug[0]\u001b[0m: \u001b[34maggregated 2 transactions (2 ops)\u001b[0m\n", + "\u001b[1m\u001b[32m[May21 22:36]\u001b[0m \u001b[33mkvs.debug[0]\u001b[0m: \u001b[34maggregated 2 transactions (2 ops)\u001b[0m\n", + "\u001b[32m[ +0.001909]\u001b[0m \u001b[33mkvs.debug[0]\u001b[0m: \u001b[34maggregated 2 transactions (2 ops)\u001b[0m\n", + "\u001b[32m[ +0.003307]\u001b[0m \u001b[33mkvs.debug[0]\u001b[0m: \u001b[34maggregated 2 transactions (2 ops)\u001b[0m\n", + "\u001b[1m\u001b[32m[May21 22:46]\u001b[0m \u001b[33mkvs.debug[0]\u001b[0m: \u001b[34maggregated 5 transactions (5 ops)\u001b[0m\n", + "\u001b[32m[ +0.000988]\u001b[0m \u001b[33mkvs.debug[0]\u001b[0m: \u001b[34maggregated 4 transactions (4 ops)\u001b[0m\n", + "\u001b[32m[ +0.003171]\u001b[0m \u001b[33mkvs.debug[0]\u001b[0m: \u001b[34maggregated 4 transactions (4 ops)\u001b[0m\n", + "\u001b[32m[ +0.004043]\u001b[0m \u001b[33mkvs.debug[0]\u001b[0m: \u001b[34maggregated 3 transactions (3 ops)\u001b[0m\n", + "\u001b[32m[ +3.195327]\u001b[0m \u001b[33mkvs.debug[0]\u001b[0m: \u001b[34maggregated 2 transactions (2 ops)\u001b[0m\n", + "\u001b[32m[ +22.074918]\u001b[0m \u001b[33mkvs.debug[0]\u001b[0m: \u001b[34maggregated 4 transactions (4 ops)\u001b[0m\n", + "\u001b[32m[ +22.077063]\u001b[0m \u001b[33mkvs.debug[0]\u001b[0m: \u001b[34maggregated 3 transactions (3 ops)\u001b[0m\n", + "\u001b[1m\u001b[32m[May21 22:47]\u001b[0m \u001b[33mkvs.debug[0]\u001b[0m: \u001b[34maggregated 2 transactions (2 ops)\u001b[0m\n", + "\u001b[32m[ +0.001051]\u001b[0m \u001b[33mkvs.debug[0]\u001b[0m: \u001b[34maggregated 2 transactions (2 ops)\u001b[0m\n", + "\u001b[32m[ +0.002549]\u001b[0m \u001b[33mkvs.debug[0]\u001b[0m: \u001b[34maggregated 2 transactions (2 ops)\u001b[0m\n", + "\u001b[1m\u001b[32m[May22 01:08]\u001b[0m \u001b[33mkvs.debug[0]\u001b[0m: \u001b[34maggregated 2 transactions (2 ops)\u001b[0m\n", + "\u001b[1m\u001b[32m[May22 01:13]\u001b[0m \u001b[33mjob-exec.debug[0]\u001b[0m: \u001b[34mexec aborted: id=ƒ3EFFE8Usy\u001b[0m\n", + "\u001b[32m[ +0.000775]\u001b[0m \u001b[33mjob-exec.debug[0]\u001b[0m: \u001b[34mexec aborted: id=ƒ3EFFFcUAK\u001b[0m\n", + "\u001b[32m[ +0.000802]\u001b[0m \u001b[33mjob-exec.debug[0]\u001b[0m: \u001b[34mexec aborted: id=ƒ3EFFFcUAL\u001b[0m\n", + "\u001b[32m[ +0.017790]\u001b[0m \u001b[33mkvs.debug[0]\u001b[0m: \u001b[34maggregated 2 transactions (2 ops)\u001b[0m\n", + "\u001b[1m\u001b[32m[May22 01:15]\u001b[0m \u001b[33mbroker.debug[0]\u001b[0m: \u001b[34mrmmod sched-fluxion-qmanager\u001b[0m\n", + "\u001b[32m[ +0.001020]\u001b[0m \u001b[33mbroker.debug[0]\u001b[0m: \u001b[34mmodule sched-fluxion-qmanager exited\u001b[0m\n", + "\u001b[32m[ +0.002652]\u001b[0m \u001b[33mjob-manager.debug[0]\u001b[0m: \u001b[34malloc: stop due to disconnect: Success\u001b[0m\n", + "\u001b[32m[ +0.002941]\u001b[0m \u001b[33msched-fluxion-resource.debug[0]\u001b[0m: \u001b[34mdisconnect_request_cb: a notify request aborted\u001b[0m\n", + "\u001b[32m[ +10.537435]\u001b[0m \u001b[33mbroker.debug[0]\u001b[0m: \u001b[34minsmod sched-simple\u001b[0m\n", + "\u001b[32m[ +10.538173]\u001b[0m \u001b[33msched-simple.debug[0]\u001b[0m: \u001b[34mservice_register\u001b[0m\n", + "\u001b[32m[ +10.538417]\u001b[0m \u001b[33msched-simple.err[0]\u001b[0m: \u001b[31mFailed to register feasibility service: File exists\u001b[0m\n", + "\u001b[32m[ +10.539072]\u001b[0m \u001b[33msched-simple.crit[0]\u001b[0m: \u001b[1m\u001b[31mmodule exiting abnormally\u001b[0m\n", + "\u001b[32m[ +10.539999]\u001b[0m \u001b[33mbroker.debug[0]\u001b[0m: \u001b[34mmodule sched-simple exited\u001b[0m\n", + "\u001b[32m[ +19.767014]\u001b[0m \u001b[33mbroker.debug[0]\u001b[0m: \u001b[34minsmod sched-simple\u001b[0m\n", + "\u001b[32m[ +19.767757]\u001b[0m \u001b[33msched-simple.debug[0]\u001b[0m: \u001b[34mservice_register\u001b[0m\n", + "\u001b[32m[ +19.767903]\u001b[0m \u001b[33msched-simple.err[0]\u001b[0m: \u001b[31mFailed to register feasibility service: File exists\u001b[0m\n", + "\u001b[32m[ +19.768274]\u001b[0m \u001b[33msched-simple.crit[0]\u001b[0m: \u001b[1m\u001b[31mmodule exiting abnormally\u001b[0m\n", + "\u001b[32m[ +19.768464]\u001b[0m \u001b[33mbroker.debug[0]\u001b[0m: \u001b[34mmodule sched-simple exited\u001b[0m\n", + "\u001b[32m[ +30.672673]\u001b[0m \u001b[33mbroker.debug[0]\u001b[0m: \u001b[34mrmmod sched-fluxion-resource\u001b[0m\n", + "\u001b[32m[ +30.673611]\u001b[0m \u001b[33mbroker.debug[0]\u001b[0m: \u001b[34mmodule sched-fluxion-resource exited\u001b[0m\n", + "\u001b[32m[ +30.673747]\u001b[0m \u001b[33mresource.debug[0]\u001b[0m: \u001b[34maborted 1 resource.acquire(s)\u001b[0m\n", + "\u001b[1m\u001b[32m[May22 01:16]\u001b[0m \u001b[33mbroker.debug[0]\u001b[0m: \u001b[34minsmod sched-simple\u001b[0m\n", + "\u001b[32m[ +0.000757]\u001b[0m \u001b[33msched-simple.debug[0]\u001b[0m: \u001b[34mservice_register\u001b[0m\n", + "\u001b[32m[ +0.004032]\u001b[0m \u001b[33msched-simple.debug[0]\u001b[0m: \u001b[34mresource update: {\"resources\":{\"version\":1,\"execution\":{\"R_lite\":[{\"rank\":\"0-3\",\"children\":{\"core\":\"0-9\"}}],\"starttime\":0.0,\"expiration\":0.0,\"nodelist\":[\"a100cf62dc4f,a100cf62dc4f,a100cf62dc4f,a100cf62dc4f\"]}},\"up\":\"0-3\"}\u001b[0m\n", + "\u001b[32m[ +0.004357]\u001b[0m \u001b[33mjob-manager.debug[0]\u001b[0m: \u001b[34mscheduler: hello +partial-ok\u001b[0m\n", + "\u001b[32m[ +0.004577]\u001b[0m \u001b[33mjob-manager.debug[0]\u001b[0m: \u001b[34mscheduler: ready limited\u001b[0m\n", + "\u001b[32m[ +0.006012]\u001b[0m \u001b[33msched-simple.debug[0]\u001b[0m: \u001b[34mready: 40 of 40 cores: rank[0-3]/core[0-9]\u001b[0m\n" + ] + } + ], + "source": [ + "!flux dmesg -H" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "tags": [] + }, + "source": [ + "
\n", + "\n", + "## Flux Module\n", + "\n", + "
\n", + "Description: Manager and query Flux modules.\n", + "
\n", + "\n", + "\n", + "To manage and query modules, Flux provides the `flux module` command. This is more of an advanced use case. The sub-commands provided by `flux module` can be seen by running:\n", + "\n", + "```bash\n", + "flux module --help\n", + "```\n", + "\n", + "We can see that these services are loaded and available by running:\n", + "\n", + "```bash\n", + "flux module list\n", + "```\n", + "\n", + "Some examples of Flux modules include:\n", + "* `job-ingest` (used by Flux submission commands like `flux batch` and `flux run`)\n", + "* `job-list` (used by `flux jobs`)\n", + "* `sched-fluxion-qmanager` (used by `flux tree`)\n", + "* `sched-fluxion-resource` (also used by `flux tree`)\n", + "\n", + "Try looking at job manager stats, and then using `jq` to get a specific value.\n", + "\n", + "```bash\n", + "flux module stats job-manager\n", + "flux module stats job-manager | jq .inactive_jobs\n", + "```\n", + "\n", + "Users and system administrators can easily load and unload modules using the `flux module load` and `flux module remove` commands. " + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### flux kvs\n", + "\n", + "
\n", + "Description: The Flux key value store.\n", + "
\n", + "\n", + "One of the core services built into Flux is the key-value store (KVS). It is used in many other services, including most of Flux's resource management services, and the `flux archive` service below.\n", + "\n", + "The `flux kvs` command provides a utility to list and manipulate values of the KVS. As a example of using `flux kvs`, let's use the command to examine information saved by the `resource` service.\n", + "\n", + "```bash\n", + "flux kvs ls\n", + "flux kvs ls resource\n", + "flux kvs get resource.R | jq\n", + "```\n", + "\n", + "The KVS is such an essential component of Flux that we provide C and Python APIs to interact with it. To learn more about interacting with the KVS from these languages, take a look at these documentation pages:\n", + "* C's `flux_kvs_commit` [family of functions](https://flux-framework.readthedocs.io/projects/flux-core/en/latest/man3/flux_kvs_commit.html)\n", + "* C's `flux_kvs_copy` [family of functions](https://flux-framework.readthedocs.io/projects/flux-core/en/latest/man3/flux_kvs_copy.html)\n", + "* C's `flux_kvs_getroot` [family of functions](https://flux-framework.readthedocs.io/projects/flux-core/en/latest/man3/flux_kvs_getroot.html)\n", + "* C's `flux_kvs_lookup` [family of functions](https://flux-framework.readthedocs.io/projects/flux-core/en/latest/man3/flux_kvs_lookup.html)\n", + "* C's `flux_kvs_namespace_create` [family of functions](https://flux-framework.readthedocs.io/projects/flux-core/en/latest/man3/flux_kvs_namespace_create.html)\n", + "* C's `flux_kvs_txn_create` [family of functions](https://flux-framework.readthedocs.io/projects/flux-core/en/latest/man3/flux_kvs_txn_create.html)\n", + "* Python's `flux.kvs` [module](https://flux-framework.readthedocs.io/projects/flux-core/en/latest/python/autogenerated/flux.kvs.html#module-flux.kvs)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "## a humorous example of writing to the Flux KVS using the python bindings\n", + "from functools import partial\n", + "\n", + "import flux\n", + "import flux.future\n", + "import flux.kvs\n", + "\n", + "handle = flux.Flux()\n", + "\n", + "def we_await_donuts(future):\n", + " print(\"We await donuts\")\n", + " with flux.kvs.KVSTxn(flux_handle=handle) as kt:\n", + " kt.mkdir(\"donuts\")\n", + " kt.put(\"donuts.old_fashioned\", \"best\")\n", + " kt.put(\"donuts.plain_raised\", \"good ol' standby\")\n", + " kt.put(\"donuts.apple_fritter\", \"excellent alternative\")\n", + "\n", + "def samir_brought_donuts(hand, watcher, revents, args, future):\n", + " ## Fetch donut ratings from the flux key-value store\n", + " donuts = flux.kvs.KVSDir(hand, \".donuts\")\n", + " for donut in donuts.files():\n", + " print(f'{donut}: {flux.kvs.get(hand, donuts.key_at(donut))}')\n", + " future.fulfill()\n", + "\n", + "def donuts_are_here(future):\n", + " print(\"The donut future has been fulfilled\")\n", + "\n", + "## Create a future for the donuts on our flux handle\n", + "donut_future = flux.future.FutureExt(we_await_donuts, flux_handle=handle)\n", + "\n", + "## When the donut future (promise) is fulfilled, we tell everyone the donuts are here\n", + "donut_future.then(donuts_are_here)\n", + "\n", + "## We schedule the donut future to be fulfilled in 1 minute \n", + "donut_watcher = handle.timer_watcher_create(6, partial(samir_brought_donuts, future=donut_future))\n", + "\n", + "donut_watcher.start()\n", + "\n", + "handle.reactor_run()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## flux archive 📚️\n", + "\n", + "
\n", + "Description: Creating file and content archives to access later and between ranks\n", + "
\n", + "\n", + "As Flux is used more in cloud environments, we might find ourselves in a situation where we have a cluster without a shared filesystem. The `flux archive` command helps with this situation. At a high level, `flux archive` allows us to save named pieces of data (e.g., files) to the Flux KVS for later retrieval.\n", + "\n", + "When using `flux archive`, we first have to create an named archive. In the code below, we will create a text file and then save it into an archive using `flux archive`. Note that, for larger files, you can speed up the creation and extraction of archives by using the `--mmap` flag." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "tags": [] + }, + "source": [ + "# This concludes Chapter 3.\n", + "\n", + "You can next learn about running AI/ML workloads with Flux and User-space Kubernetes in [Chapter 4](../ch4/04_flux_framework_usernetes.ipynb)" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.12.3" + } + }, + "nbformat": 4, + "nbformat_minor": 4 +} diff --git a/2026/HPCIC-AWS/tutorial/ch4/04_flux_framework_usernetes.ipynb b/2026/HPCIC-AWS/tutorial/ch4/04_flux_framework_usernetes.ipynb new file mode 100644 index 00000000..6cbc9119 --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/ch4/04_flux_framework_usernetes.ipynb @@ -0,0 +1,453 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "1a7f998b-6956-4bbc-b224-17fab1597e8f", + "metadata": {}, + "source": [ + "
\n", + "
\n", + "
\n", + "\n", + "# Chapter 4: Flux with User-space Kubernetes\n", + "\n", + "We are developing a new setup where it is possible to run Flux alongside user-space Kubernetes, or Usernetes.\n", + "This is a small demo of what that setup affords. In this tutorial, we will:\n", + "1. Start Usernetes\n", + "2. Run LAMMPS with Flux\n", + "3. Run an AI/ML training Job \n", + "4. Run simulations alongside an AI/ML Model Server\n", + "5. Run MuMMI Components with the Flux Operator (Flux -> Kubernetes -> Flux)\n", + "
\n", + "\n", + "Conceptually, this means that you can run traditional workloads using your HPC workload manager, and then use Kubernetes for associated services for AI/ML models, databases, and message queues.\n", + "\n", + "![img/usernetes-flux.png](img/usernetes-flux.png)\n", + "\n", + "If you haven't yet, open up a Jupyter terminal and put it alongside this notebook so you can copy paste commands into it." + ] + }, + { + "cell_type": "markdown", + "id": "a2ea7230-96f9-48af-85b7-10942406bc0a", + "metadata": { + "jp-MarkdownHeadingCollapsed": true + }, + "source": [ + "## 1. Start Usernetes\n", + "\n", + "Normally, we start Usernetes under a Flux batch job, meaning you (the user) do not have to do it, and do not see it. Here, we are going to show you all the setup. Run the following script in the terminal to your left to watch your control plane come up!\n", + "\n", + "```bash\n", + "bash /home/ubuntu/start-usernetes.sh\n", + "```\n", + "\n", + "When the script finishes, you'd normally export the `KUBECONFIG` to your environment. This is your credentials to interact with the cluster. However, you don't need to because we write the default to `~/.kube/config`.\n", + "\n", + "```bash\n", + "export KUBECONFIG=/home/ubuntu/usernetes/kubeconfig\n", + "```\n", + "\n", + "Try looking at the nodes:\n", + "\n", + "```bash\n", + "kubectl get nodes\n", + "```\n", + "\n", + "Right now you just have a single control plane, and we have modified it to allow for running work. This is how we map Usernetes nodes to physical nodes in HPC, with a 1:1 ratio. Finally, you can enable auto-complete (with TAB) for `kubectl`:\n", + "\n", + "```bash\n", + "source <((kubectl completion bash))\n", + "```\n", + "\n", + "Take a look at all the service pods! In Kubernetes we have the concept of [namespaces](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/) to organize things. By default we interact with `default`.\n", + "\n", + "```bash\n", + "kubectl get pods --all-namespaces \n", + "```\n", + "\n", + "\n", + "### Understanding the Setup\n", + "\n", + "You are running on a single AWS EC2 instance with 64 cores. It is a Graviton 3 (ARM) instance. Directly installed on that instance is Flux, and the Jupyter notebook was started with `flux start --test-size=4`, meaning we are already inside of a Flux instance to emulate a batch job. We can see our Flux resources:\n", + "\n", + "```bash\n", + "flux resource list\n", + " STATE NNODES NCORES NGPUS NODELIST\n", + " free 1 64 0 ip-10-0-25-10\n", + " allocated 0 0 0 \n", + " down 0 0 0 \n", + "```\n", + "\n", + "We have deployed user-space Kubernetes inside of the Flux instance (job) and we can see that one Usernetes \"node\" is mapped to our single instance node.\n", + "\n", + "```bash\n", + "kubectl get nodes -o wide\n", + "```\n", + "```console\n", + "NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME\n", + "u7s-ip-10-0-25-10 Ready control-plane 5h16m v1.33.1 10.0.25.10 Debian GNU/Linux 12 (bookworm) 6.8.0-1029-aws containerd://2.1.1\n", + "```\n", + "\n", + "If we had a job with multiple nodes, we would see a `control-plane` role along with worker roles! For our tutorial today we just have one control plane that will also function as a worker.\n", + "\n", + "\n", + "### Debugging Tips 🪲\n", + "\n", + "When something isn't working, here is a series of logical things to come back to look at. First, the pod logs, or metadata in YAML.\n", + "\n", + "```bash\n", + "# Pod logs, one shot\n", + "kubectl logs \n", + "\n", + "# Pod logs, dangling\n", + "kubectl logs -f\n", + "\n", + "# YAML manifest\n", + "kubectl get pod -o yaml\n", + "```\n", + "\n", + "You might also want to use `describe` to see events.\n", + "\n", + "```bash\n", + "kubectl describe pods \n", + "```\n", + "\n", + "Often there is an associated service not working. Look for it, and then again use logs and describe to see what might be going on.\n", + "\n", + "```bash\n", + "kubectl get pods --all-namespaces\n", + "```\n", + "\n", + "When you use a `Deployment` or `Job` or `Service` it is often helpful to do the equivalent `get` or `describe` for that abstraction." + ] + }, + { + "cell_type": "markdown", + "id": "98c4254f-6563-42ce-b4a0-366b6dab3d8d", + "metadata": { + "jp-MarkdownHeadingCollapsed": true + }, + "source": [ + "## 2. Run LAMMPS with Flux\n", + "\n", + "
\n", + "Description: Running LAMMPS with Flux in two ways 🍳 -- directly on the virtual machine, analogous to \"bare metal\" and then in user-space Kubernetes.\n", + "
\n", + "\n", + "LAMMPS -- the Large-scale Atomic/Molecular Massively Parallel Simulator -- is a widely used molecular dynamics (MD) open-source code. You can learn more about it [here](https://www.lammps.org). Let's start with a simple LAMMPS run, specifically LAMMPS with Flux Framework. This would be akin to sitting in an allocation you've created and interacting with your workload manager. First, inspect the resources that you have.\n", + "\n", + "```bash\n", + "flux resource list\n", + "```\n", + "\n", + "Run a LAMMPS job that uses them!\n", + "\n", + "```bash\n", + "flux run -N1 -n 64 -o cpu-affinity=per-task --cwd /opt/lammps/examples/reaxff/HNS lmp -v x 4 -v y 4 -v z 4 -in in.reaxff.hns -nocite\n", + "```\n", + "\n", + "Just as we learned in the previous tutorial, if you change `flux run` to `flux submit` the job will be non-blocking. You can also change the LAMMPS problem size to be larger to have a longer running time. \n", + "\n", + "### Flux in Kubernetes\n", + "Next, we are going to make a [turducken](https://en.wikipedia.org/wiki/Turducken) - running Flux in Kubernetes, which (in our HPC setups) is already running under Flux! \n", + "If we had more than one physical node, this would give us a powerful means to run HPC workloads in Kubernetes, with features that HPC cannot easily support such as elasticity and dynamism, declarative management, and modularity. \n", + "\n", + "\n", + " \n", + "\n", + " \n", + " \n", + "
Flux Framework and Usernetes setup. Your virtual machine is provisioned with both the workload manager Flux\n", + "Framework and User-space Kubernetes (1). We are emulating in this notebook you, the HPC user, running a batch job with Flux that has brought up your own user-space Kubernetes cluster (2). We will then deploy the Flux Operator (3) inside that cluster to run LAMMPS.
\n", + "\n", + "Let's install the Flux Operator. Note we are installing for ARM.\n", + "\n", + "```bash\n", + "kubectl apply -f https://raw.githubusercontent.com/flux-framework/flux-operator/refs/heads/main/examples/dist/flux-operator-arm.yaml\n", + "```\n", + "\n", + "If you haven't, make sure you CD to the chapter 4 tutorial directory.\n", + "\n", + "```bash\n", + "cd tutorial/ch4/\n", + "```\n", + "\n", + "Create the Flux MiniCluster, which is the Custom Resource Definition (CRD) that the Flux Operator manages. This will take about 2 minutes, 20 seconds to go from creation to running, and most of that time is to pull the image.\n", + "\n", + "```bash\n", + "kubectl apply -f flux-minicluster-lammps.yaml\n", + "```\n", + "\n", + "You can use `kubectl get pods` to watch the pods transition from `Init:0/1` (this is where we add Flux to the application container and configure the cluster) to `PodInitializing` (this is when your application container is being pulled) to `Running`. This entire sequence for this image takes approximately 3 minutes. Try using `--watch` to easily monitor for updates.\n", + "\n", + "```bash\n", + "kubectl get pods --watch\n", + "```\n", + "When you see `Running` you can press Control+C.\n", + "\n", + "Here is a way to use `kubectl get pods` with jq to programmatically get the lead broker pod identifier, which we can use to stream logs and watch LAMMPS output.\n", + "\n", + "```bash\n", + "lead_broker=$(kubectl get pods -o json | jq -r .items[0].metadata.name)\n", + "kubectl logs $lead_broker -f\n", + "```\n", + "\n", + "And when you are done, clean up the MiniCluster.\n", + "\n", + "```bash\n", + "kubectl delete -f ./flux-minicluster-lammps.yaml\n", + "```" + ] + }, + { + "cell_type": "markdown", + "id": "2e5aefd2-a0f0-4e90-9522-82589f614787", + "metadata": {}, + "source": [ + "## 3. Run an AI/ML Training Job\n", + "\n", + "
\n", + "Description: Running AI/ML training jobs in Kubernetes is a first class citizen.\n", + "
\n", + "\n", + "The newly released [Kubeflow Trainer](https://www.kubeflow.org/docs/components/trainer/getting-started/) project makes it easy to run Kubernetes components, specifically for Artificial Intelligence and Machine Learning workloads (AI/ML) in Kubernetes directly from Python. In your terminal to the left, use `kubectl` to install the Kubeflow Training Operator: \n", + "\n", + "```bash\n", + "# This is for JobSet and the Trainer Manager\n", + "VERSION=v2.0.0\n", + "kubectl apply --server-side -k \"https://github.com/kubeflow/trainer.git/manifests/overlays/manager?ref=${VERSION}\"\n", + "\n", + "# This is for the runtimes (we need to give the webhook some time to create)\n", + "sleep 5\n", + "kubectl apply --server-side -k \"https://github.com/kubeflow/trainer.git/manifests/overlays/runtimes?ref=${VERSION}\"\n", + "```\n", + "\n", + "If you get a `failed calling webhook` error, wait 30 seconds and try again. Hooks sometimes have a delay in starting up.\n", + "Next, let's run Mnist training. Mnist is a well-known [dataset of handwritten digits](https://en.wikipedia.org/wiki/MNIST_database) used for machine learning tasks. We will be using Kubeflow, and using YAML for this tutorial. Note that if you like Python, the entire interaction of using Kubeflow in Kubernetes can be done using the [Python SDK](https://www.kubeflow.org/docs/components/trainer/getting-started/)." + ] + }, + { + "cell_type": "markdown", + "id": "62c41285-4850-455d-a729-b1e06dda7ab0", + "metadata": { + "jp-MarkdownHeadingCollapsed": true + }, + "source": [ + "### Run MNIST\n", + "\n", + "While we can use the Python SDK for all our interactions, let's create the job the \"old school\" way - by applying a YAML file. Take a look at [pytorch-mnist.yaml](pytorch-mnist.yaml) and then in your terminal, run the job in your cluster by using `kubectl apply` with `-f` for a file.\n", + "\n", + "```bash\n", + "kubectl apply -f ./pytorch-mnist.yaml\n", + "```\n", + "\n", + "To see the pods, you can use `kubectl get` on the Pod resource type. Note that we will have two. Index 0 is the master, and 1 is the worker.\n", + "\n", + "```bash\n", + "kubectl get pods\n", + "\n", + "# More information about the hosts in \"output wide\" mode\n", + "kubectl get pods -o wide\n", + "```\n", + "\n", + "And then get the pod identifier and look at the output. The `-f` will keep the output streaming.\n", + "\n", + "```bash\n", + "kubectl logs pytorch-simple-node-0-0-xxxx -f\n", + "```\n", + "\n", + "When the logs appear to be done, check the pods to see they are `Completed`\n", + "\n", + "```bash\n", + "kubectl get pods\n", + "```\n", + "\n", + "When you are ready to clean up, just `kubectl delete` the same file. Note that a pod in `Completed` state does not consume resources.\n", + "\n", + "```bash\n", + "kubectl delete -f pytorch-mnist.yaml\n", + "```\n", + "\n", + "Congratulations - you just ran your first AI/ML Job in User-space Kubernetes!" + ] + }, + { + "cell_type": "markdown", + "id": "93bae8ea-99f5-4f33-bd0b-90fa80afc3e2", + "metadata": {}, + "source": [ + "## 4. Run Simulations alongside an AI/ML Model Server\n", + "\n", + "
\n", + "Description: Usernetes is powerful to provide a means to run AI/ML services alongside traditional HPC.\n", + "
\n", + "\n", + "The most likely use case for user-space Kubernetes working with HPC is supporting services for your job simulations or workloads. As an example, you might want to deploy a database, message queue, or model server. For this example, we will deploy a [streaming ML model server](https://github.com/converged-computing/lammps-stream-ml/) backed by [RiverML](https://riverml.xyz) that allows you to build models that predict LAMMPS running time from input parameters X, Y, and Z. \n", + "\n", + "### Setup of Server\n", + "\n", + "While there are many ways to expose service ports, we will use a simple means to enable ingress via Nginx.\n", + "\n", + "```bash\n", + "kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/kind/deploy.yaml\n", + "kubectl wait --namespace ingress-nginx \\\n", + " --for=condition=ready pod \\\n", + " --selector=app.kubernetes.io/component=controller \\\n", + " --timeout=90s\n", + "```\n", + "\n", + "The second command above ensures that the container is `Running` before we continue. Let's next deploy the ML server that will provide River models.\n", + "\n", + "```bash\n", + "kubectl apply -f ml-server-deployment.yaml\n", + "```\n", + "\n", + "Take a look at the service and ingress that are allowing for us to interact with the service, and when the ml-server-xxxx container is running, take a look at its logs to see the database being created and service starting.\n", + "\n", + "```bash\n", + "kubectl get service ml-service-endpoint\n", + "kubectl get ingress ml-server-ingress\n", + "```\n", + "\n", + "When you send a request to your node's public IP address on port 80, the flow is:\n", + "\n", + "```console\n", + "Your Request -> Host Port 80 -> Ingress NGINX Pod -> ml-service-endpoint (Service) -> ml-server Pod (on port 8080)\n", + "```\n", + "\n", + "There are a lot of layers! Importantly, when we run HPC applications like simulations inside of Usernetes, we use a host bypass mechanism (e.g., EFA or Infiniband) that will not be subject to the overhead added by [slirp4netns](https://arxiv.org/html/2402.00365v1).\n", + "\n", + "### Test Server\n", + "\n", + "Let's test it! The `/api` endpoint should deliver metadata about the service.\n", + "\n", + "```bash\n", + "curl -ks localhost/api/ | jq\n", + "```\n", + "```python\n", + "{\n", + " \"id\": \"django_river_ml\",\n", + " \"status\": \"running\",\n", + " \"name\": \"Django River ML Endpoint\",\n", + " \"description\": \"This service provides an api for models\",\n", + " \"documentationUrl\": \"https://vsoch.github.io/django-river-ml\",\n", + " \"storage\": \"shelve\",\n", + " \"river_version\": \"0.21.0\",\n", + " \"version\": \"0.0.21\"\n", + "}\n", + "```\n", + "\n", + "Awesome! Let's next create our models. We are going to create [three regression](https://riverml.xyz/latest/api/linear-model/LinearRegression/) types. You can take a look at [model-scripts/1-create-models.py](model-scripts/1-create-models.py) to see how we will do that. River is akin to scikit-learn and other libraries that can create models from scratch or load on demand. We are creating them from scratch.\n", + "\n", + "```bash\n", + "python3 model-scripts/1-create-models.py\n", + "```\n", + "\n", + "We are next going to train our models directly on bare metal. We will submit jobs to Flux to run LAMMPS. Each will randomly select problem sizes, run, and send the features (X, Y, and Z problem size) and the value we want to predict, Y (the running time) to all the models. in our server. The default number of iterations is 20, but of course, for a better model you might consider increasing it.\n", + "\n", + "```bash\n", + "# This is the default\n", + "python3 model-scripts/2-run-lammps-flux.py train --iters 20\n", + "\n", + "# This would be a better model (but take longer)\n", + "python3 model-scripts/2-run-lammps-flux.py train --iters 100\n", + "```\n", + "\n", + "Once your models are trained, let's create predictions to test. This will save an output file to your local machine that we can generate plots with.\n", + "\n", + "```bash\n", + "python3 model-scripts/2-run-lammps-flux.py predict --out test-predict.json\n", + "```\n", + "\n", + "### Example Results\n", + "\n", + "Here are metrics for models trained on `t4g.2xlarge` with a few hundred iterations.\n", + "\n", + "```console\n", + "⭐️ Performance for: angry-truffle\n", + " R Squared Error: 0.6087271917414379\n", + " Mean Squared Error: 4.0731499339716315\n", + " Mean Absolute Error: 1.6454122148608918\n", + " Root Mean Squared Error: 2.0182046313423303\n", + "\n", + "⭐️ Performance for: frigid-nalgas\n", + " R Squared Error: -1.146171741238755\n", + " Mean Squared Error: 22.34164782629544\n", + " Mean Absolute Error: 4.571366798469615\n", + " Root Mean Squared Error: 4.726695233066697\n", + "\n", + "⭐️ Performance for: gassy-nunchucks\n", + " R Squared Error: 0.8824891368445993\n", + " Mean Squared Error: 1.2232880854477222\n", + " Mean Absolute Error: 0.7817525756655499\n", + " Root Mean Squared Error: 1.1060235465159511\n", + "```" + ] + }, + { + "cell_type": "markdown", + "id": "26dfdea4-d270-48c4-a10d-c6804d4500a5", + "metadata": {}, + "source": [ + "## 5. Run MuMMI Components with the Flux Operator\n", + "\n", + "
\n", + "Description: Orchestrate MuMMI components with a an entire HPC cluster (1 node) in Kubernetes with the Flux Operator.\n", + "
\n", + "\n", + "This is an advanced example provided if you get through all of the above and want to learn more! For this step, we are going to use the Flux Operator to run an example component from a well-known complex workflow, MuMMI. \n", + "\n", + "\n", + "\n", + " \n", + "\n", + " \n", + " \n", + "
MuMMI and its mini variant use a custom workload manager to orchestrate simulations leveraging Flux and backed by a message queue and the filesystem. Running in Kubernetes, we use an OCI registry to share assets between nodes, and a state machine design instead of a message queue.
\n", + "\n", + "The components include:\n", + "\n", + "- `mlrunner`: This is a machine learning runner that generates simulation data to kick off a sequence of steps.\n", + "- `createsims`: Simulation setup that follows the machine learning step.\n", + "- `cganalysis`: Simulation (runs for 6 minutes with our configuration) but can take 12+ hours on HPC.\n", + "\n", + "We will run the `createsims` step, which provides interesting (and fun) output and will run for approximately 20 minutes.\n", + "\n", + "```bash\n", + "# Run the createsims step, the second step\n", + "kubectl apply -f flux-mummi-createsims.yaml\n", + "```\n", + "\n", + "The `mlrunner` and `cganalysis` steps included in [extra/mummi-components](extra/mummi-components) can be run if you are interested. The model provided in the `mlrunner` container takes longer (4m54s) to pull. The `mlrunner` will extract a model and run quickly (under 20 seconds), the `createsims` setup takes approximately 10 minutes, and `cganysis` is configured to run in ~6. For any component you choose, see logs like this:\n", + "\n", + "```bash\n", + "kubectl logs mlrunner-0-xxxx -f\n", + "kubectl logs createsims-0-xxxx -f\n", + "kubectl logs cganalysis-0-xxxx -f\n", + "```" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.12.3" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/2026/HPCIC-AWS/tutorial/ch4/extra/mummi-components/flux-mummi-cganalysis.yaml b/2026/HPCIC-AWS/tutorial/ch4/extra/mummi-components/flux-mummi-cganalysis.yaml new file mode 100644 index 00000000..fa75cbd9 --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/ch4/extra/mummi-components/flux-mummi-cganalysis.yaml @@ -0,0 +1,22 @@ +apiVersion: flux-framework.org/v1alpha2 +kind: MiniCluster +metadata: + name: cganalysis +spec: + size: 1 + # Setting tasks to 0 uses what is found on the node. + tasks: 0 + logging: + quiet: true + + flux: + container: + image: ghcr.io/converged-computing/flux-view-rocky:arm-9 + + containers: + - image: ghcr.io/converged-computing/sc25-flux-eks:cganalysis-arm + command: bash /entrypoint.sh + volumes: + shared-memory: + emptyDir: true + emptyDirMedium: "memory" diff --git a/2026/HPCIC-AWS/tutorial/ch4/extra/mummi-components/flux-mummi-mlrunner.yaml b/2026/HPCIC-AWS/tutorial/ch4/extra/mummi-components/flux-mummi-mlrunner.yaml new file mode 100644 index 00000000..0807260f --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/ch4/extra/mummi-components/flux-mummi-mlrunner.yaml @@ -0,0 +1,23 @@ + +apiVersion: flux-framework.org/v1alpha2 +kind: MiniCluster +metadata: + name: mlrunner +spec: + size: 1 + # Setting tasks to 0 uses what is found on the node (avoiding the flux default of 1) + tasks: 0 + logging: + quiet: true + + flux: + container: + image: ghcr.io/converged-computing/flux-view-rocky:arm-9 + + containers: + - image: ghcr.io/converged-computing/sc25-flux-eks:mlrunner-arm + command: /root/.cargo/bin/pixi run --manifest-path /pixi-env/pixi.toml bash /entrypoint.sh + volumes: + shared-memory: + emptyDir: true + emptyDirMedium: "memory" diff --git a/2026/HPCIC-AWS/tutorial/ch4/extra/usernetes-load.sh b/2026/HPCIC-AWS/tutorial/ch4/extra/usernetes-load.sh new file mode 100755 index 00000000..8960577c --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/ch4/extra/usernetes-load.sh @@ -0,0 +1,22 @@ +#!/bin/bash + +# Load a container into usernetes. +# This is intended for the HPCIC 25 tutorial, hence hard coded paths. + +URI=${1:-"ghcr.io/converged-computing/flux-view-rocky:arm-9"} +SAVE_URI=$(echo $URI | tr '/:' '_-').tar +mkdir -p /home/ubuntu/usernetes/images +docker pull $URI +echo "Saving ${URI} to ${SAVE_URI} for loading..." +docker save -o /home/ubuntu/usernetes/images/${SAVE_URI} ${URI} + +# Using the shared path (usernetes root) tell containerd to load it +echo "Importing ${URI} into usernetes node" +docker exec -it usernetes-node-1 ctr --namespace k8s.io image import /usernetes/images/${SAVE_URI} +docker exec -it usernetes-node-1 crictl images | grep ${URI} + +# Times hpc7g with g3 +# make up -> cluster 116 seconds +# pulling mlrunner: 5m55s to running +# pull to load lammps efa container: 7m +# just to load from .tar: 2m 59s. diff --git a/2026/HPCIC-AWS/tutorial/ch4/flux-minicluster-lammps.yaml b/2026/HPCIC-AWS/tutorial/ch4/flux-minicluster-lammps.yaml new file mode 100644 index 00000000..5d1e916c --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/ch4/flux-minicluster-lammps.yaml @@ -0,0 +1,34 @@ +apiVersion: flux-framework.org/v1alpha2 +kind: MiniCluster +metadata: + name: flux-sample +spec: + size: 1 + tasks: 64 + + # If quiet is false, we see all flux logging + logging: + quiet: true + + # The view should match the architecture of the application image. + # You aren't required to use a view for Flux + # You can also install to your application container + flux: + optionFlags: -o cpu-affinity=per-task + container: + image: ghcr.io/converged-computing/flux-view-rocky:arm-9 + + # This container is built for AWS EFA with libfabric + containers: + # This was built on hpc7g + - image: ghcr.io/converged-computing/lammps-reax-efa:ubuntu2404-efa + + # You can set the working directory if your container WORKDIR is not correct. + workingDir: /opt/lammps/examples/reaxff/HNS + command: lmp -v x 4 -v y 4 -v z 4 -in in.reaxff.hns -nocite + + # This is more expert optimization to ensure /dev/shm does not limit efa + volumes: + shared-memory: + emptyDir: true + emptyDirMedium: "memory" diff --git a/2026/HPCIC-AWS/tutorial/ch4/flux-mummi-createsims.yaml b/2026/HPCIC-AWS/tutorial/ch4/flux-mummi-createsims.yaml new file mode 100644 index 00000000..fc45a43b --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/ch4/flux-mummi-createsims.yaml @@ -0,0 +1,22 @@ +apiVersion: flux-framework.org/v1alpha2 +kind: MiniCluster +metadata: + name: createsims +spec: + size: 1 + # Setting tasks to 0 uses what is found on the node. + tasks: 0 + logging: + quiet: true + + flux: + container: + image: ghcr.io/converged-computing/flux-view-rocky:arm-9 + + containers: + - image: ghcr.io/converged-computing/sc25-flux-eks:createsims-arm + command: bash /entrypoint.sh + volumes: + shared-memory: + emptyDir: true + emptyDirMedium: "memory" diff --git a/2026/HPCIC-AWS/tutorial/ch4/img/flux-usernetes-turkducken.png b/2026/HPCIC-AWS/tutorial/ch4/img/flux-usernetes-turkducken.png new file mode 100644 index 00000000..bf8a1258 Binary files /dev/null and b/2026/HPCIC-AWS/tutorial/ch4/img/flux-usernetes-turkducken.png differ diff --git a/2026/HPCIC-AWS/tutorial/ch4/img/mummi.png b/2026/HPCIC-AWS/tutorial/ch4/img/mummi.png new file mode 100644 index 00000000..580e2f38 Binary files /dev/null and b/2026/HPCIC-AWS/tutorial/ch4/img/mummi.png differ diff --git a/2026/HPCIC-AWS/tutorial/ch4/img/mummi.svg b/2026/HPCIC-AWS/tutorial/ch4/img/mummi.svg new file mode 100644 index 00000000..8a4bfe7d --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/ch4/img/mummi.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/2026/HPCIC-AWS/tutorial/ch4/img/usernetes-flux.png b/2026/HPCIC-AWS/tutorial/ch4/img/usernetes-flux.png new file mode 100644 index 00000000..cd0b5c37 Binary files /dev/null and b/2026/HPCIC-AWS/tutorial/ch4/img/usernetes-flux.png differ diff --git a/2026/HPCIC-AWS/tutorial/ch4/ml-server-deployment.yaml b/2026/HPCIC-AWS/tutorial/ch4/ml-server-deployment.yaml new file mode 100644 index 00000000..d4d70eca --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/ch4/ml-server-deployment.yaml @@ -0,0 +1,64 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: ml-server +spec: + selector: + matchLabels: + run: ml-service + replicas: 1 + template: + metadata: + labels: + run: ml-service + spec: + containers: + - name: ml-service + image: ghcr.io/converged-computing/lammps-stream-ml:server-arm + env: + - name: SHELVE_SECRET_KEY + value: noodles + - name: JWT_SECRET_KEY + value: noodles + - name: SECRET_KEY + value: "qbg(sz#n+4ji)(16xrn!gti%^ig#a5ni!sxg18nv7sv*n#h*7%" + # We don't use hostPort here because we are using nginx ingress controller + ports: + - containerPort: 8080 +--- +apiVersion: v1 +kind: Service +metadata: + name: ml-service-endpoint # A name for the service +spec: + selector: + run: ml-service # This MUST match the labels on your pod + ports: + - protocol: TCP + # The port the service will listen on + port: 80 + # The port to forward traffic to on the pod + targetPort: 8080 +--- +# Ingress exposes the service to you +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: ml-server-ingress +spec: + # This line is important for the NGINX ingress controller + ingressClassName: "nginx" + rules: + - http: + paths: + - path: / + pathType: Prefix + backend: + service: + # Must match the Service name + name: ml-service-endpoint + port: + # Must match the Service's port + number: 80 + + diff --git a/2026/HPCIC-AWS/tutorial/ch4/model-scripts/1-create-models.py b/2026/HPCIC-AWS/tutorial/ch4/model-scripts/1-create-models.py new file mode 100644 index 00000000..1feac244 --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/ch4/model-scripts/1-create-models.py @@ -0,0 +1,36 @@ +#!/usr/bin/env python3 + +# This can be run from inside a client (or the lammps container) +# that has river and the client installed. + +import sys + +from river import linear_model, preprocessing +from riverapi.main import Client + +url = "http://localhost" +if len(sys.argv) > 1: + url = sys.argv[1] + +print(f"Preparing to create models for client URL {url}") + +# Connect to the server running here +cli = Client(url) + +# Upload several models to test for lammps - these are different kinds of regressions +regression_model = preprocessing.StandardScaler() | linear_model.LinearRegression( + intercept_lr=0.1 +) + +# That's kind of cool, although I'm not sure I like PA people, not sure how I feel about ML models :) +# https://www.geeksforgeeks.org/passive-aggressive-classifiers/ +pa_regression = model = linear_model.PARegressor( + C=0.01, mode=2, eps=0.1, learn_intercept=False +) + +# https://riverml.xyz/latest/api/linear-model/BayesianLinearRegression/ +bayesian_regression = linear_model.BayesianLinearRegression() + +for model in [regression_model, bayesian_regression, pa_regression]: + model_name = cli.upload_model(model, "regression") + print("Created model %s" % model_name) diff --git a/2026/HPCIC-AWS/tutorial/ch4/model-scripts/2-run-lammps-flux.py b/2026/HPCIC-AWS/tutorial/ch4/model-scripts/2-run-lammps-flux.py new file mode 100644 index 00000000..b019462f --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/ch4/model-scripts/2-run-lammps-flux.py @@ -0,0 +1,379 @@ +#!/usr/bin/env python3 + +# After the models are created with 1-create-models.py we can run +# this demo that will run lammps some number of times (in serial since I'm +# on my local machine) and use the matrix data for training the models it +# discovers. This script has combined the test and train functions to be +# able to use shared logic to run lammps (to avoid duplication) + +# This script requires the riverapi +# pip3 install riverapi + +import argparse +import random +import shutil +import subprocess +import json +import sys + +from riverapi.main import Client +from river import metrics + +# Find the software we need, flux and lammps +flux = shutil.which("flux") +lmp = shutil.which("lmp") + +if not flux or not lmp: + sys.exit("Cannot find flux or lmp executable.") + + +def get_parser(): + parser = argparse.ArgumentParser( + description="LAMMPS Run (Train or Test) Flux", + formatter_class=argparse.RawTextHelpFormatter, + ) + subparsers = parser.add_subparsers( + help="actions", + title="actions", + description="actions", + dest="command", + ) + + # print version and exit + train = subparsers.add_parser("train", description="train models") + test = subparsers.add_parser( + "predict", + description="test models by making predictions and comparing to truth", + ) + + # Add output file for test (actual and predictions) + test.add_argument( + "--out", + help="Output json to write actual and predicted values with x,y,z", + ) + + for command in [train, test]: + command.add_argument( + "--url", + help="URL where ml-server is deployed", + default="http://localhost", + ) + command.add_argument( + "--workdir", + default="/opt/lammps/examples/reaxff/HNS", + help="Working directory to run lammps from.", + ) + command.add_argument( + "--in", + dest="inputs", + default="in.reaxff.hns -nocite", + help="Input and parameters for lammps", + ) + command.add_argument( + "--nodes", + help="number of nodes (N)", + default=1, + type=int, + ) + command.add_argument( + "--log", + help="write log to path", + default="/tmp/lammps.log", + ) + command.add_argument( + "--np", + help="number of processes per node", + default=64, + type=int, + ) + + # Mins and maxes for each parameter - I decided to allow up to 32, 32, 32 for testing. + # On the cluster with cpu affinity set this is 4 minutes 41 seconds + command.add_argument( + "--x-min", + dest="x_min", + help="min dimension for x", + default=1, + type=int, + ) + command.add_argument( + "--x-max", + dest="x_max", + help="max dimension for x", + default=3, + type=int, + ) + command.add_argument( + "--y-min", + dest="y_min", + help="min dimension for y", + default=1, + type=int, + ) + command.add_argument( + "--y-max", + dest="y_max", + help="max dimension for y", + default=3, + type=int, + ) + command.add_argument( + "--z-min", + dest="z_min", + help="min dimension for z", + default=1, + type=int, + ) + command.add_argument( + "--z-max", + dest="z_max", + help="max dimension for z", + default=3, + type=int, + ) + command.add_argument( + "--iters", + help="iterations to run of lammps", + default=20, + type=int, + ) + return parser + + +def validate(args): + for dim, min_value, max_value in [ + ["x", args.x_min, args.x_max], + ["y", args.y_min, args.y_max], + ["z", args.z_min, args.z_max], + ]: + if min_value < 1: + sys.exit(f"Min value for {dim} must be greater than or equal to 1") + if min_value >= max_value: + sys.exit(f"Max value for {dim} must be greater than or equal to min") + if max_value < 1: + sys.exit( + f"Max for {dim} also needs to be positive >1. Also, we should never get here." + ) + + +def parse_time(line): + line = line.rsplit(" ", 1)[-1] + hours, minutes, seconds = line.split(":") + return (int(hours) * 60 * 60) + (int(minutes) * 60) + int(seconds) + + +def run_lammps(args): + """ + Shared function to run lammps for train or testing. + + We return (yield) chosen x,y,z and time in seconds as we run + """ + # Input files + inputs = args.inputs.split(" ") + + # Sanity check values + validate(args) + + # Choose ranges to allow for each of x, y, and z. + x_choices = list(range(args.x_min, args.x_max + 1)) + y_choices = list(range(args.y_min, args.y_max + 1)) + z_choices = list(range(args.z_min, args.z_max + 1)) + + for i in range(args.iters): + x = random.choice(x_choices) + y = random.choice(y_choices) + z = random.choice(z_choices) + print(f"\n🥑 Running iteration {i} with chosen x: {x} y: {y} z: {z}") + + # flux run -N 1 --ntasks 64 -c 1 -o cpu-affinity=per-task --cwd /opt/lammps/examples/reaxff/HNS lmp -v x 32 -v y 8 -v z 16 -in in.reaxff.hns -nocite + # Separate commands for easier printing + flux_cmd = [ + flux, + "run", + "-N", + str(args.nodes), + "--cwd", + args.workdir, + "--ntasks", + str(args.np), + # These aren't exposed as options because we pretty much always want them + "-c", + "1", + "-o", + "cpu-affinity=per-task", + ] + lammps_cmd = [ + lmp, + "-v", + "x", + str(x), + "-v", + "y", + str(y), + "-v", + "z", + str(z), + "-log", + args.log, + "-in", + ] + inputs + + print(" flux => " + " ".join(flux_cmd)) + print(" lammps => " + " ".join(lammps_cmd)) + cmd = flux_cmd + lammps_cmd + p = subprocess.Popen( + cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True + ) + + # This will hang until it's done, either submit or run + # We could save the log here, but I'm just going to grab the time + output, errors = p.communicate() + line = [x for x in output.split("\n") if x][-1] + + # Note this is currently written to run experiments, meaning we use all resources available + # for each run, and can just wait for the run and parse output. If you want to use flux submit, + # you can instead write each to a log file, read the log file, and parse the same. + if "total wall time" not in line.lower(): + print(f"Warning, there was an issue with iteration {i}") + print(output) + print(errors) + continue + + seconds = parse_time(line) + print(f" result => Lammps run took {seconds} seconds") + yield x, y, z, seconds + + +def make_prediction(cli, args, x, y, z): + """ + Make a prediction. + """ + test_x = {"x": x, "y": y, "z": z} + for model_name in cli.models()["models"]: + pred = cli.predict(model_name, x=test_x)["prediction"] + print(f"Model {model_name} predicts {pred}") + yield model_name, pred + + +def submit_train_result(cli, args, x, y, z, seconds): + """ + Submit a training result + """ + print(f"Preparing to send LAMMPS data to {args.url}") + + # Send this to the server to train each model + train_x = {"x": x, "y": y, "z": z} + for model_name in cli.models()["models"]: + print(f" Training {model_name} with {train_x} to predict {seconds}") + res = cli.learn(model_name, x=train_x, y=seconds) + if "successful learn" not in res.lower(): + print(f"Issue with learn: {res}") + + +def show_metrics(cli, y_true, y_pred): + """ + Show metrics (and return simple view for each model) + """ + results = {} + + # When we are done, calculate metrics for each + for model_name in cli.models()["models"]: + # Mean squared error + mse_metric = metrics.MSE() + + # Root mean squared error + rmse_metric = metrics.RMSE() + + # Mean absolute error + mae_metric = metrics.MAE() + + # Coefficient of determination () score - r squared + # proportion of the variance in the dependent variable that is predictable from the independent variable(s) + r2_metric = metrics.R2() + + for yt, yp in zip(y_true, y_pred[model_name]): + mse_metric.update(yt, yp) + rmse_metric.update(yt, yp) + mae_metric.update(yt, yp) + r2_metric.update(yt, yp) + + print(f"\n⭐️ Performance for: {model_name}") + print(f" R Squared Error: {r2_metric.get()}") + print(f" Mean Squared Error: {mse_metric.get()}") + print(f" Mean Absolute Error: {mae_metric.get()}") + print(f" Root Mean Squared Error: {rmse_metric.get()}") + + results[model_name] = { + "r_squared": r2_metric.get(), + "mean_squared_error": mse_metric.get(), + "mean_absolute_error": mae_metric.get(), + "root_mean_squared_error": rmse_metric.get(), + "stats": cli.stats(model_name), + "metrics": cli.metrics(model_name), + "model": cli.get_model_json(model_name), + "model_name": model_name, + } + return results + + +def write_output(filename, result): + """ + Write output to json file + """ + with open(filename, "w") as fd: + fd.write(json.dumps(result, indent=4)) + + +def main(): + parser = get_parser() + + # If an error occurs while parsing the arguments, the interpreter will exit with value 2 + args, _ = parser.parse_known_args() + + # I actually don't think I need this check. + if not args.command: + sys.exit("Please run and specify 'train' or 'test'") + if args.command not in ["train", "predict"]: + sys.exit(f"{args.command} is not recognized.") + + print(f"Preparing to run lammps and {args.command} models with Flux") + + # Connect to the server running here + cli = Client(args.url) + + # Do a test to the client + res = cli.info() + print(json.dumps(res, indent=4)) + + # If we are predicting, we will save true / predicted values + # https://riverml.xyz/latest/api/metrics/Accuracy/ + # Keep a listing actual and predictions (predictions namespaced by model) + y_true = [] + y_pred = {} + dims = [] + + for x, y, z, seconds in run_lammps(args): + # If we are training, we are done here! + if args.command == "train": + submit_train_result(cli, args, x, y, z, seconds) + else: + # Add true value to vector, and save dimensions + y_true.append(seconds) + dims.append({"x": x, "y": y, "z": z}) + + # Make a prediction + for model_name, pred in make_prediction(cli, args, x, y, z): + if model_name not in y_pred: + y_pred[model_name] = [] + y_pred[model_name].append(pred) + + # When we are finished running, if we are predicting, give final results + if args.command == "predict": + results = show_metrics(cli, y_true, y_pred) + if args.out is not None: + results.update({"dims": dims, "y_pred": y_pred, "y_true": y_true}) + write_output(args.out, results) + + +if __name__ == "__main__": + main() diff --git a/2026/HPCIC-AWS/tutorial/ch4/pytorch-mnist.yaml b/2026/HPCIC-AWS/tutorial/ch4/pytorch-mnist.yaml new file mode 100644 index 00000000..a5edd4ac --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/ch4/pytorch-mnist.yaml @@ -0,0 +1,14 @@ +apiVersion: trainer.kubeflow.org/v1alpha1 +kind: TrainJob +metadata: + name: pytorch-simple +spec: + runtimeRef: + name: torch-distributed + trainer: + numNodes: 2 + image: kubeflowkatib/pytorch-mnist-cpu:v1beta1-a524f33 + command: + - "python3" + - "/opt/pytorch-mnist/mnist.py" + - "--epochs=1" diff --git a/2026/HPCIC-AWS/tutorial/ch4/usernetes-workspace.jupyterlab-workspace b/2026/HPCIC-AWS/tutorial/ch4/usernetes-workspace.jupyterlab-workspace new file mode 100644 index 00000000..dab57ec1 --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/ch4/usernetes-workspace.jupyterlab-workspace @@ -0,0 +1 @@ +{"data":{"file-browser-filebrowser:columns":{"sizes":{"name":139.5,"file_size":null,"is_selected":18,"last_modified":86.5}},"layout-restorer:data":{"main":{"dock":{"type":"split-area","orientation":"horizontal","sizes":[0.41970629208139754,0.5802937079186026],"children":[{"type":"tab-area","currentIndex":0,"widgets":["terminal:1"]},{"type":"tab-area","currentIndex":0,"widgets":["notebook:tutorial/ch4/04_flux_framework_usernetes.ipynb"]}]},"current":"terminal:1"},"down":{"size":0,"widgets":[]},"left":{"collapsed":false,"visible":true,"current":"filebrowser","widgets":["filebrowser","running-sessions","@jupyterlab/toc:plugin","extensionmanager.main-view"],"widgetStates":{"jp-running-sessions":{"sizes":[0.14285714285714285,0.14285714285714285,0.14285714285714285,0.14285714285714285,0.14285714285714285,0.14285714285714285,0.14285714285714285],"expansionStates":[false,false,false,false,false,false,false]},"extensionmanager.main-view":{"sizes":[0.3333333333333333,0.3333333333333333,0.3333333333333333],"expansionStates":[false,false,false]}}},"right":{"collapsed":true,"visible":true,"widgets":["jp-property-inspector","debugger-sidebar"],"widgetStates":{"jp-debugger-sidebar":{"sizes":[0.2,0.2,0.2,0.2,0.2],"expansionStates":[false,false,false,false,false]}}},"relativeSizes":[0.19204284621270085,0.8079571537872992,0],"top":{"simpleVisibility":true}},"docmanager:recents":{"opened":[{"path":"tutorial/ch4","contentType":"directory","root":"/home/ubuntu"},{"path":"tutorial/ch4/04_flux_framework_usernetes.ipynb","contentType":"notebook","factory":"Notebook","root":"/home/ubuntu"},{"path":"tutorial/ch4/Untitled.ipynb","contentType":"notebook","factory":"Notebook","root":"/home/ubuntu"}],"closed":[]},"file-browser-filebrowser:cwd":{"path":"tutorial/ch4"},"notebook:tutorial/ch4/04_flux_framework_usernetes.ipynb":{"data":{"path":"tutorial/ch4/04_flux_framework_usernetes.ipynb","factory":"Notebook"}},"terminal:1":{"data":{"name":"1"}}},"metadata":{"id":"usernetes-workspace","last_modified":"2025-08-02T04:09:20.922759+00:00","created":"2025-08-02T04:09:20.922759+00:00"}} \ No newline at end of file diff --git a/2026/HPCIC-AWS/tutorial/ch5/05_flux_quantum_braket.ipynb b/2026/HPCIC-AWS/tutorial/ch5/05_flux_quantum_braket.ipynb new file mode 100644 index 00000000..c1e6e75e --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/ch5/05_flux_quantum_braket.ipynb @@ -0,0 +1,271 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "
\n", + "
\n", + "
\n", + "\n", + "# Chapter 5: Hybrid Quantum-Classical Workflows with Flux\n", + "\n", + "In the previous chapter you saw Flux run classical HPC and AI/ML workloads and orchestrate user-space Kubernetes. In this chapter we add a *quantum* backend to the mix. We will run a **Quantum Approximate Optimization Algorithm (QAOA)** for graph max-cut, a hybrid workload where a classical optimizer running under Flux repeatedly evaluates a quantum circuit on a **quantum simulator**.\n", + "\n", + "Everything in this chapter runs on a **local simulator** that ships with the [AWS Braket SDK](https://github.com/amazon-braket/amazon-braket-sdk-python). It executes right here on the VM, so there is:\n", + "- **no AWS account or credentials required**,\n", + "- **no network calls**, and\n", + "- **no cost**.\n", + "\n", + "In this tutorial we will:\n", + "1. Confirm the local simulator is available\n", + "2. Run QAOA under Flux with `flux run`\n", + "3. Submit it non-blocking and inspect it with `flux submit` / `flux jobs`\n", + "4. Scale out to an **ensemble** of quantum simulations with Flux\n", + "5. (Advanced) Run the same job as a **container in Usernetes**\n", + "
\n", + "\n", + "The classical part (building circuits, the COBYLA optimizer, computing the cut) and the quantum part (executing the circuit, sampling measurements) both run on your Flux-managed VM. This is exactly the hybrid quantum-classical pattern used in real research, and it maps naturally onto an HPC workload manager: Flux schedules the classical driver, and each circuit evaluation is a call into the quantum simulator.\n", + "\n", + "The application here is a condensed, single-file version of the pipeline in [converged-computing/quantum-braket](https://github.com/converged-computing/quantum-braket).\n", + "\n", + "If you haven't yet, open a Jupyter terminal alongside this notebook so you can copy-paste commands into it." + ], + "id": "cell-00" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Background: what is QAOA max-cut?\n", + "\n", + "**Max-cut** asks: given a graph, partition its nodes into two groups so that the number of edges crossing between the groups (the \"cut\") is as large as possible. It is a classic NP-hard combinatorial optimization problem.\n", + "\n", + "**QAOA** encodes each node as a qubit and builds a parameterized quantum circuit with two alternating layers: a *cost* layer (parameter `gamma`) and a *mixer* layer (parameter `beta`). Measuring the circuit yields bitstrings that assign each node to one of the two groups. A classical optimizer (here, COBYLA) adjusts `gamma` and `beta` to push the expected cut value higher.\n", + "\n", + "Each optimizer iteration builds one circuit, runs it on the quantum backend, gets back measurement counts, and computes the average cut. That is the hybrid loop, and it is what Flux will schedule for us.\n", + "\n", + "
\n", + "\u2705 Local and free. We use the Braket SDK's built-in local state-vector simulator. It runs the identical circuits an AWS device would, but entirely on this VM. No credentials, no network, no cost. (If you later want to run on real AWS Braket hardware or the cloud SV1 simulator, the same script supports it with a flag \u2014 see the optional appendix at the end.)\n", + "
" + ], + "id": "cell-01" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 1. Setup\n", + "\n", + "Move into the chapter directory:\n", + "\n", + "```bash\n", + "cd ~/tutorial/ch5\n", + "```\n", + "\n", + "The Braket SDK and SciPy are pre-installed on the tutorial image. If for some reason they are missing, install them into your user environment (no AWS packages or configuration are needed):\n", + "\n", + "```bash\n", + "pip install --user amazon-braket-sdk scipy\n", + "```\n", + "\n", + "Confirm the local simulator works. This creates a tiny Bell-state circuit and samples it \u2014 a quick sanity check that runs in a fraction of a second, entirely locally:\n", + "\n", + "```bash\n", + "python3 -c \"\n", + "from braket.circuits import Circuit\n", + "from braket.devices import LocalSimulator\n", + "circ = Circuit().h(0).cnot(0, 1)\n", + "result = LocalSimulator().run(circ, shots=100).result()\n", + "print('Bell-state counts:', dict(result.measurement_counts))\n", + "\"\n", + "```\n", + "\n", + "You should see counts split between `00` and `11`, e.g. `{'00': 52, '11': 48}`. No AWS calls happen here." + ], + "id": "cell-02" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 2. Run QAOA under Flux\n", + "\n", + "
\n", + "Description: Run the full hybrid loop as a Flux job. The classical optimizer and the quantum simulation both run locally; Flux schedules the driver just like any other job.\n", + "
\n", + "\n", + "We are already inside a Flux instance (the notebook was launched with `flux start`), so we can hand the driver to Flux. `flux run` blocks until the job finishes and streams its output:\n", + "\n", + "```bash\n", + "flux run python3 scripts/qaoa_maxcut.py --graph-nodes 6 --max-iter 5\n", + "```\n", + "\n", + "> **Note:** `--graph-nodes 6` is the *problem size* \u2014 a 6-vertex graph, i.e. 6 qubits. It has nothing to do with Flux compute nodes (this tutorial instance has **4**). Each `flux run` / `flux submit` here uses a single node and task unless you request more.\n", + "\n", + "You will see each evaluation print the current cut value as COBYLA searches, followed by a summary with the best cut and the approximation ratio. The backend defaults to `local`, so nothing leaves the VM.\n", + "\n", + "The script is a single file that generates a random 3-regular graph, builds the QAOA circuit, runs it on the local simulator, and drives the optimizer. Take a look:\n", + "\n", + "```bash\n", + "less scripts/qaoa_maxcut.py\n", + "```" + ], + "id": "cell-03" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 3. Submit non-blocking with `flux submit`\n", + "\n", + "
\n", + "Description: Swap flux run for flux submit to queue the job without blocking. This is closer to how you would drive many circuit evaluations in a real workflow.\n", + "
\n", + "\n", + "```bash\n", + "# Submit without blocking; Flux returns a job ID immediately\n", + "flux submit --job-name qaoa python3 scripts/qaoa_maxcut.py --graph-nodes 6 --max-iter 5\n", + "\n", + "# Watch the job in the queue\n", + "flux jobs -a\n", + "\n", + "# Attach to the most recent job to see its output\n", + "flux job attach $(flux job last)\n", + "```" + ], + "id": "cell-04" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 4. Scale out: an ensemble of quantum simulations with Flux\n", + "\n", + "
\n", + "Description: A key strength of a workload manager is running many independent jobs at once. Let's have Flux launch several QAOA instances \u2014 each on a different random graph \u2014 as a small ensemble.\n", + "
\n", + "\n", + "This mirrors the bulk-submit pattern from Chapter 2, but now each Flux job drives a quantum simulation:\n", + "\n", + "```bash\n", + "for seed in 1 2 3 4; do\n", + " flux submit -N1 --job-name qaoa-$seed \\\n", + " python3 scripts/qaoa_maxcut.py --graph-nodes 6 --max-iter 3 --seed $seed\n", + "done\n", + "\n", + "# See all four running/completed\n", + "flux jobs -a\n", + "```\n", + "\n", + "When they finish, inspect the result of any one of them:\n", + "\n", + "```bash\n", + "flux job attach $(flux jobs -a --no-header | awk 'NR==1{print $1}')\n", + "```\n", + "\n", + "Here `-N1` asks Flux for one node per job, so on this 4-node instance the four jobs run one-per-node at the same time. Each job independently builds and simulates its circuits, and Flux tracks them as a group. Conceptually this is how you would fan out a parameter sweep or a batch of variational circuits from an HPC scheduler \u2014 and because it is all local, you can run as many as you like for free." + ], + "id": "cell-05" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 5. (Advanced) Run as a container in Usernetes\n", + "\n", + "
\n", + "Description: Optional. Run the same QAOA job as a container inside the user-space Kubernetes cluster from Chapter 4. Because everything is local, there are no secrets and no credentials to manage \u2014 the pod just runs the simulator.\n", + "
\n", + "\n", + "Make sure Usernetes is running (from Chapter 4) and your kubeconfig is set:\n", + "\n", + "```bash\n", + "export KUBECONFIG=/home/ubuntu/usernetes/kubeconfig\n", + "kubectl get nodes\n", + "```\n", + "\n", + "We deliver the script to the pod with a ConfigMap (no image build required), then run it on a stock Python image:\n", + "\n", + "```bash\n", + "# Package the script as a ConfigMap\n", + "kubectl create configmap qaoa-script \\\n", + " --from-file=qaoa_maxcut.py=scripts/qaoa_maxcut.py \\\n", + " --dry-run=client -o yaml | kubectl apply -f -\n", + "\n", + "# Launch the Job (installs the SDK, then runs the local simulator)\n", + "kubectl apply -f braket-qaoa-local.yaml\n", + "```\n", + "\n", + "Watch it run and read the output:\n", + "\n", + "```bash\n", + "kubectl get pods -w\n", + "kubectl logs -f job/qaoa-local\n", + "```\n", + "\n", + "Clean up when you are done:\n", + "\n", + "```bash\n", + "kubectl delete -f braket-qaoa-local.yaml\n", + "kubectl delete configmap qaoa-script\n", + "```\n", + "\n", + "The pod makes no external calls beyond pulling the base image and pip packages, and it never handles AWS credentials." + ], + "id": "cell-06" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Wrap-up\n", + "\n", + "You just ran a hybrid quantum-classical optimization from an HPC workload manager, with no cloud account and no credentials:\n", + "\n", + "- The **classical driver** (circuit construction + COBYLA) ran as an ordinary Flux job.\n", + "- Each **circuit evaluation** ran on the **local** quantum simulator on the VM.\n", + "- Flux scheduled a **single run**, a **non-blocking submission**, and an **ensemble** of quantum jobs.\n", + "- The same work ran as a **container in Usernetes** \u2014 still credential-free.\n", + "\n", + "Thanks for following along!\n", + "\n", + "Please take our 2026 survey: https://converged-computing.org/community-survey\n", + "\n", + "---\n", + "\n", + "### Optional appendix: running on AWS Braket SV1\n", + "\n", + "If you have your own AWS Braket access and *want* to run the identical workload on the cloud SV1 state-vector simulator, the script supports it with a flag:\n", + "\n", + "```bash\n", + "flux run python3 scripts/qaoa_maxcut.py --backend sv1 --graph-nodes 6 --shots 100 --max-iter 5\n", + "```\n", + "\n", + "This path requires AWS credentials with Braket permissions and incurs a small per-minute cost (SV1 is billed at \\$0.075/minute in `us-east-1`, with a 1-hour/month free tier for the first 12 months; a run like this is a few cents at most). It is **not** required for the tutorial \u2014 the local simulator above covers everything we teach here. An example least-privilege IAM policy scoped to only the SV1 device is included in the repository at `ec2/jupyterhub-braket-policy.json` for administrators who choose to enable it." + ], + "id": "cell-07" + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.12.3" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/2026/HPCIC-AWS/tutorial/ch5/README.md b/2026/HPCIC-AWS/tutorial/ch5/README.md new file mode 100644 index 00000000..dcc1f092 --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/ch5/README.md @@ -0,0 +1,46 @@ +# Chapter 5: Hybrid Quantum-Classical Workflows with Flux and AWS Braket + +This chapter adds a quantum backend to the tutorial. It runs a QAOA max-cut +optimization as a hybrid quantum-classical workflow: a classical COBYLA optimizer +scheduled by Flux repeatedly evaluates a quantum circuit on the AWS Braket **SV1** +state-vector simulator. + +The notebook, [05_flux_quantum_braket.ipynb](05_flux_quantum_braket.ipynb), walks +through five parts: + +1. Setup and confirming SV1 access +2. QAOA on the free **local** simulator under Flux (no cost) +3. QAOA on **AWS Braket SV1** under Flux +4. An **ensemble** of quantum tasks with Flux +5. (Advanced) The same pipeline as **containers in Usernetes** + +## Files + +- `scripts/qaoa_maxcut.py` — self-contained QAOA max-cut driver. Condenses the + four-stage pipeline from + [converged-computing/quantum-braket](https://github.com/converged-computing/quantum-braket) + (problem-generator → transpiler → gateway → optimizer) into one script that Flux + can launch. Supports `--backend local` (free) and `--backend sv1`. +- `braket-qaoa-pipeline.yaml` — the containerized pipeline as a Kubernetes Pod for + the optional Usernetes section, using the published + `ghcr.io/converged-computing/quantum-braket-*` images. + +## Credentials and IAM + +The tutorial VM has **no** stored AWS keys (`~/.aws` is removed at boot). It runs +with an IAM instance role whose only Braket permission is the SV1 device ARN +`arn:aws:braket:::device/quantum-simulator/amazon/sv1`. The Braket SDK picks up the +instance-role credentials automatically via instance metadata, so the notebook +never configures a key, and any attempt to reach another device is denied. + +The role's Braket + S3 permissions come from +[`../../ec2/jupyterhub-braket-policy.json`](../../ec2/jupyterhub-braket-policy.json). +See the top-level [README](../../README.md) for the one-time deployment commands +(attaching the policy and creating the Braket service-linked role). + +## Cost + +SV1 is billed at **$0.075 / minute** in `us-east-1`, with **one free hour of +simulation per month** for the first 12 months. The circuits here run in seconds, +so a full run is a few cents at most (often $0 under the free tier). Use the +`local` backend for cost-free testing. diff --git a/2026/HPCIC-AWS/tutorial/ch5/braket-qaoa-local.yaml b/2026/HPCIC-AWS/tutorial/ch5/braket-qaoa-local.yaml new file mode 100644 index 00000000..b3f74bf3 --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/ch5/braket-qaoa-local.yaml @@ -0,0 +1,38 @@ +apiVersion: batch/v1 +kind: Job +metadata: + name: qaoa-local + labels: + app: quantum-braket +spec: + backoffLimit: 1 + template: + metadata: + labels: + app: quantum-braket + spec: + restartPolicy: Never + containers: + - name: qaoa + image: python:3.11-slim + # Runs the SAME qaoa_maxcut.py on the LOCAL Braket simulator inside the + # pod. There are no AWS calls and no credentials anywhere in this Job -- + # the script is delivered via the 'qaoa-script' ConfigMap (created from + # scripts/qaoa_maxcut.py) and mounted read-only at /app. + command: ["/bin/sh", "-c"] + args: + - | + set -e + pip install --quiet --no-cache-dir amazon-braket-sdk scipy + python3 /app/qaoa_maxcut.py --backend local --nodes 6 --shots 100 --max-iter 5 + env: + - name: BACKEND + value: "local" + volumeMounts: + - name: qaoa-script + mountPath: /app + readOnly: true + volumes: + - name: qaoa-script + configMap: + name: qaoa-script diff --git a/2026/HPCIC-AWS/tutorial/ch5/scripts/qaoa_maxcut.py b/2026/HPCIC-AWS/tutorial/ch5/scripts/qaoa_maxcut.py new file mode 100644 index 00000000..0cdc0172 --- /dev/null +++ b/2026/HPCIC-AWS/tutorial/ch5/scripts/qaoa_maxcut.py @@ -0,0 +1,279 @@ +#!/usr/bin/env python3 +""" +qaoa_maxcut.py -- a single-file QAOA max-cut demo for the Flux + Usernetes tutorial. + +This is a self-contained condensation of the four-stage pipeline from +https://github.com/converged-computing/quantum-braket +(problem-generator -> transpiler -> gateway -> optimizer) into one script so it +can be launched directly by Flux (``flux run`` / ``flux submit``) on the tutorial +VM. It builds a QAOA ansatz for max-cut on a random k-regular graph, evaluates it +on a quantum backend, and optimizes the variational parameters with COBYLA. + +Credentials +----------- +The default backend is the fully local simulator, which needs NO AWS account and +NO credentials. Nothing is configured or transmitted; the circuits run on this VM. + +The optional --backend sv1 path uses AWS Braket. Only use it if you have your own +Braket access; the tutorial itself never requires AWS credentials. (On an instance +that happens to carry a Braket-enabled IAM role, boto3 would pick that up +automatically via instance metadata -- but that is not needed for this tutorial.) + +Backends +-------- +--backend local Local state-vector simulator in the Braket SDK. No AWS calls, + no credentials, no cost. This is the DEFAULT and what the + tutorial uses throughout. +--backend sv1 (Optional) AWS Braket SV1 on-demand simulator. Requires AWS + Braket access and is billed per minute. Not used by the + tutorial by default. + +Cost +---- +The default local backend is free. The optional SV1 backend is $0.075/minute in +us-east-1 (with a 1-hour/month free tier for the first 12 months); a tiny circuit +runs in seconds, and the Braket Cost Tracker prints the estimate when used. + +Examples +-------- + # Local run (default, no AWS, no cost): full hybrid optimization + flux run python3 scripts/qaoa_maxcut.py --nodes 6 --max-iter 5 + + # Local, single circuit evaluation (no optimizer loop) + flux run python3 scripts/qaoa_maxcut.py --backend local --max-iter 0 + + # (Optional) real SV1 run, only if you have AWS Braket access + flux run python3 scripts/qaoa_maxcut.py --backend sv1 --nodes 6 --max-iter 5 + +Environment variables (overridden by CLI flags if given): + N_NODES, K_REGULAR, SEED, P_LAYERS, N_SHOTS, MAX_ITER, BRAKET_DEVICE, BACKEND +""" + +import argparse +import json +import math +import os +import random +import sys +import time + +SV1_ARN = "arn:aws:braket:::device/quantum-simulator/amazon/sv1" + + +# --------------------------------------------------------------------------- # +# 1. Problem generation: random k-regular graph (from problem-generator) +# --------------------------------------------------------------------------- # +def generate_k_regular_graph(n, k, seed): + """Random k-regular graph on n nodes via the configuration/pairing model.""" + if n * k % 2 != 0: + raise ValueError(f"n*k must be even (got n={n}, k={k})") + if k >= n: + raise ValueError(f"k must be less than n (got k={k}, n={n})") + + rng = random.Random(seed) + for _attempt in range(100): + stubs = [] + for node in range(n): + stubs.extend([node] * k) + rng.shuffle(stubs) + + edges = set() + valid = True + for i in range(0, len(stubs), 2): + u, v = stubs[i], stubs[i + 1] + if u == v or (min(u, v), max(u, v)) in edges: + valid = False + break + edges.add((min(u, v), max(u, v))) + if valid: + return [(u, v, 1.0) for u, v in sorted(edges)] + + raise RuntimeError( + f"Could not build a {k}-regular graph on {n} nodes in 100 tries; " + "try a different seed or smaller k." + ) + + +# --------------------------------------------------------------------------- # +# 2. Circuit construction (from transpiler / gateway) +# --------------------------------------------------------------------------- # +def build_qaoa_circuit(n_qubits, edges, gammas, betas): + """Build a Braket Circuit for the current (gammas, betas). Supports p >= 1.""" + from braket.circuits import Circuit + + circ = Circuit() + for i in range(n_qubits): + circ.h(i) + + for layer in range(len(gammas)): + gamma = gammas[layer] + beta = betas[layer] + # Cost unitary: CNOT - RZ(gamma) - CNOT on every edge + for u, v, _ in edges: + circ.cnot(u, v) + circ.rz(v, gamma) + circ.cnot(u, v) + # Mixer unitary: RX(2*beta) on every qubit + for i in range(n_qubits): + circ.rx(i, 2 * beta) + return circ + + +def initial_params(p, seed): + rng = random.Random(seed) + gammas = [rng.uniform(0.1, math.pi - 0.1) for _ in range(p)] + betas = [rng.uniform(0.1, math.pi / 2 - 0.1) for _ in range(p)] + return gammas, betas + + +# --------------------------------------------------------------------------- # +# 3. Cost function (from gateway) +# --------------------------------------------------------------------------- # +def maxcut_cost(counts, edges, n_qubits): + """Expected cut value over the measured bitstrings.""" + total_shots = sum(counts.values()) + total = 0.0 + for bitstring, shots in counts.items(): + if len(bitstring) != n_qubits: + bitstring = bitstring.zfill(n_qubits) + bits = [int(b) for b in bitstring] + cut = sum(w for u, v, w in edges if bits[u] != bits[v]) + total += shots * cut + return total / total_shots + + +# --------------------------------------------------------------------------- # +# Backend wiring +# --------------------------------------------------------------------------- # +def get_device(backend, device_arn): + if backend == "local": + from braket.devices import LocalSimulator + + return LocalSimulator() + from braket.aws import AwsDevice + + return AwsDevice(device_arn) + + +def run_circuit(device, circ, shots): + task = device.run(circ, shots=shots) + return task.result().measurement_counts + + +# --------------------------------------------------------------------------- # +# Main driver +# --------------------------------------------------------------------------- # +def main(): + p = argparse.ArgumentParser(description="QAOA max-cut on AWS Braket SV1 or local sim.") + p.add_argument("--backend", default=os.environ.get("BACKEND", "local"), + choices=["local", "sv1"], help="Quantum backend to use.") + p.add_argument("--device-arn", default=os.environ.get("BRAKET_DEVICE", SV1_ARN), + help="Braket device ARN (only used when --backend sv1).") + p.add_argument("--nodes", type=int, default=int(os.environ.get("N_NODES", 6))) + p.add_argument("--k", type=int, default=int(os.environ.get("K_REGULAR", 3))) + p.add_argument("--seed", type=int, default=int(os.environ.get("SEED", 42))) + p.add_argument("--p-layers", type=int, default=int(os.environ.get("P_LAYERS", 1))) + p.add_argument("--shots", type=int, default=int(os.environ.get("N_SHOTS", 100))) + p.add_argument("--max-iter", type=int, default=int(os.environ.get("MAX_ITER", 5)), + help="COBYLA iterations. Use 0 for a single circuit evaluation.") + args = p.parse_args() + + print(f"[qaoa] backend={args.backend} nodes={args.nodes} k={args.k} " + f"p={args.p_layers} shots={args.shots} max_iter={args.max_iter}") + + edges = generate_k_regular_graph(args.nodes, args.k, args.seed) + n_qubits = args.nodes + print(f"[qaoa] graph: {n_qubits} nodes, {len(edges)} edges " + f"(upper-bound cut = {len(edges)})") + + gammas0, betas0 = initial_params(args.p_layers, args.seed) + + # Build the device once. On SV1 this validates the device ARN via GetDevice. + try: + device = get_device(args.backend, args.device_arn) + except Exception as e: # noqa: BLE001 + print(f"[qaoa] ERROR creating device: {e}", file=sys.stderr) + sys.exit(1) + + # Optional cost tracking (SV1 only; the tracker is a no-op cost for local). + tracker_ctx = None + if args.backend == "sv1": + try: + from braket.tracking import Tracker + + tracker_ctx = Tracker() + tracker_ctx.__enter__() + except Exception: # noqa: BLE001 + tracker_ctx = None + + eval_count = [0] + + def evaluate(gammas, betas): + circ = build_qaoa_circuit(n_qubits, edges, gammas, betas) + t0 = time.time() + counts = run_circuit(device, circ, args.shots) + dt = time.time() - t0 + cost = maxcut_cost(counts, edges, n_qubits) + print(f"[qaoa] eval {eval_count[0]:2d} cut={cost:7.4f} " + f"gamma0={gammas[0]:.4f} beta0={betas[0]:.4f} ({dt:.2f}s)") + eval_count[0] += 1 + return cost + + # Initial evaluation + best_cost = evaluate(gammas0, betas0) + best_gammas, best_betas = gammas0, betas0 + + # Optional COBYLA optimization loop (maximize cut == minimize -cut) + if args.max_iter > 0: + try: + import numpy as np + from scipy.optimize import minimize + except ImportError: + print("[qaoa] scipy/numpy not available; skipping optimization.", + file=sys.stderr) + else: + x0 = np.array(gammas0 + betas0, dtype=float) + pl = args.p_layers + + def objective(x): + g = list(x[:pl]) + b = list(x[pl:]) + return -evaluate(g, b) + + res = minimize(objective, x0, method="COBYLA", + options={"maxiter": args.max_iter, "rhobeg": 0.5}) + best_gammas = list(res.x[:pl]) + best_betas = list(res.x[pl:]) + best_cost = -res.fun + print(f"[qaoa] optimizer done: {res.message}") + + approx_ratio = best_cost / len(edges) if edges else 0.0 + print("\n[qaoa] ===== Result =====") + print(f" best cut : {best_cost:.4f}") + print(f" approximation ratio : {approx_ratio:.4f}") + print(f" evaluations : {eval_count[0]}") + + if tracker_ctx is not None: + try: + cost = tracker_ctx.simulator_tasks_cost() + print(f" estimated SV1 cost : ${float(cost):.4f} USD") + tracker_ctx.__exit__(None, None, None) + except Exception: # noqa: BLE001 + pass + + # Emit a machine-readable summary line (handy for Flux output scraping) + summary = { + "backend": args.backend, + "n_nodes": args.nodes, + "n_edges": len(edges), + "p": args.p_layers, + "shots": args.shots, + "best_cut": round(best_cost, 6), + "approximation_ratio": round(approx_ratio, 6), + "evaluations": eval_count[0], + } + print(f"[qaoa] SUMMARY {json.dumps(summary)}") + + +if __name__ == "__main__": + main() diff --git a/2026/HPCIC-AWS/tutorial/img/dl-training-io.png b/2026/HPCIC-AWS/tutorial/img/dl-training-io.png new file mode 100644 index 00000000..6129d0ef Binary files /dev/null and b/2026/HPCIC-AWS/tutorial/img/dl-training-io.png differ diff --git a/2026/HPCIC-AWS/tutorial/img/flux-batch.jpg b/2026/HPCIC-AWS/tutorial/img/flux-batch.jpg new file mode 100644 index 00000000..f7282bb4 Binary files /dev/null and b/2026/HPCIC-AWS/tutorial/img/flux-batch.jpg differ diff --git a/2026/HPCIC-AWS/tutorial/img/flux-broker-design.png b/2026/HPCIC-AWS/tutorial/img/flux-broker-design.png new file mode 100644 index 00000000..267f1a66 Binary files /dev/null and b/2026/HPCIC-AWS/tutorial/img/flux-broker-design.png differ diff --git a/2026/HPCIC-AWS/tutorial/img/flux-instance-pre-tbon.png b/2026/HPCIC-AWS/tutorial/img/flux-instance-pre-tbon.png new file mode 100644 index 00000000..bc40a7e4 Binary files /dev/null and b/2026/HPCIC-AWS/tutorial/img/flux-instance-pre-tbon.png differ diff --git a/2026/HPCIC-AWS/tutorial/img/flux-instance-w-tbon.png b/2026/HPCIC-AWS/tutorial/img/flux-instance-w-tbon.png new file mode 100644 index 00000000..93a276e8 Binary files /dev/null and b/2026/HPCIC-AWS/tutorial/img/flux-instance-w-tbon.png differ diff --git a/2026/HPCIC-AWS/tutorial/img/flux-tree.png b/2026/HPCIC-AWS/tutorial/img/flux-tree.png new file mode 100644 index 00000000..a0dba825 Binary files /dev/null and b/2026/HPCIC-AWS/tutorial/img/flux-tree.png differ diff --git a/2026/HPCIC-AWS/tutorial/img/instance-submit.png b/2026/HPCIC-AWS/tutorial/img/instance-submit.png new file mode 100644 index 00000000..84ce558e Binary files /dev/null and b/2026/HPCIC-AWS/tutorial/img/instance-submit.png differ diff --git a/2026/HPCIC-AWS/tutorial/img/scaled-submit.png b/2026/HPCIC-AWS/tutorial/img/scaled-submit.png new file mode 100644 index 00000000..a5dc3468 Binary files /dev/null and b/2026/HPCIC-AWS/tutorial/img/scaled-submit.png differ diff --git a/2026/HPCIC-AWS/tutorial/img/single-submit.png b/2026/HPCIC-AWS/tutorial/img/single-submit.png new file mode 100644 index 00000000..0592defe Binary files /dev/null and b/2026/HPCIC-AWS/tutorial/img/single-submit.png differ